A/B Test

A feature that allows you to test different message patterns to measure and compare the effectiveness of message delivery

Updated: August 25, 2025

Benefits of Using This Feature

• Compare the effectiveness of different message patterns objectively
• Improve delivery effectiveness through data-driven decision making
• Create a fair testing environment through random distribution
• Conduct A/B tests for both automated and manual delivery

A/B Testing Process and Data Flow

1. Message Creation

• Create two different message patterns
• Example: Messages with 500 yen OFF coupon vs. 10,000 yen OFF coupon

2. Workflow/Chatflow Configuration

• Specify the created A/B test message in your workflow or chatflow

3. Delivery and Test Execution

• The system automatically distributes recipients randomly between A/B patterns and executes delivery
• For manual delivery messages: Pattern A is delivered first, followed by Pattern B after completion
• For automated delivery messages: Pattern A and Pattern B are automatically sent to half of the recipients each during delivery

Setup Instructions

1. Preparing A/B Test Messages: Follow these detailed steps to create A/B test messages:

• Creating Pattern A Message
  • Create Pattern A message using either manual or automated delivery
  • Example: Pattern A features "500 yen OFF coupon gift" announcement
• Creating Pattern B Message
  • Create Pattern B message using the same delivery type (manual or automated delivery)
  • Example: Pattern B features "10,000 yen OFF coupon gift" announcement, with the same text structure as Pattern A but only the coupon amount changed
    
    
  • Example: Pattern B features "10,000 yen OFF coupon gift" announcement, with the same text structure as Pattern A but only the coupon amount changed

• Sending Messages and Creating Templates
  • Once message creation is complete, save and proceed with confirmation and delivery

2. Workflow/Chatflow Configuration How to configure when using A/B test functionality:

• Open the target workflow or chatflow
• Select message send action
• Specify the created A/B test message

▼Workflow Image

▼Chatflow Image

Practical Usage Examples

【Pattern A Example】
Subject: "500 Yen OFF Coupon Gift!"
Body: Thank you for your continued patronage.
We're giving you an exclusive 500 yen OFF coupon.
Please take advantage of this opportunity.

【Pattern B Example】  
Subject: "10,000 Yen OFF Coupon Gift!"
Body: Thank you for your continued patronage.
We're giving you an exclusive 10,000 yen OFF coupon.
Please take advantage of this opportunity.

Operation Verification

Message Creation Verification

• Confirm that both Pattern A and Pattern B messages have been created correctly

Delivery Settings Verification

• Confirm that the A/B test message has been selected in the workflow or chatflow

Delivery Results Verification

• After test delivery, confirm through reports that both A/B patterns have been delivered appropriately

A/B Test Message Performance Measurement

For A/B test delivered messages, you can individually measure and compare the performance of Version A and Version B.

How to Check A/B Test Results Click on a delivered A/B test message to view detailed performance for each version in the "Test Results" tab.

Performance Metrics Details The following metrics are measured for each version (A/B):

• Delivery Count
  • Number of LINE friends each version was sent to. In A/B tests, recipients are automatically distributed equally between versions.
• Reach Count
  • Number of friends each version was delivered to without being blocked.
• Open Count
  • Number of times each version was opened in the LINE app.
• Open Rate
  • Percentage of reached recipients who opened each version in the LINE app.
• Click Count
  • Number of times links within each version's message were clicked.
• Click Rate
  • Percentage of recipients who opened the message and then clicked each version.

How to Interpret A/B Test Results

• Compare metrics for each version to determine which pattern was more effective
• The version with higher click rate likely had more attractive message content
• The version with higher open rate possibly had more eye-catching titles or images

Utilizing Performance Results Based on A/B test results, selecting more effective patterns for future message creation can improve overall message performance.

Troubleshooting

Common Issues

Q: A/B test is not working correctly
A: Please confirm that both Pattern A and Pattern B messages have been created correctly. Also, recheck your workflow or chatflow settings.

Q: About delivery timing
A: Currently, Pattern B is delivered after Pattern A completion. Simultaneous A/B delivery functionality is under development.

Q: Want to change recipient distribution ratio
A: Currently, distribution is equal (50:50). Ratio adjustment functionality is planned for future updates.