API Documentation
Please click here for our API Documentation on A/B Split Testing.
A/B Split testing allows you to send multiple variants of the same push notification to different groups of users so that you can test which variant performs the best. An A/B split test can be set up for any campaign that includes a push notification. This includes both card campaigns and push notification campaigns that bring the user directly to a section of an app or a URL.
Setting Up an A/B Split Test
To set up an A/B split test, create a new campaign and open the push notification screen. On this page, you will see all the widgets under Variant A by default and indicates the proportion of users this variant would be sent to, which is 100% since there are no other variants.
Variants
If you wish to send a campaign with only the original variant, do not add any additional variants. To add a new version of the campaign, click the Add Variant Button and choose whether you would like to:
- Duplicate from original
- New Blank Variant
- Control group
- Learn more
If you choose to duplicate the original variant all of the text, images, buttons, and settings that you have entered into the original variant (Variant A) will be duplicated in the new variant. This is useful if you just want to make an incremental change to each variant (such as adding a different emoji) and not make each variant completely independent of the other.
Duplicate Variant
However, you can also choose to add a blank variant which will add a new variant that is completely blank and does not inherit anything from the original variant.
Blank Variant
Each variant is assigned a unique letter from A-C and a unique color so it is easy to differentiate between them. You can add a maximum of three variants and one control group for each campaign. Each variant can be completely different or each can contain a single modification. However, to maintain statistical relevance, it is recommended that the changes that you introduce to each variant are as minimal as possible so that it is easy to pinpoint which change caused each variant to perform better or worse than the others.
To remove any of the variants that have been set up, hover on that variant and click the X button to remove it. Please note that if a variant is deleted, all other variants will be updated to ensure that the order of the variants always stays in sequence from A-I.
Delete a Variant
Validation
Please note that to progress from the push notification screen, it is required that the push notification details for all variants are filled in. At a minimum, each variant must include notification text and a notification destination. Other fields and details may be required depending on what push notification options are selected.
Clicking on Control Group in the dropdown would redirect you to the Set Up page where the control group field would be automatically toggled on and you need to enter the percentage. Pulsate will automatically adjust the size of each variant based on the control group that is configured so that every variant is sent to the same percentage of users.
Choose Recipients
Statistical Relevance
Please make sure that you send the campaign to enough users so that the results are statistically relevant. If you would like some help making sure that your A/B split test campaigns are statistically relevant, please contact your Customer Success representative.
Choosing a Conversion Goal
The primary conversion goal that you choose for each campaign will be the metric that Pulsate uses to measure the performance of each variant and decide the winning variant. For more information on setting up Conversion Goals, click here
However, even though the primary conversion goal will act as the primary deciding factor in choosing what variant is crowned as the winning variant, Pulsate will also inform you about other key stats for each variant such as direct and combined open rates. For more information on Campaign Stats, see the Reports section below.
Just like Conversion Goals, A/B split tests also work extremely well alongside a control group.
Campaign Review and Sending
When you have finished configuring each variant and set up the A/B split test, press Next and go through the rest of the campaign steps including Targeting and Delivery to choose which users should be sent the campaign and when they should receive it. When all the required campaign options have been selected, you will be taken to the Review screen. Here you can review all the settings and content for the campaign, including the variants that were set up.
A/B Split Test Review
Just like on the push notification screen, you can see the different variants on the Review screen and check what content you added for each variant. If you wish to change any of the variants, press the Edit button and you will be taken to the push notification screen to edit it. When you have finished reviewing the campaign settings and content, press Send Campaign and confirm that you wish to send the campaign.
Pulsate will then take care of delivering each variant to the correct number of eligible recipients. As we usually do not know how many people would be sent a campaign until the campaign is complete, the process of selecting which users receive each variant is completely random. For this reason, there may be some differences between the number of recipients that are sent each variant as the campaign may end before the number of users across all variants can be equalized.
A/B Test Results
As soon as campaigns are sent to users, Pulsate will commence collecting stats for each variant. To access this data, click on View Report on the campaign list screen or navigate to Campaign Stats and click on the individual campaign that you want to review. You would be able to view the following metrics for all the variants:
- Sent
- Total Opens
- Unique Opens
You can switch between the stats of variants by clicking on the corresponding buttons (for example, Push A or Push B or Push C) near the mobile preview.
AB Graph