{"_id":"58e77dd39df3dc2d000d5e0f","githubsync":"","__v":0,"version":{"_id":"576a656c6f15260e001f8938","__v":3,"project":"54aa7f773b56130b0056c86e","createdAt":"2016-06-22T10:16:12.661Z","releaseDate":"2016-06-22T10:16:12.661Z","categories":["576a656c6f15260e001f8939","576a656c6f15260e001f893a","576a656c6f15260e001f893b","576a656c6f15260e001f893c","576a656c6f15260e001f893d","576a656c6f15260e001f893e","576a656c6f15260e001f893f","5797787ce2d8f40e0052b4a6","5b041bc0769904000374c645"],"is_deprecated":false,"is_hidden":false,"is_beta":false,"is_stable":false,"codename":"For Marketers","version_clean":"4.0.0","version":"4.0"},"category":{"_id":"5797787ce2d8f40e0052b4a6","version":"576a656c6f15260e001f8938","project":"54aa7f773b56130b0056c86e","__v":0,"sync":{"url":"","isSync":false},"reference":false,"createdAt":"2016-07-26T14:49:32.231Z","from_sync":false,"order":2,"slug":"pulsate-campaigns","title":"Pulsate Campaigns"},"project":"54aa7f773b56130b0056c86e","user":"577cf7765fd4de0e00cc3d50","parentDoc":null,"updates":[],"next":{"pages":[],"description":""},"createdAt":"2017-04-07T11:53:55.871Z","link_external":false,"link_url":"","sync_unique":"","hidden":false,"api":{"results":{"codes":[]},"settings":"","auth":"required","params":[],"url":""},"isReference":false,"order":9,"body":"[block:callout]\n{\n  \"type\": \"success\",\n  \"title\": \"API Documentation\",\n  \"body\": \"Please click [here](https://pulsate.readme.io/v2.0/docs/campaigns-create-one) for our API Documentation on A/B Split Testing.\"\n}\n[/block]\nA/B Split testing allows you to send multiple variants of the same push notification to different groups of users so that you can test which variant performs the best. An A/B split test can be be setup for any campaign that includes a push notification. This includes both card based campaigns and push notification campaigns that bring the user directly to a section of an app or a URL.\n\n#Setting Up an A/B Split Test\n\nTo set up an A/B split test, start a new campaign and open the push notification screen. On this screen you will see a new Variants bar underneath the top navbar. \n\n\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/ab3e717-Screenshot_2017-04-07_14.22.54.png\",\n        \"Screenshot 2017-04-07 14.22.54.png\",\n        2370,\n        1070,\n        \"#346779\"\n      ],\n      \"caption\": \"Variants\"\n    }\n  ]\n}\n[/block]\n By default, the first push notification that you see is Variant A and this is denoted by a purple colour. If you wish to send a campaign with only the original variant, do not add any additional variants. To add a new variant, click the Add Button and choose whether you would like to:\n\n- Duplicate the original variant\n- Add a blank variant\n- Add a control group\n\nIf you choose to duplicate the original variant all of the text, images, buttons and settings that you have entered into the original variant (Variant A) will be duplicated in the new variant. This is useful if you just want to make an incremental change to each variant (such as adding a different emoji) and not make each variant completely independent of each other. \n\n\n\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/2965389-Duplicate_variant.gif\",\n        \"Duplicate variant.gif\",\n        2370,\n        1108,\n        \"#f1f8f8\"\n      ],\n      \"caption\": \"Duplicate Variant\"\n    }\n  ]\n}\n[/block]\nHowever, you can also choose to add a blank variant which will add a new variant that is completely blank and does not inherit anything from the original variant. \n\n\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/e7600aa-Blank_Variant.gif\",\n        \"Blank Variant.gif\",\n        2370,\n        1108,\n        \"#f8f9f8\"\n      ],\n      \"caption\": \"Blank Variant\"\n    }\n  ]\n}\n[/block]\nEach variant is assigned a unique letter from A-I and a unique color so it is easy to differentiate between them. You can add a maximum of nine variants and one control group for each campaign. Each variant can be completely different or each can contain a single modification. However, to maintain statistical relevance, it is recommended that the changes that you introduce to each variant are as minimal as possible so that it is easy to pinpoint which change caused each variant to perform better or worse than the others. \n\n\nTo remove any of the variants that have been setup, hover on that variant and click the X button to remove it. Please note that if a variant is deleted, all other variants will be updated to ensure that the order of the variants always stays in sequence from A-I. \n\n\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/e00bea7-Delete_Variant.gif\",\n        \"Delete Variant.gif\",\n        2370,\n        1108,\n        \"#f1f8f8\"\n      ],\n      \"caption\": \"Delete a Variant\"\n    }\n  ]\n}\n[/block]\n\n[block:callout]\n{\n  \"type\": \"warning\",\n  \"title\": \"Validation\",\n  \"body\": \"Please note that to progress from the push notification screen, it is required that the push notification details for all variants are filled in. At a minimum, each variant must include notification text and a notification destination. Other fields and details may be required depending on what push notification options are selected.\"\n}\n[/block]\n\n\n\n#A/B Settings\n\nWhen you click A/B Settings you will be shown a new modal that includes a bar that lets you set the amount of eligible users who should receive the campaign. By default this bar is set to 100% but it is possible to drag the bar to any value if you wish to send the campaign to just a subset of users. \n\n\n\n\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/23d59d1-Set_Audience.gif\",\n        \"Set Audience.gif\",\n        2370,\n        1108,\n        \"#f1f8f8\"\n      ],\n      \"caption\": \"Choose Recipients\"\n    }\n  ]\n}\n[/block]\nAs you move the scroll bar to choose the percentage of total users who should be sent the campaign, Pulsate will automatically adjust the size of each variant so that every variant is sent to the same percentage of users. \n[block:callout]\n{\n  \"type\": \"danger\",\n  \"title\": \"Statistical Relevance\",\n  \"body\": \"Please make sure that you send the campaign to enough users so that the results are statistically relevant. If you would like some help making sure that your A/B split test campaigns are statistically relevant, please contact your Customer Success representative.\"\n}\n[/block]\n#Choosing a Conversion Goal \n\nThe primary conversion goal that you choose for each campaign will be the metric that Pulsate uses to measure the performance of each variant and decide the winning variant. When you click on A/B Settings you will be able to view the primary conversion goal that you setup for the campaign and edit that goal if required. For more information on setting up Conversion Goals, click [here](doc:campaign-goals) \n\nHowever, even though the primary conversion goal will act as the primary deciding factor in choosing what variant is crowned as the winning variant, Pulsate will also inform you about other key stats for each variant such as direct and combined open rates. For more information on Campaign Stats, see the Reports section below. \n\nJust like Conversion Goals, A/B split tests also work extremely well alongside a control group.\n\n#Campaign Review and Sending\n\nWhen you have finished configuring each variant and setting up the A/B split test, press Next and go through the rest of the campaign steps including Targeting and Delivery to choose which users should be sent the campaign and when they should receive it. When all of the required campaign options have been selected, you will be taken to the Review screen. Here you can review all of the settings and content for the campaign, including the AB Split test and the variants that were setup. \n\n\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/eb19853-AB_Campaign_-_Review_Step.png\",\n        \"AB Campaign - Review Step.png\",\n        1080,\n        1170,\n        \"#f1f8f8\"\n      ],\n      \"caption\": \"A/B Split Test Review\"\n    }\n  ]\n}\n[/block]\nJust like on the push notification screen, you can scroll through the different variants on the Review screen and check what content you added for each variant. If you wish to change any of the variants, press the Edit variant and you will be taken to the push notification screen to edit it. When you have finished reviewing the campaign settings and content, press Send Campaign and confirm that you wish to send the campaign. \n\nPulsate will then take care of delivering each variant to the correct amount of eligible recipients. As we usually do not know how many people should be sent a campaign until the campaign is complete, the process of selecting which users receive each variant is completely random. For this reason, there may be some differences between the amount of recipients that are sent each variant as the campaign may end before the amount of users across all variants can be equalized.\n\n#A/B Test Results\n\nAs soon as the campaign begins sending to users, Pulsate will commence collecting stats for each variant. To access this data, click on View Report on the campaign list screen or go to Campaign Stats and click on the individual campaign that you want to review. The first thing you will see on the campaign review screen is the campaign results graph. This graph shows you the performance of different variants based on a number of different criteria including:\n\n- Total Sends\n- Unique Sends\n- Total Notification Opens\n- Unique Notification Opens\n- Total Notification + Card Opens\n- Unique Notification + Card Opens\n- Primary Conversions\n- Secondary Conversions\n\nYou can decide to include or exclude any variants that were sent for the campaign by ticking on or off the checkbox beside the variant name. And you can filter what criteria the graph should display by clicking the dropdown menu and selecting one of the above options. The color of each graph line is representative of the original colour of the variant when it was created so you can easily see how each variant is performing. \n\n\n\n\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/c2af0c8-AB_Graph.gif\",\n        \"AB Graph.gif\",\n        2382,\n        1309,\n        \"#2f68c6\"\n      ],\n      \"caption\": \"AB Graph\"\n    }\n  ]\n}\n[/block]\nIf you hover over any points of the graph lines, a popup will give you the total number of users who have been recorded for each action across each variant for that date. \n\n\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/76cc523-Screen_Shot_2017-06-13_at_09.46.09.png\",\n        \"Screen Shot 2017-06-13 at 09.46.09.png\",\n        1769,\n        673,\n        \"#54a1b8\"\n      ]\n    }\n  ]\n}\n[/block]\nIf you scroll down the page from the graph, you will come to the A/B split test result table. This table gives you all the info you need for each variant including: \n\n- Variant\n- Total Sends\n- Unique Sends\n- Notification Open Rate\n- Notification + Card Open Rate\n- Primary Conversion Rate\n\nThis table allows you to quickly view the key statistics for each variant and to get a picture of the relative success of each variant. To help identify which variant has been most succsesful, Pulsate will highlight the winning variant (based on the Primary Conversion Rate) and add a crown icon to that winning variant. \n\n\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/559539b-Screenshot_2017-04-07_16.44.22.png\",\n        \"Screenshot 2017-04-07 16.44.22.png\",\n        2336,\n        535,\n        \"#e8f8f8\"\n      ],\n      \"caption\": \"A/B Split Test Result Table\"\n    }\n  ]\n}\n[/block]\nThese results will update in real-time so it is a good idea to keep an eye on this table throughout the lifetime of every campaign that includes variants. if you would like to check the overall campaign stats for all of the combined variants, scroll down and view the overall campaign stats.","excerpt":"","slug":"ab-split-testing","type":"basic","title":"A/B Split Testing"}
[block:callout] { "type": "success", "title": "API Documentation", "body": "Please click [here](https://pulsate.readme.io/v2.0/docs/campaigns-create-one) for our API Documentation on A/B Split Testing." } [/block] A/B Split testing allows you to send multiple variants of the same push notification to different groups of users so that you can test which variant performs the best. An A/B split test can be be setup for any campaign that includes a push notification. This includes both card based campaigns and push notification campaigns that bring the user directly to a section of an app or a URL. #Setting Up an A/B Split Test To set up an A/B split test, start a new campaign and open the push notification screen. On this screen you will see a new Variants bar underneath the top navbar. [block:image] { "images": [ { "image": [ "https://files.readme.io/ab3e717-Screenshot_2017-04-07_14.22.54.png", "Screenshot 2017-04-07 14.22.54.png", 2370, 1070, "#346779" ], "caption": "Variants" } ] } [/block] By default, the first push notification that you see is Variant A and this is denoted by a purple colour. If you wish to send a campaign with only the original variant, do not add any additional variants. To add a new variant, click the Add Button and choose whether you would like to: - Duplicate the original variant - Add a blank variant - Add a control group If you choose to duplicate the original variant all of the text, images, buttons and settings that you have entered into the original variant (Variant A) will be duplicated in the new variant. This is useful if you just want to make an incremental change to each variant (such as adding a different emoji) and not make each variant completely independent of each other. [block:image] { "images": [ { "image": [ "https://files.readme.io/2965389-Duplicate_variant.gif", "Duplicate variant.gif", 2370, 1108, "#f1f8f8" ], "caption": "Duplicate Variant" } ] } [/block] However, you can also choose to add a blank variant which will add a new variant that is completely blank and does not inherit anything from the original variant. [block:image] { "images": [ { "image": [ "https://files.readme.io/e7600aa-Blank_Variant.gif", "Blank Variant.gif", 2370, 1108, "#f8f9f8" ], "caption": "Blank Variant" } ] } [/block] Each variant is assigned a unique letter from A-I and a unique color so it is easy to differentiate between them. You can add a maximum of nine variants and one control group for each campaign. Each variant can be completely different or each can contain a single modification. However, to maintain statistical relevance, it is recommended that the changes that you introduce to each variant are as minimal as possible so that it is easy to pinpoint which change caused each variant to perform better or worse than the others. To remove any of the variants that have been setup, hover on that variant and click the X button to remove it. Please note that if a variant is deleted, all other variants will be updated to ensure that the order of the variants always stays in sequence from A-I. [block:image] { "images": [ { "image": [ "https://files.readme.io/e00bea7-Delete_Variant.gif", "Delete Variant.gif", 2370, 1108, "#f1f8f8" ], "caption": "Delete a Variant" } ] } [/block] [block:callout] { "type": "warning", "title": "Validation", "body": "Please note that to progress from the push notification screen, it is required that the push notification details for all variants are filled in. At a minimum, each variant must include notification text and a notification destination. Other fields and details may be required depending on what push notification options are selected." } [/block] #A/B Settings When you click A/B Settings you will be shown a new modal that includes a bar that lets you set the amount of eligible users who should receive the campaign. By default this bar is set to 100% but it is possible to drag the bar to any value if you wish to send the campaign to just a subset of users. [block:image] { "images": [ { "image": [ "https://files.readme.io/23d59d1-Set_Audience.gif", "Set Audience.gif", 2370, 1108, "#f1f8f8" ], "caption": "Choose Recipients" } ] } [/block] As you move the scroll bar to choose the percentage of total users who should be sent the campaign, Pulsate will automatically adjust the size of each variant so that every variant is sent to the same percentage of users. [block:callout] { "type": "danger", "title": "Statistical Relevance", "body": "Please make sure that you send the campaign to enough users so that the results are statistically relevant. If you would like some help making sure that your A/B split test campaigns are statistically relevant, please contact your Customer Success representative." } [/block] #Choosing a Conversion Goal The primary conversion goal that you choose for each campaign will be the metric that Pulsate uses to measure the performance of each variant and decide the winning variant. When you click on A/B Settings you will be able to view the primary conversion goal that you setup for the campaign and edit that goal if required. For more information on setting up Conversion Goals, click [here](doc:campaign-goals) However, even though the primary conversion goal will act as the primary deciding factor in choosing what variant is crowned as the winning variant, Pulsate will also inform you about other key stats for each variant such as direct and combined open rates. For more information on Campaign Stats, see the Reports section below. Just like Conversion Goals, A/B split tests also work extremely well alongside a control group. #Campaign Review and Sending When you have finished configuring each variant and setting up the A/B split test, press Next and go through the rest of the campaign steps including Targeting and Delivery to choose which users should be sent the campaign and when they should receive it. When all of the required campaign options have been selected, you will be taken to the Review screen. Here you can review all of the settings and content for the campaign, including the AB Split test and the variants that were setup. [block:image] { "images": [ { "image": [ "https://files.readme.io/eb19853-AB_Campaign_-_Review_Step.png", "AB Campaign - Review Step.png", 1080, 1170, "#f1f8f8" ], "caption": "A/B Split Test Review" } ] } [/block] Just like on the push notification screen, you can scroll through the different variants on the Review screen and check what content you added for each variant. If you wish to change any of the variants, press the Edit variant and you will be taken to the push notification screen to edit it. When you have finished reviewing the campaign settings and content, press Send Campaign and confirm that you wish to send the campaign. Pulsate will then take care of delivering each variant to the correct amount of eligible recipients. As we usually do not know how many people should be sent a campaign until the campaign is complete, the process of selecting which users receive each variant is completely random. For this reason, there may be some differences between the amount of recipients that are sent each variant as the campaign may end before the amount of users across all variants can be equalized. #A/B Test Results As soon as the campaign begins sending to users, Pulsate will commence collecting stats for each variant. To access this data, click on View Report on the campaign list screen or go to Campaign Stats and click on the individual campaign that you want to review. The first thing you will see on the campaign review screen is the campaign results graph. This graph shows you the performance of different variants based on a number of different criteria including: - Total Sends - Unique Sends - Total Notification Opens - Unique Notification Opens - Total Notification + Card Opens - Unique Notification + Card Opens - Primary Conversions - Secondary Conversions You can decide to include or exclude any variants that were sent for the campaign by ticking on or off the checkbox beside the variant name. And you can filter what criteria the graph should display by clicking the dropdown menu and selecting one of the above options. The color of each graph line is representative of the original colour of the variant when it was created so you can easily see how each variant is performing. [block:image] { "images": [ { "image": [ "https://files.readme.io/c2af0c8-AB_Graph.gif", "AB Graph.gif", 2382, 1309, "#2f68c6" ], "caption": "AB Graph" } ] } [/block] If you hover over any points of the graph lines, a popup will give you the total number of users who have been recorded for each action across each variant for that date. [block:image] { "images": [ { "image": [ "https://files.readme.io/76cc523-Screen_Shot_2017-06-13_at_09.46.09.png", "Screen Shot 2017-06-13 at 09.46.09.png", 1769, 673, "#54a1b8" ] } ] } [/block] If you scroll down the page from the graph, you will come to the A/B split test result table. This table gives you all the info you need for each variant including: - Variant - Total Sends - Unique Sends - Notification Open Rate - Notification + Card Open Rate - Primary Conversion Rate This table allows you to quickly view the key statistics for each variant and to get a picture of the relative success of each variant. To help identify which variant has been most succsesful, Pulsate will highlight the winning variant (based on the Primary Conversion Rate) and add a crown icon to that winning variant. [block:image] { "images": [ { "image": [ "https://files.readme.io/559539b-Screenshot_2017-04-07_16.44.22.png", "Screenshot 2017-04-07 16.44.22.png", 2336, 535, "#e8f8f8" ], "caption": "A/B Split Test Result Table" } ] } [/block] These results will update in real-time so it is a good idea to keep an eye on this table throughout the lifetime of every campaign that includes variants. if you would like to check the overall campaign stats for all of the combined variants, scroll down and view the overall campaign stats.