A/B testing is a marketing technique in which you dispatch two variations viz. A and B of a single email campaign to a specific portion of your customer base in order to compare the two variations and thereby ascertain which one statistically performs better and drives business metrics such as click rate & open rate. The variation that performs better can then be deployed to the rest of your customer base in order to boost conversions. In case none of the variations performs well enough in terms of business metrics then this occurrence indicates that you need to make urgent amendments to your campaign’s design and to your marketing strategy.

NotifyVisitors makes it significantly convenient for you to deduce which of the two variations performed better by providing you with detailed metrics such as total deliveries, unique opens, unique clicks etc. for each variation.

How to review A/B test statistics for email campaigns

In order to review the A/B Test Statistics for an email campaign, you need to first create one with two different variations viz. A and B. Next, you need to dispatch these variations to distinct portions of your customer base in order to ultimately check which variation performed the best. Once you have successfully dispatched an email campaign with two variations, you can follow the process delineated below to review the A/B Test Statistics for the campaign.

1. Navigate to the NotifyVisitors Dashboard> Campaigns> Campaign. 

   Notice that your deployed email campaign gets listed under the section titled ‘Campaign’.

2. Click on the button with three dots, present adjacent to the campaign’s name.

3. Click on the ‘Analytics’ option. Upon doing so, a new frame would appear on-screen which contains all the details of the campaign. Click on the tab titled ‘A/B testing’ within this frame.

A/B test analytical data

This section describes the statistical results and analytical data available under the ‘A/B testing’ tab. The A/B Test Analytical Data is bifurcated into two sections viz. Test Period and Entire Sent. 

The Test Period section records and displays the statistics for that portion of the customer base on which the three variations of the email campaign were tested. On the other hand, the Entire Sent section records and displays the statistics for the entire customer base i.e. it displays the overall statistics for the portion of customers to which the winning variation was dispatched together with the portion of customers on which the three variations viz. A, B and Control of the campaign were tested.

Under both the sections there are three common pieces of information namely Winner, Winning Metric and Test Size. These three pieces of datum are described as follows:

Winner 

As the name suggests, this piece of information conveys the ultimate winner of the three variations viz. A, B and the parent variation of the email campaign. Kindly note that even the parent or the control variation of the email campaign can be a winner. 

The winner is ascertained first via comparing the Click Rates of the three variations and if the Click Rate for all three variations is equivalent then the winning variation is ascertained by comparing the Open Rates of all the variations. If one variation has a higher Click Rate than the remaining two variations then this particular variation becomes the Winner and in this case there is no need to compare the Open Rates of the variations.

Winning metric

This particular piece of information conveys the metric that was fundamental in ascertaining the winner amongst the three variations viz. A, B and the control variation of the email campaign. This metric could either be the Click Rate or the Open Rate. Additionally, the Winning Metric also mentions the percentage by which the winning variation was ahead of the other two variations.

Test size

This piece of information conveys the percentage of that portion of the entire customer base on which the three variations (A, B and Control) of the email campaign were tested.

Under both the sections i.e. Test Period and Entire Sent, there also exists a table which displays the total Delivers, Unique Opens and Unique Clicks for all the three variations of the email campaign. However, the data in the table will vary for both the sections since, the Test Period section displays data only for a portion of the customer base while the Entire Sent section displays data for the entire customer base.