How to Measure the Impact of an A/B Testing Program


How to Measure the Impact of an A/B Testing Program

A/B testing is an indispensable tool for gaining deep customer insights, enhancing user experiences, boosting conversions, and trimming acquisition costs.

While its benefits are clear, determining the holistic value of your A/B testing program can be a challenging endeavor. In this comprehensive guide, we'll walk you through an 8-step process for accurately measuring the impact of your A/B testing program.

Step 1: Calculate Revenue Per Session

The first step in assessing your A/B testing program's impact is calculating the revenue generated per session. To do this, divide the total revenue by the number of sessions for each treatment group. This figure lays the foundation for subsequent calculations.


 Control Treatment: $2,081,976 ÷ 62,000 = $33.58 revenue per session

 Test Treatment: $2,181,976 ÷ 62,754 = $34.77 revenue per session

Step 2: Calculate Incremental Lift in Revenue Per Session

The incremental lift in revenue per session reveals the additional revenue gained by running the winning experiment. It's a crucial metric for understanding the direct impact of your A/B testing efforts.

To calculate this, subtract revenue per session of the control from the test treatment. Then, divide that number by the revenue per session of test treatment and multiply the answer by 100.


 $34.77 – $33.58 = $1.19 incremental revenue per session ($)

 $1.19 ÷ $34.77 x 100 = 3.54% incremental revenue per session (%)

Step 3: Calculate the Cost of Running the Test

Running A/B tests isn't cost-free. If the sessions that viewed the control treatment had received the test treatment instead, we would expect that those sessions would also generate 3.54% in incremental revenue. However, since we ran a split 50/50 test, 50% of sessions did not garner incremental revenue. Therefore, there was a cost to run the test.

To estimate the cost of running the test, multiply the total revenue generated by the control treatment by the incremental revenue per session.


 Control Treatment: $2,081,976 x 3.54% = $73,783 cost of test

Step 4: Identify a Multiplier for Test Distribution

The multiplier accounts for how your traffic was divided during the A/B test. It's used to estimate the value of the winning experiment when rolled out to 100% of your traffic. Use the following table to identify your multiplier based on how you’ve split your traffic in your test.

Chart laying out the test distributions and multipliers

Since our A/B ran at a 50/50 split, we will use a 2.00 multiplier in the remaining calculation.

Step 5: Calculate the Value Gained If Rolled Out 100%

With the cost of running the test and the multiplier in hand, you can estimate the value of the winning experiment if it were applied to 100% of your traffic. Multiply the test cost by the multiplier.


 Test Treatment: $73,783 ✕ 2.00 = $147,566 value of test at 100%

Step 6: Calculate Value Gained From Testing Period

Now, let's determine the value gained from running the winning test solely during the testing period. Subtract the cost of running the test from the estimated value if rolled out to 100% of your traffic.


 $147,566 – $73,783 = $73,783 value of testing during testing period

Step 7: Forecast Incremental Revenue Gain

Assuming that the winning treatment is applied to 100% of your traffic, you can predict how much incremental revenue you'll gain over a specific period, usually around 120 days. To do this, divide the value of the test at 100% by the number of days it was tested, and then multiply the daily gain by your forecasted number of days.


 $147,566 ÷ 35 days = $4,216 daily gain

 $4,216 X 120 days = $505,940 forecast incremental revenue gain

Step 8: Calculate Overall Program Value

Your A/B testing program will likely include multiple tests, some successful and others not. To calculate the total value of your A/B testing program, you'll need to apply the above steps to all tests, both winners and losers. Add the values together and subtract your operational cost (or use a $10K benchmark) to find the overall value of your A/B testing program.

This comprehensive approach allows you to make informed decisions about your testing program's profitability and determine which tests are generating positive value gains. Regardless of the metric you use to measure success, this method provides an essential framework for evaluating your testing efforts.

Investing in Testing

Ready to take your A/B testing and Conversion Rate Optimization efforts to the next level? Look no further than PeakActivity, a leader in the field with a proven track record of helping businesses maximize their online potential. Our team has conducted tens of thousands of A/B testing programs for clients, and we're here to provide you with the expertise and guidance you need. Whether you're looking to boost conversions, enhance user experiences, or reduce acquisition costs, we have the tools and experience to drive results. Don't leave your online performance to chance – partner with PeakActivity and unlock the full potential of your digital presence.


Top 10 Photos of Developers Staring At Screens



5 Ways Retailers Can Make Influencer Marketing More Influential



How Many Emails is Too Many?