CONTACT

OPTIMIZATION

How to Measure the Impact of an A/B Testing Program

Mat Vermilyer Headshot

Mat Vermilyer

Director, Data & Analytics

GET MORE INSIGHTS

OPTIMIZATION

How to Measure the Impact of an A/B Testing Program

Mat Vermilyer Headshot

Mat Vermilyer

Director, Data & Analytics

GET MORE INSIGHTS
Confused young woman holding an abacus and scratching her chin
Confused young woman holding an abacus and scratching her chin

While A/B testing remains one of the best tactics to gain customer insights, improve user experience, increase conversions, and lower acquisition costs, being able to quantify the value of your overall A/B testing program can often be challenging.

Every A/B test has a specific control and experiment. When you have a winning scenario, it’s easy to know the incremental value that has been gained during the testing experiment. The challenge arises, however, once you roll out a winning experiment at 100%. Is it fair to assume that it will garner the same incremental value for an infinite amount of time? Probably not. Additionally, most testing programs run more than one test at a time. What happens when some experiments are not winners or there are gaps of time between tests? Now, you can see why it becomes a lot more difficult to quantify the total impact of your overall testing program.

8 Steps to Measure the Impact of an A/B Testing Program

Having conducted tens of thousands of A/B testing programs for clients, PeakActiivty developed the following 8-step process to calculate the overall impact or value associated with your A/B testing program.

To begin, you must have already pulled basic data from your test that will serve as the basis for the formulas moving forward. For the purposes of this tutorial, we’ve provided you with sample data that will be used throughout the examples below.

Control Treatment:

 Number of sessions = 62,000

 Revenue generated = $2,081,976

Test Treatment:

 Number of sessions = 62,754

 Revenue generated = $2,181,976

Step 1: Calculate Revenue Per Session

First, you need to calculate the revenue per session, but you won’t be able to calculate that without knowing the number of sessions and revenue generated during your test. Once equipped with this data, you can begin step 1.

Formula:

Divide revenue per treatment by number of sessions per treatment.

Example:

 Control Treatment: $2,081,976 ÷ 62,000 = $33.58 revenue per session

 Test Treatment: $2,181,976 ÷ 62,754 = $34.77 revenue per session

Step 2: Calculate Incremental Lift in Revenue Per Session

This step will tell you the incremental revenue gained per session by running the winning experiment.

Formula:

Subtract revenue per session of the control from the test treatment. Then, divide that number by the revenue per session of test treatment and multiply the answer by 100.

Example:

 $34.77 – $33.58 = $1.19 incremental revenue per session ($)

 $1.19 ÷ $34.77 x 100 = 3.54% incremental revenue per session (%)

Step 3: Calculate the Cost of Running the Test

If the sessions that viewed the control treatment had received the test treatment instead, we would expect that those sessions would also generate 3.54% in incremental revenue. However, since we ran a split 50/50 test, 50% of sessions did not garner incremental revenue. Therefore, there was a cost to run the test.

Formula:

Multiply revenue generated for control treatment by the incremental revenue per session (%).

Example:

 Control Treatment: $2,081,976 x 3.54% = $73,783 cost of test

Step 4: Identify a Multiplier for Test Distribution

A multiplier is needed to estimate the value of the winning experiment once rolled out to 100% of traffic. Use the following table to identify your multiplier based on how you’ve split your traffic in your test.

Chart laying out the test distributions and multipliers

Since our A/B ran at a 50/50 split, we will use a 2.00 multiplier in the remaining calculation.

Step 5: Calculate the Value Gained If Rolled Out 100%

Now that we know the cost of running the test and the multiplier, we can calculate the value of the winning experience if it were to be rolled out to 100% of traffic.

Formula:

Multiply the cost of running the winning test by multiplier.

Example:

 Test Treatment: $73,783 ✕ 2.00 = $147,566 value of test at 100%

Step 6: Calculate Value Gained From Testing Period

We now know how much it cost to run this test and the cost to be gained if it were to be rolled out at 100% of traffic. We can, therefore, now calculate the value gained by the winning test only during the testing period.

Formula:

Subtract the cost of running the test from the value gained if rolled out by 100%.

Example:

 $147,566 – $73,783 = $73,783 value of testing during testing period

Step 7: Forecast Incremental Revenue Gain

Once a treatment wins and is rolled out at 100%, you can assume that treatment will stay in place and continue to gain incremental revenue for a set period of time. Companies may vary what period of time they use to forecast revenue, but we typically use 120 days as a general benchmark, as incrementality tends to decline after that for a variety of reasons, including seasonality, customer behavior, other A/B tests, and promotional periods.

Formula:

Divide the value of the test at 100% by the number of days that the test ran. Then, multiply the daily gain by the forecasted number of days.

Example:

 $147,566 ÷ 35 days = $4,216 daily gain

 $4,216 X 120 days = $505,940 forecast incremental revenue gain

Step 8: Calculate Overall Program Value

Your program will likely run multiple tests, and your hypothesis will likely win in some experiments and lose in others. That’s ok. However, to calculate the total value of your A/B testing program, you need to take it all into account.

To do so, you’ll need to complete the above steps for every step you run, winners and losers. You’ll then take that total and subtract the operational cost of running said program. We typically estimate approximately $10K to run a program. Although it will vary depending on factors like staffing, overhead, time period, it can be used as a general benchmark if you don’t already know your own internal operational cost.

Formula:

Add all winning and losing tests together. Then, subtract your operational cost (or $10K benchmark) from the total cost. The new number is the overall value of your A/B testing program.

Investing in Testing

As you begin to run more tests you’ll want to make sure that your testing program is profitable. Leveraging the above formulas will ensure that you are running impactful tests with positive value gains. While not all tests will have revenue per session (RPS) as a primary metric, a similar attribution model can still be used. Remember, no matter what your actual metric for success is, it has to be measured in order to be evaluated.

Want even more educational content?

Subscribe below to get timely content delivered to your inbox,
or fill out the form below to speak to a Peakster about your next project.

SUBSCRIBE

DIGITAL MARKETING

Top 10 Photos of Developers Staring At Screens

READ MORE

DIGITAL STRATEGY

5 Ways Retailers Can Make Influencer Marketing More Influential

READ MORE

ECOMMERCE

How Many Emails is Too Many?

READ MORE

Speak With a Peakster

Interested in joining our team?