A/B testing is the golden ticket for optimizing paid search campaigns. However, while there are several pitfalls in measuring A/B testing success, one tends to stand out, and that’s focusing solely on average metrics.
Segment analysis is vital to understanding how different segments of potential customers are responding to your A/B Tests. An improvement in CTR on the front end, may change the mix of potential customers coming in.
Here is an example from a recent audit. It was established that certain messaging was more important to the main customer persona and new ads were added into rotation. The new ads performed better in terms of CTR and everything looked stable. However, on further dissection of the data, when segmenting by the three different versions of the product, it became clear that the product mix was skewed. While sales of the more expensive product were increasing and the revenue looked good, the company was losing the sales from the price conscious customers that wanted to see the lower priced products in the ads and were generally buying the cheaper version. There was still a desire for more net new customers even at the lower entry. Revising the ads to include a price range along with the other winning assets, resulted in a higher CTR ‘hybrid’ ad, combined with a price extensions, resulted in a better mix of conversions.
It’s crucial to look beyond the average and examine how different segments are responding to you’re A/B tests. Are younger users reacting differently than older users? Is there a geographical variation in how the ad is received? Don’t just presume all is well, just because the average CTR or CVR increased.
While the averages to give you a quick snapshot, to truly understand the impact of your A/B tests in paid search, it’s important to look beyond the average metrics and determine how changes affect different segments of your audience. Only then can you truly optimize your paid search campaigns for all customers, not just the illusory “average” one.