The Value of A/B Testing
Before we get into the what/how/why of website A/B testing, let’s look at whether it’s even worth your time.
How much do you think you might be able to improve your conversion rate through A/B testing? Go ahead; pick a number. Make it conservative. How about just 4%? That means improving from (for example) 2.75% to 2.86%. That sounds doable. It also sounds pretty insignificant, right? But let’s see what the math tells us.
The example below makes a few assumptions, but you can plug in your own numbers at blend.travel/websitecalculator. For a client with 25k sessions every month, a conversion rate of 2.75%, and a $250 average order value, the 1-year value of a 4% conversion rate improvement is $82,500.
A/B testing isn’t the only way to improve your conversion rate, but it’s one of the tools in the toolbox. And the math tells us the bottom line: this is absolutely worth your time.
What Is Website A/B Testing?
You’re likely familiar with A/B testing for email subject lines or advertising. Website A/B testing is the same concept: You’re comparing multiple versions of a web page (or series of pages) to see which one performs better.
But to clarify: A/B testing is NOT trying version A for a couple of weeks and then trying version B for a couple of weeks, and comparing results. Too much can change in that time—such as seasonality, weather, or availability. You need a tool that compares both versions at the same time, typically by directing 50% of traffic to one version, and 50% to another. The tool will also ensure that each person sees the same version of the experiment every time they visit the website.
Real-Life A/B Testing Examples
There’s a lot to cover about how to run a test and how to set up the tool, but let’s look at some real-life examples.
I’m going to share scenarios and outcomes here, but the purpose is only to illustrate what kinds of tests are possible and how valuable an outcome can be to a client. We cannot extrapolate every outcome to be equally applicable to every client, so don’t take these outcomes as conclusive for your situation. You really need to run your own A/B tests.
Does Showing The Ticket Price On The Home Page Improve The Conversion Rate?
The Scenario: We advocate for up-front pricing on tourism sites, but we recognized that we had neglected to put pricing on the home page of a client website. We took the opportunity to run a test, by showing the price to half of users and hiding it from the other half (just for the home page; the pricing was already available on tour pages).
The Outcome: When pricing was shown on the home page, the conversion rate was 5.2% higher, with a 95.6% confidence level. We tested about 34,000 visitors. If this were to hold over the entire year (which is not guaranteed), the extra revenue for this client would be over $100k per year. Even if they saw just a fraction of this increase, the cost of the design tweak and the A/B test would be eclipsed by the extra revenue many times over.
Is It Better For Tickets To Be Shown From Lowest Cost to Highest Cost, or Vice Versa?
The Scenario: One client has several ticket options, ranging from basic admission to more expensive tickets with additional activities and benefits. We wanted to see whether the revenue per user was higher when we showed the tickets in order from lowest to highest cost, or from highest cost to lowest cost. The thought was that starting with the higher price might cause people to abandon the purchase and that showing them lower-cost options first would keep them moving through the purchase process.
The Outcome: On mobile devices, both the conversion rate and the revenue per user improved by 9.2% when the tickets were ordered from lowest cost to highest cost. On desktop devices, the improvement was smaller, with a lower confidence level. Interestingly, the results differ at different times of the year, so we’re looking for the optimal solution by month or by season.
Does Moving The Video Higher On Tour Page Improve Results?
The Scenario: We had reason to believe that moving a tour video higher on the tour page might improve the conversion rate, so we gave that a try.
The Outcome: We saw mixed results here. On one tour, moving the video to the top of the page resulted in a 6% improvement in the desktop conversion rate (25,000 visitors; 94.5% confidence), but there was no meaningful change in the mobile conversion rate. On another tour, the improvement was very small and not statistically significant. In total, the net result is positive, leading us to move toward a permanent change in the template.
Is The New Design Responsible For The Lower Conversion Rate?
The Scenario: We redesigned a brand and website, but were surprised to see lower conversion rates after launch. We hypothesized that the conversion rate decline may have been due to a new technique we tried, in an attempt to improve the user experience. So we ran an A/B test to see whether removing that new technique—and reverting to a more traditional design for that part of the site—would yield better results.
The Outcome: The traditional design was 30% more effective (with 98% confidence) for desktop devices, leading us to make a permanent change. After that, the site has been converting a minimum of 5% better than the old site, leading to year-over-year revenue gains of an estimated $80k+ for a partial year.
How Does A/B Testing Work?
The examples above illustrate the process, but here are some specifics to help you get started:
1) Determine What To Try
- Identify your goal. Often, this is improving conversion rate. But it can be other things too, such as increasing average order value or increasing the number of leads.
- Choose something to test in an attempt to reach that goal. This could be headings, images, calls-to-action, how pricing is displayed, the order of sections on a page, etc. (Technically, this step is called creating your hypothesis.) With most tools, you can choose from a few different test types:
- A/B: Make changes on a single page.
- Split URL: Test 2 different pages altogether.
- Multivariate: Test multiple components to see which combination gives the best results.
2) Set Up And Deploy The Test
(You’ll need an A/B testing tool already selected and configured; we cover that below.)
- In the test, select the goal that you identified, such as the purchase conversion rate.
- Determine where the test is going to take place. This might be your home page or a tour page, for example.
- Determine who you’re targeting. People who have visited a certain number of pages? Just mobile users? Or everyone?
- Make the changes on your site by using the tool. This allows you to do things that you can’t normally/easily accomplish on your site.
- Determine how traffic will be split (often 50/50).
3) Analyze
If you set everything up correctly, the tool will handle a lot of the analysis for you, indicating the performance of each version, and telling you whether the result is statistically significant. A couple of tips:
- Avoid the temptation to get too excited about early results, or to declare a winner too early. You’ll often want several thousand visitors tested, over a couple of weeks, before declaring a winner. Follow the tool’s advice on this. It is common to see results change a lot over the first several days; making a decision too early could be very detrimental.
- Segment. Look at mobile results vs. desktop results for example. It’s common to see very different results on different device types.
Selecting and Setting Up an A/B Testing Tool
Selecting a Service
Google Optimize was our go-to solution. It was free and easy to use. Sadly, Google shut it down in 2023. Some tools for small to mid-size tour, activity, and attraction companies include the following. Some of these seem expensive, but remember to run the calculations to see just how much money is being left on the table by not testing. Also, if you have a marketing agency, you can often save money if they allow you to use their plan.
- Convert Experiences. This is the tool we landed on for now. It starts at $299/mo.
- Zoho Pagesense. This is among the most affordable options we’ve seen, but we have not tested this solution yet. Pagesense is a “Conversion Optimization and Personalization Platform,” so it includes more than A/B testing.
- VWO (Visual Website Optimizer). A limited free plan is available, but prices escalate quickly. They seem to be focused more on enterprises than small businesses.
- AB Tasty. Pricing is not made available publicly.
Getting Started
- Install the code. Every platform has a piece of code you have to install on your site. Follow directions carefully about the code placement.
- Set up goal tracking. This can include importing Google Analytics goals such as making a purchase, but it can also include other things such as visiting a certain page, clicking a certain link, or submitting a form.
From here, you can follow the steps in How Does A/B Testing Work? > Set Up And Deploy The Test.
Things To Watch Out For
- Set a traffic limit or time limit. This will prevent you from using up all of your allocated traffic on one test, or possibly incurring overage charges.
- Preview the test variants. This is important; depending on the complexity of your A/B test, it can take some troubleshooting to get everything to work properly. Don’t assume everything’s working; test it.
- Testing may not fully work with 3rd party booking platforms, so do your research and understand the limitations.
About 3rd Party Booking Platforms
A/B testing tools don’t always work seamlessly with 3rd party booking platforms. With convert.com and Fareharbor for example, we aren’t yet sure whether it’s possible to perform A/B tests on mobile devices, when our selected goal is purchases. We need to do more testing and continue working with their support team on this. We have seen some success on desktop devices though, and even on mobile devices, you can use a backup goal—such as clicking the Book Now button.
Booking platforms that send events to the data layer—such as Ventrata and others—have no such limitations related to A/B testing.
A/B Testing Tools For Personalization
A/B testing tools aren’t only for seeing what works better, but also for deploying what works best, especially when you find that a certain personalization works. (Note: the examples we provide below are both dependent upon geotargeting, which isn’t supported in all plans for all platforms.)
Personalization Example 1: Geo-Targeted Offers
One Hawaiian client offers a discount to locals. We’re able to hide that offer from anyone browsing from any state other than Hawaii.
Another Hawaiian client offers a discount for booking at least 7 days in advance. That’s typically not useful for vacationers currently on the island. Rather than showing them that they missed an opportunity for a discount, we simply remove that when anyone’s browsing from Hawaii, allowing them to focus on only what’s relevant to them.
Personalization Example 2: Metro-specific ‘Home’ Pages
We’ve found that for clients within driving distance of metro areas, a customized home page for visitors from that metro area slightly improves the conversion rate. The A/B testing tool allows us to redirect people to their ‘personalized’ home page. For example, Lake Geneva Ziplines and Adventures is within driving distance of Chicago, and when someone visits the site from Chicago, they are redirected to https://lakegenevaadventures.com/chicago/. That page has a Chicago-specific headline, testimonials from Chicagoans, some recommended stops between Chicago and the destination, etc.
Technical Considerations
There are some technical issues to consider, such as:
- Impact on page speed. A/B testing tools are very aware of the potential impact on page load time, and they try to mitigate any performance issues; but it may be worth testing multiple tools if you’re watching your page speed closely. D
- Data center locations. For speed and compliance, you may need to select a data center in the EU (for example). You’ll need a tool that meets your requirements.
Can I Actually Do This Myself?
So is this a how-to article, or an article just explaining A/B testing and its value?
A little of both.
There are aspects of this that may require some assistance from web developers or technical marketers, depending on some factors, including your own technical skills. You could also take a hybrid approach: Have someone set up the tool for you, and then run some of your own tests from that point forward. Even then, selecting the right things to test, and going through the process of setting up and deploying a test, is not for everyone.
If you lack either the technical skill or the knowledge about conversion rate optimization (CRO), a technical marketer with CRO experience is the best person to run this for you. But don’t let that slow you down. Remember the value of A/B testing; this is a revenue-generating investment, and one that your competitors may be ignoring, making it potentially worthy of making your high-priority list right now.
