A/B Testing: How to Improve Your Website Through Trial and Error (2023)

No matter what you use it for, you’ve probably spent a lot of time, money and energy creating and building your website— so make the most of it!

From driving email subscriber sign-ups to getting clicks on the elusive “Buy Now” button, every site owner wants to get to know how to make the magic happen. And there’s actually a pretty easy way to do it: A/B testing.

While it might sound technical, even newbies can get the hang of split testing. Here’s everything you need to know to get started — from running your first test to decoding the results.

But first, a little background.

What Is A/B Testing and How Does It Work?

Great question! A/B testing is the comparison of two variations of a single webpage, design, ad, or any other marketing media to determine which version converts more successfully.

It is conducted by randomly serving two versions of the same website to different users with just one change to the website (such as the color, size, or position of a call-to-action (CTA) button, for example) to see which performs better.The traffic is split 50-50 using the testing tools.

You’re right. It’s still a little confusing. So we reached out to a few online marketing experts to see if they could break it down for us. First up, Zack Gallinger, president of Talent Hero Media:

“A/B testing works by creating two versions of a landing page, called version A and version B,” Gallinger explains. “The landing pages are identical except for one item — anything from the amount of copy to the color of the Submitbutton on the form. You show each version to 50 percent of your site’s visitors and record which version performs better. The version that performs better in a statistically significant test is the winner and will be used as the official landing page in the future.”

Sounds promising, right? And it really is! In fact, there are plenty of reasons why it’s worth devoting time and resources to A/B testing.

“Instead of haphazardly making changes to your site because of a gut feel, A/B testing allows you to give your site’s visitors the ultimate decision on whether one design outperforms the next,” says Ayat Shukairy, co-founder of Invesp, a conversion optimization company.

“Ultimately, your customers should drive design, messaging, and engagement decisions on your site.”

Shukairy continues, “On the flipside, the testing will give you a deeper understanding of customer motivations and barriers so you can understand how to better cater your product, services, and site to meet their growing needs.”

Those insights often are hard to come by without testing. “The results can be conducted in real time while a website is already live, making it a perfect option to further optimize a website that has already launched,” says Zach Edelstein, Analytics Manager atMajux Marketing.

As a result, A/B testing can have a tremendous impact on your bottom line. “For example, imagine you have a sales funnel that gets 1,000 new leads per month,” Gallingersays. “The conversion rate of your funnel is 3 percent, and each new customer is worth $1,000 in profit. At this rate, you are making 1,000 * 3 percent * $1,000 = $30,000 per month.If you can figure out how to increase your conversion rate to 4 percent through A/B testing, you can increase your profits by $10,000 per month.”

It’s all about the Benjamins, baby, which is exactly why A/B testing has been rising in popularity the last few years.

What Can You Test to Improve Your Website?

Practically anything can be A/B tested, and it can be used in a variety of ways to improve your website.

“In broad terms, you can test the design, copy, or offer,” says Sid Bharath, a SAAS marketing consultant. “Design means the colors, layout, fonts, and visuals. Copy is the content on the page, like headers or product descriptions. Finally, the offer is what exactly you’re asking people to do, like signing up for an e-course, or buying a product.”

However, just because you can use A/B testing to measure almost anything doesn’t mean it should be used for everything. It’s crucial to have a strategy to employ it in ways to truly make the most of it.

“There was a popular saying when testing became popular in the early aughts to ‘always be testing,’” Shukairy says. “What people assumed was test a color and call to action on your site, and you will see massive improvements in conversions. This is not true at all.”

There is an art and science to testing, requiring:

• Regular analysis of your site to spot potential problem areas and investigate the why.
• Qualitative polling, surveys, and usability tests to understand better what customers are actually struggling with.

“Only after you’ve compiled issues can you start prioritizing them and deciding what solutions could be tested to improve upon the issues uncovered,” Shukairy says. “A hypothesis is necessary for the test because ultimately it is statistics that makes A/B testing viable.”

Where Should a Beginner Start A/B Testing?

First, you’ll want to come up with a game plan. “You can test almost anything on your site, from the length of your landing page to the color of your Buy Now button, but you want to prioritize those items that you think will make the biggest difference to your results,” Gallinger says. “A beginner should often start with one of the big four.”

1. Copy Length

Since the length of a website’s text can often have a dramatic impact on the conversion rate of a page, but it’s hard to know in advance if you need more or less copy to explain your idea until you show it to actual potential customers in an A/B test.

2. Headline

Theheadline is usually the first thing that people see, so it is critically important to use it effectively to grab their attention. Some sites will use A/B testing on 10 headlines or more before settling on a final option.

3. CTA

This small bit of text can often make a big difference. For example, a button that says “Buy Now” instead of “Buy” may create a sense of urgency that encourages people to click it or generate a sense of false urgency that makes people avoid it.

4. Images

Images convey important emotions, but it’s not always easy to know in advance which images work and which ones will fall flat with your audience. Much like headlines, you can test a wide variety of them before choosing the best one.

If you’d like to look into other areas of your site, spend some time getting to know your audience first. “By using Google Analytics, you should be able to highlight where the biggest drop-off points are. I would suggest starting your split testing there,” advises Andrew Wheller, digital marketing & SEO executive atPierre Cardin Stationery. “Reviewing your top exit pages is a nice starting point for highlighting your page, which can have the potential for the biggest uplift.”

Unclear how to find your drop-off points?

Bharath recommends starting with your existing data. For example, on an e-commerce website, the traffic flow might look like this: Home Page → Product Page → Cart → Checkout. “At each step, you’ll notice people dropping out and leaving the site,” he says. “Find where the largest drop-off is and run a test on that step to reduce it. Prioritize tests based on where the largest improvement can come from.”

How Long Should You Run an A/B Test?

There’s no magic time frame when it comes to testing. “The time to declare what we call ‘statistical significance’ for an A/B test depends on the traffic flowing through your site,” says Blake Puryear, product lead at Engine Insights.

“If you’re getting a trickle of traffic, let that A/B test run for a while. You need about a week’s worth of traffic through both sides of your test before you can start to infer anything,” Puryear adds.

“Even if a test is killing it after an afternoon, that might be an anomaly of traffic. It’s easy to try and make gut decisions or favor a variant because you like it better. This is where it’s really important to step back and let the data speak.”

For number nerds, there are tools that can help you determine your testing time. “An A/B test is complete when there is a statistically significant difference between the A and B versions of your landing page,” Gallinger says. “Most A/B testing software has a built-in calculator to determine when a statistically significant winner has been reached so you don’t have to do the math.”

If you want to use a separate calculator, this one from Kissmetricsworks well and is easy to understand.

How Do You Set up A/B Tests?

Depending on your skill level, you might be able to set up the test yourself. “A/B testing can seem quite daunting at first, but there is no reason you shouldn’t try it yourself,” Gallinger says. “Before you do, make sure you have a good understanding of the changes you want to test and the tools you are going to use. If you need help setting up an A/B test, contact a web developer to assist with the setup process.”

The next step is to use software to show each version of your site to 50 percent of the site visitors. “You can use tools like Optimizely, VWO, or Omniconvert to create A/B tests on your site,” Bharath says. “With their point-and-click tools, you can create variations of a page and change elements to test without requiring any coding knowledge.”

There’s also Google Content Experiments, which Puryear recommends. “If a client already has a WordPress plugin, I’ll adapt to their toolset and use that for the duration of the project,” he says. “The tools usually work in tandem with Google Analytics, so you’ll need to lean on that heavily.” For those with more complex needs, Gallingerrecommends Kissmetrics, Crazyeggor Optimizely.

WordPress Plugins for A/B testing can be helpful to beginners. “They give you some bounds to operate in,” Puryear says. “Think of these plugins as training wheels. One that I’ve seen several clients use is Nelio Native A/B Testing Service for WordPressand another that’s frequently used is Simple Page Tester.”

There are plenty of tools that won’t cost you a thing. “We use our own internal A/B testing tool which is currently free in beta,” says Shukairy. “In addition to being a robust testing tool, the platform contains many of the qualitative and quantitative tools needed to conduct the initial research phase prior to testing. But ultimately, you need an accurate tool that can help you split the traffic to two different designs and give you an output of information to signify the winner and other data points and metrics that may matter to you.”

How Do You Analyze Your Results?

“A/B testing tools usually always have an analysis or statistics dashboard,” Puryear says. “If your tool doesn’t, you’re using the wrong tool for sure. These dashboards should show you the performance of your A variant versus your B variant. Look at the spread of performance and which variant had the most occurrences of the event you’re testing for. Good tools will provide you with a confidence value based off of the amount of traffic that has run through the test.”

At the end of the experiment, look for a lift or possible decrease in conversion rate for the variant group. “It is important to consider that sometimes after an A/B test, the control group will remain the better option,” Edelsteinsays. “This is a common mistake as it can be tempting to favor changing the site due to an insignificant conversion increase.”And make sure you have enough data to give you a statistically significant result.

“Analyzing results is an important and often ignored part of testing,” Shukairy says. “First, you have to see whether your hypothesis was validated or not. Then, based on the results, verify if there are any other follow-up tests that can be considered.”

For instance, Shukairy tested the removal of a video and the placement of static image on a client’s subscription site. Once her team validated that the static image increased conversions and confirmed the group’s hypothesis to declutter the top of the page, they reintroduced the video in a different manner with the same static image and the results doubled.

“The test reveals a lot about the customer behavior patterns, likes and dislikes, of course, depending on what element you are actually testing,” she says. “This data can influence future tests.”

How Can You Archive Past Tests?

Even after you’ve moved on from a particular A/B test, you’ll want to save the data in case you want to refer to it in the future. “It’s a good idea to keep all test results and findings together,” Wheller suggests. “I personally use a separate folder on Google Drive; this is particularly good as you can easily share your findings with relevant team members and decision-makers.”

Bharathrecommends taking an extra step to create a learning document where you state your test description, hypothesis, before and after results, and what you learned from it.

Most A/B testing tools will save test results for future reference.“Any A/B testing platform worth its salt is going to have an archive or reference feature that you can go back and look at old tests,” Puryear says. “One of my favorite things to do in a new year is to look at old tests that really moved the needle and then run those tests again to see how they hold up six, eight, or12 months later. You might find the results to be really interesting as your customers or traffic profile can change over the course of a year.”

How Can You Really Make A/B Testing Work?

To make the most of A/B testing, be patient and put in the hours to do it right.“Don’t rush into testing haphazardly and end up making mistakes,” Shukairy says. “Take your time to create a strategy that is accurate and will not pollute data.Including A/B testing into your strategy can be an amazing asset to your company — if done right. It’s not impossible to acquire the knowledge and skills it takes to be a professional conversion optimizer and A/B tester; however, it will take a lot of reading, hard work, and attention to detail in order to maintain a successful testing program.”

Another thing newbies should keep in mind is not to bite off more than they can chew. “Start with small tests until you’re familiar with the statistical significance of your site and your traffic patterns,” Puryear says. “I’ve seen terrible things happen when someone jumps feet first into a huge test as their first stab at A/B testing. Optimize something small and become familiar with what you’re doing.”

Just like anything else, practice makes perfect. “Start by identifying which page you want to focus on first by looking at the conversion data,” Bharath says. “Then, make a hypothesis about what element to change to improve conversions. With the above tools, it becomes really easy to whip up a new test. The more tests you run, the faster you can improve your conversions and sales.”