A/B Testing: How to Improve Your Website Through Trial and Error (2023)

No matter what you use it for, you’ve probably spent a lot of time, money and energy creating and building your website— so make the most of it!

From driving email subscriber sign-ups to getting clicks on the elusive “Buy Now” button, every site owner wants to get to know how to make the magic happen. And there’s actually a pretty easy way to do it: A/B testing.

While it might sound technical, even newbies can get the hang of split testing. Here’s everything you need to know to get started — from running your first test to decoding the results.

But first, a little background.

What Is A/B Testing and How Does It Work?

Great question! A/B testing is the comparison of two variations of a single webpage, design, ad, or any other marketing media to determine which version converts more successfully.

It is conducted by randomly serving two versions of the same website to different users with just one change to the website (such as the color, size, or position of a call-to-action (CTA) button, for example) to see which performs better.The traffic is split 50-50 using the testing tools.

You’re right. It’s still a little confusing. So we reached out to a few online marketing experts to see if they could break it down for us. First up, Zack Gallinger, president of Talent Hero Media:

“A/B testing works by creating two versions of a landing page, called version A and version B,” Gallinger explains. “The landing pages are identical except for one item — anything from the amount of copy to the color of the Submitbutton on the form. You show each version to 50 percent of your site’s visitors and record which version performs better. The version that performs better in a statistically significant test is the winner and will be used as the official landing page in the future.”

Sounds promising, right? And it really is! In fact, there are plenty of reasons why it’s worth devoting time and resources to A/B testing.

“Instead of haphazardly making changes to your site because of a gut feel, A/B testing allows you to give your site’s visitors the ultimate decision on whether one design outperforms the next,” says Ayat Shukairy, co-founder of Invesp, a conversion optimization company.

“Ultimately, your customers should drive design, messaging, and engagement decisions on your site.”

Shukairy continues, “On the flipside, the testing will give you a deeper understanding of customer motivations and barriers so you can understand how to better cater your product, services, and site to meet their growing needs.”

(Video) Avoid these 6 Common Errors in Marketing A/B Testing

Those insights often are hard to come by without testing. “The results can be conducted in real time while a website is already live, making it a perfect option to further optimize a website that has already launched,” says Zach Edelstein, Analytics Manager atMajux Marketing.

As a result, A/B testing can have a tremendous impact on your bottom line. “For example, imagine you have a sales funnel that gets 1,000 new leads per month,” Gallingersays. “The conversion rate of your funnel is 3 percent, and each new customer is worth $1,000 in profit. At this rate, you are making 1,000 * 3 percent * $1,000 = $30,000 per month.If you can figure out how to increase your conversion rate to 4 percent through A/B testing, you can increase your profits by $10,000 per month.”

It’s all about the Benjamins, baby, which is exactly why A/B testing has been rising in popularity the last few years.

A/B Testing Your Website?

Start out on the right foot. DreamHost makes sure your website is fast, secure, and always up so your visitors trust you. Plans start at $2.59/mo.

Choose Your Plan

A/B Testing: How to Improve Your Website Through Trial and Error (1)

What Can You Test to Improve Your Website?

A/B Testing: How to Improve Your Website Through Trial and Error (2)Practically anything can be A/B tested, and it can be used in a variety of ways to improve your website.

“In broad terms, you can test the design, copy, or offer,” says Sid Bharath, a SAAS marketing consultant. “Design means the colors, layout, fonts, and visuals. Copy is the content on the page, like headers or product descriptions. Finally, the offer is what exactly you’re asking people to do, like signing up for an e-course, or buying a product.”

(Video) Postman Beginner's Course - API Testing

However, just because you can use A/B testing to measure almost anything doesn’t mean it should be used for everything. It’s crucial to have a strategy to employ it in ways to truly make the most of it.

“There was a popular saying when testing became popular in the early aughts to ‘always be testing,’” Shukairy says. “What people assumed was test a color and call to action on your site, and you will see massive improvements in conversions. This is not true at all.”

There is an art and science to testing, requiring:

  • Regular analysis of your site to spot potential problem areas and investigate the why.
  • Qualitative polling, surveys, and usability tests to understand better what customers are actually struggling with.

“Only after you’ve compiled issues can you start prioritizing them and deciding what solutions could be tested to improve upon the issues uncovered,” Shukairy says. “A hypothesis is necessary for the test because ultimately it is statistics that makes A/B testing viable.”

Where Should a Beginner Start A/B Testing?

First, you’ll want to come up with a game plan. “You can test almost anything on your site, from the length of your landing page to the color of your Buy Now button, but you want to prioritize those items that you think will make the biggest difference to your results,” Gallinger says. “A beginner should often start with one of the big four.”

1. Copy Length

Since the length of a website’s text can often have a dramatic impact on the conversion rate of a page, but it’s hard to know in advance if you need more or less copy to explain your idea until you show it to actual potential customers in an A/B test.

2. Headline

Theheadline is usually the first thing that people see, so it is critically important to use it effectively to grab their attention. Some sites will use A/B testing on 10 headlines or more before settling on a final option.

3. CTA

This small bit of text can often make a big difference. For example, a button that says “Buy Now” instead of “Buy” may create a sense of urgency that encourages people to click it or generate a sense of false urgency that makes people avoid it.

4. Images

Images convey important emotions, but it’s not always easy to know in advance which images work and which ones will fall flat with your audience. Much like headlines, you can test a wide variety of them before choosing the best one.

If you’d like to look into other areas of your site, spend some time getting to know your audience first. “By using Google Analytics, you should be able to highlight where the biggest drop-off points are. I would suggest starting your split testing there,” advises Andrew Wheller, digital marketing & SEO executive atPierre Cardin Stationery. “Reviewing your top exit pages is a nice starting point for highlighting your page, which can have the potential for the biggest uplift.”

Unclear how to find your drop-off points?

Bharath recommends starting with your existing data. For example, on an e-commerce website, the traffic flow might look like this: Home Page → Product Page → Cart → Checkout. “At each step, you’ll notice people dropping out and leaving the site,” he says. “Find where the largest drop-off is and run a test on that step to reduce it. Prioritize tests based on where the largest improvement can come from.”

(Video) How to stress test a PC to find errors and crashes

How Long Should You Run an A/B Test?

There’s no magic time frame when it comes to testing. “The time to declare what we call ‘statistical significance’ for an A/B test depends on the traffic flowing through your site,” says Blake Puryear, product lead at Engine Insights.

“If you’re getting a trickle of traffic, let that A/B test run for a while. You need about a week’s worth of traffic through both sides of your test before you can start to infer anything,” Puryear adds.

“Even if a test is killing it after an afternoon, that might be an anomaly of traffic. It’s easy to try and make gut decisions or favor a variant because you like it better. This is where it’s really important to step back and let the data speak.”

For number nerds, there are tools that can help you determine your testing time. “An A/B test is complete when there is a statistically significant difference between the A and B versions of your landing page,” Gallinger says. “Most A/B testing software has a built-in calculator to determine when a statistically significant winner has been reached so you don’t have to do the math.”

If you want to use a separate calculator, this one from Kissmetricsworks well and is easy to understand.

How Do You Set up A/B Tests?

Depending on your skill level, you might be able to set up the test yourself. “A/B testing can seem quite daunting at first, but there is no reason you shouldn’t try it yourself,” Gallinger says. “Before you do, make sure you have a good understanding of the changes you want to test and the tools you are going to use. If you need help setting up an A/B test, contact a web developer to assist with the setup process.”

The next step is to use software to show each version of your site to 50 percent of the site visitors. “You can use tools like Optimizely, VWO, or Omniconvert to create A/B tests on your site,” Bharath says. “With their point-and-click tools, you can create variations of a page and change elements to test without requiring any coding knowledge.”

There’s also Google Content Experiments, which Puryear recommends. “If a client already has a WordPress plugin, I’ll adapt to their toolset and use that for the duration of the project,” he says. “The tools usually work in tandem with Google Analytics, so you’ll need to lean on that heavily.” For those with more complex needs, Gallingerrecommends Kissmetrics, Crazyeggor Optimizely.

WordPress Plugins for A/B testing can be helpful to beginners. “They give you some bounds to operate in,” Puryear says. “Think of these plugins as training wheels. One that I’ve seen several clients use is Nelio Native A/B Testing Service for WordPressand another that’s frequently used is Simple Page Tester.”

There are plenty of tools that won’t cost you a thing. “We use our own internal A/B testing tool which is currently free in beta,” says Shukairy. “In addition to being a robust testing tool, the platform contains many of the qualitative and quantitative tools needed to conduct the initial research phase prior to testing. But ultimately, you need an accurate tool that can help you split the traffic to two different designs and give you an output of information to signify the winner and other data points and metrics that may matter to you.”

How Do You Analyze Your Results?

A/B Testing: How to Improve Your Website Through Trial and Error (3)“A/B testing tools usually always have an analysis or statistics dashboard,” Puryear says. “If your tool doesn’t, you’re using the wrong tool for sure. These dashboards should show you the performance of your A variant versus your B variant. Look at the spread of performance and which variant had the most occurrences of the event you’re testing for. Good tools will provide you with a confidence value based off of the amount of traffic that has run through the test.”

At the end of the experiment, look for a lift or possible decrease in conversion rate for the variant group. “It is important to consider that sometimes after an A/B test, the control group will remain the better option,” Edelsteinsays. “This is a common mistake as it can be tempting to favor changing the site due to an insignificant conversion increase.”And make sure you have enough data to give you a statistically significant result.

(Video) Webinar: Is your SEO ready for Black Friday?

“Analyzing results is an important and often ignored part of testing,” Shukairy says. “First, you have to see whether your hypothesis was validated or not. Then, based on the results, verify if there are any other follow-up tests that can be considered.”

For instance, Shukairy tested the removal of a video and the placement of static image on a client’s subscription site. Once her team validated that the static image increased conversions and confirmed the group’s hypothesis to declutter the top of the page, they reintroduced the video in a different manner with the same static image and the results doubled.

“The test reveals a lot about the customer behavior patterns, likes and dislikes, of course, depending on what element you are actually testing,” she says. “This data can influence future tests.”

How Can You Archive Past Tests?

Even after you’ve moved on from a particular A/B test, you’ll want to save the data in case you want to refer to it in the future. “It’s a good idea to keep all test results and findings together,” Wheller suggests. “I personally use a separate folder on Google Drive; this is particularly good as you can easily share your findings with relevant team members and decision-makers.”

Bharathrecommends taking an extra step to create a learning document where you state your test description, hypothesis, before and after results, and what you learned from it.

Most A/B testing tools will save test results for future reference.“Any A/B testing platform worth its salt is going to have an archive or reference feature that you can go back and look at old tests,” Puryear says. “One of my favorite things to do in a new year is to look at old tests that really moved the needle and then run those tests again to see how they hold up six, eight, or12 months later. You might find the results to be really interesting as your customers or traffic profile can change over the course of a year.”

Get Content Delivered Straight to Your Inbox

Subscribe to our blog and receive great content just like this delivered straight to your inbox.

How Can You Really Make A/B Testing Work?

To make the most of A/B testing, be patient and put in the hours to do it right.“Don’t rush into testing haphazardly and end up making mistakes,” Shukairy says. “Take your time to create a strategy that is accurate and will not pollute data.Including A/B testing into your strategy can be an amazing asset to your company — if done right. It’s not impossible to acquire the knowledge and skills it takes to be a professional conversion optimizer and A/B tester; however, it will take a lot of reading, hard work, and attention to detail in order to maintain a successful testing program.”

Another thing newbies should keep in mind is not to bite off more than they can chew. “Start with small tests until you’re familiar with the statistical significance of your site and your traffic patterns,” Puryear says. “I’ve seen terrible things happen when someone jumps feet first into a huge test as their first stab at A/B testing. Optimize something small and become familiar with what you’re doing.”

Just like anything else, practice makes perfect. “Start by identifying which page you want to focus on first by looking at the conversion data,” Bharath says. “Then, make a hypothesis about what element to change to improve conversions. With the above tools, it becomes really easy to whip up a new test. The more tests you run, the faster you can improve your conversions and sales.”


How do you overcome AB testing problems? ›

5 Common A/B Testing Problems and How to Overcome Them
  1. Focus on changes with big impact.
  2. Test with the intention of improving SEO and user experience.
  3. Skip the multivariate testing until you have more traffic.
16 Sept 2019

How do I optimize for Google AB test? ›

To create an A/B test:
  1. Go to your Optimize Account (Main menu > Accounts).
  2. Select your container.
  3. Click Create experiment.
  4. Enter an experiment name (up to 255 characters).
  5. Enter an editor page URL (the web page you'd like to test).
  6. Click A/B test.
  7. Click Create.

How can I run an AB test on a low traffic site? ›

Some tactics for successful A/B testing on low-traffic sites include:
  1. Make big changes.
  2. Use fewer variations.
  3. Test high-funnel KPIs (micro-conversions)
  4. Use different statistical measurement methods. Use statistical significance as a “guide” Use different measurement calculations. Use confidence intervals.
16 Jan 2020

What is big problem with AB test? ›

The problem is that, because of randomization, it's possible that if you let the test run to its natural end, you might get a different result. The second mistake is looking at too many metrics.

Why do ab tests fail? ›

A common reason for poor A/B test results is because the idea (the hypothesis for what was tested) was not very good. This is because businesses often just guess at what to test, with no insights being used to create each idea. And without a good hypothesis, you will find it hard to learn if the test fails.

How do I optimize my website? ›

Guidelines to speed up your website
  1. Use a Content Delivery Network (CDN) ...
  2. Move your website to a better host. ...
  3. Optimize the size of images on your website. ...
  4. Reduce the number of plugins. ...
  5. Minimize the number of JavaScript and CSS files. ...
  6. Use website caching. ...
  7. Implement Gzip Compression. ...
  8. Database optimization in CMS.
30 Apr 2018

What are optimization tools? ›

The Optimization tool solves linear programming (LP), mixed-integer linear programming (MILP), and quadratic programming (QP) optimization problems using matrix, manual, and file input modes. This tool uses the R tool.

How do you optimize a test? ›

A typical test optimization routine can consist of the following stages:
  1. Minimizing the size of the used test suites.
  2. Establishing the smallest possible subset of test suites capable of reaching the required goals.
  3. Getting rid of excessive test cases.
  4. Analyzing the optimal test coverage criteria.
3 Feb 2022

When should you not use an AB test? ›

4 reasons not to run a test
  • Don't A/B test when: you don't yet have meaningful traffic. ...
  • Don't A/B test if: you can't safely spend the time. ...
  • Don't A/B test if: you don't yet have an informed hypothesis. ...
  • Don't A/B test if: there's low risk to taking action right away.
13 Dec 2015

What is traffic in AB testing? ›

It goes more or less without saying that when it comes to A/B testing, the more traffic the better. More traffic means it's faster and easier to reach statistical significance, get results, and make decisions.

Why should changes be tested with a small number of web visitors? ›

By running the test on a small percentage of your visitors, only that tiny segment of the visitors will see the errors in the new designs. That will give you the opportunity to fix these bugs before rolling out the test to 100% of your visitors.

What should I test my website? ›

Checklist: 13 Things to Test Before Your Website Launches
  • Spelling, Grammar, Punctuation. It's cumbersome, but necessary to go through the site and check for typos, grammar, and proper punctuation sitewide. ...
  • Web Forms. ...
  • Live URLs. ...
  • Title Tags/Meta Data. ...
  • Site speed. ...
  • Context. ...
  • Compatibility. ...
  • Map all old pages to new.
18 Jun 2014

What is AB testing example? ›

Here are some examples of what that would look like in an A/B Testing Hypothesis: Shortening contact submission forms to only contain required fields would increase the number of sign-ups. Changing the call-to-action text from “Download now” to “Download this free guide” would increase the number of downloads.

How do I validate my AB test results? ›

Validating the results
  1. Check to make sure that this test was run correctly. Is the query that calculates lift and p-value correct? ...
  2. Check other metrics to make sure that this outsized result is not isolated to this one metric. ...
  3. Check that the data is correct. ...
  4. Make a final recommendation based on your conclusions.

What is one of the common mistakes when running a B tests? ›

Unfortunately, when it comes to A/B testing, stopping your test as soon as you see a statistical significant result is not just bad sportsmanship, but it also produces completely invalid results. Many tools encourage this behavior by allowing users to stop a test as soon as statistical significance has been hit.

Can you run multiple A B tests at the same time? ›

Running multiple A/B tests at the same time can theoretically lead to interferences that result in choosing an inferior combination of variants. Given that from two combinations of variants one has a stronger and opposite sign interaction than the other, it is guaranteed to happen.

What are some common pitfalls of AB testing? ›

Ten common A/B testing pitfalls and how to avoid them
  • Pitfall 1: Ignoring the effects of the significance level. ...
  • Pitfall 2: Declaring winners of multiple offer tests with no statistically significant difference. ...
  • Pitfall 3: Ignoring the effects of statistical power. ...
  • Pitfall 6: Stopping tests prematurely.

How many times should you run an AB test? ›

For you to get a representative sample and for your data to be accurate, experts recommend that you run your test for a minimum of one to two week.

Should we Ab test everything? ›

A/B Test Everything — But Start With The Big Things

Starting your A/B tests high-level will give you the biggest returns quickly. By incorporating each successful change into the next A/B test, you will compound on these benefits to your conversions even more.

Should you always test AB? ›

A/B testing is incredibly important, as it always ensures you are optimizing your content and using your resources in the most efficient and effective way possible, increasing your ROI and helping reach your brand's goals.

When should you stop doing AB tests? ›

Keep going until you reach 95-99% statistical significance. Make sure your sample size is large enough (at least 1,000 conversions). Don't stop running your test too soon. Aim for 1-2 weeks.

When should you not do an AB test? ›

4 reasons not to run a test
  • Don't A/B test when: you don't yet have meaningful traffic. ...
  • Don't A/B test if: you can't safely spend the time. ...
  • Don't A/B test if: you don't yet have an informed hypothesis. ...
  • Don't A/B test if: there's low risk to taking action right away.
13 Dec 2015

What is the drawback of a B testing? ›

A/B testing can take a lot longer to set up than other forms of testing. Setting up the A/B system can be a resource and time hog, although third-party services can help. Depending on the company size, there may be endless meetings about which variables to include in the tests.

What are some pitfalls of a B testing? ›

Ten common A/B testing pitfalls and how to avoid them
  • Pitfall 1: Ignoring the effects of the significance level. ...
  • Pitfall 2: Declaring winners of multiple offer tests with no statistically significant difference. ...
  • Pitfall 3: Ignoring the effects of statistical power. ...
  • Pitfall 6: Stopping tests prematurely.

How do you determine sample size for Ab test? ›

One way to perform the test is to calculate daily conversion rates for both the treatment and the control groups. Since the conversion rate in a group on a certain day represents a single data point, the sample size is actually the number of days.

Are AB tests worth it? ›

A/B testing is highly useful, no question here. But a lot of businesses should not be doing it. They're not ready yet. Roughly speaking, if you have less than 1000 transactions (purchases, signups, leads etc) per month – you're gonna be better off putting your effort in other stuff.

How many users does AB test have? ›

To run a valid A/B test, the larger the sample size, the better. As a general guideline, test results are valid when you achieve at least, 30,000 visitors per variant with at least 3,000 conversions on that variant.

How does Netflix use a B testing? ›

We split a random sample of Netflix members into two groups using random assignment. Group “A” receives the current product experience, while Group “B” receives some change that we think is an improvement to the Netflix experience. Here, Group “B” receives the “Upside Down” product experience.

What is novelty effect in AB testing? ›

So, it is easy to select a certain sample for testing and eliminate factors or variants that might result in their hypothesis being incorrect. Novelty effect: error or changes in test results that are an outcome of introducing something unusual or new that the visitor is not used to.

What is primacy effect in AB testing? ›

The primacy effect refers to the experienced users adjust to the new version. For example, we are interested in learning how a new version update would affect the CTR. The new version is so different from the old version, making experienced users confused and click (open) multiple links.

What are AB tests? ›

What is A/B testing? A/B testing (also known as split testing or bucket testing) is a method of comparing two versions of a webpage or app against each other to determine which one performs better.

How do you find the novelty effect? ›

Therefore, the most commonly used way to check for novelty effect is by segmenting users in new vs returning. If the feature is winning for returning users, but not for new users, that's a really strong sign that novelty effect dynamics are happening.

What is a false positive for a B testing? ›

There are two types of mistakes we can make in acting on test results. A false positive (also called a Type I error) occurs when the data from the test indicates a meaningful difference between the control and treatment experiences, but in truth there is no difference.

What is a fixed horizon in a B testing? ›

The classic breed of A/B testing methods—known as fixed-horizon tests—assumes that a specific sample size has been committed to in advance; the statistics rely on this assumption for validity.

How do you overcome the novelty effect? ›

3 Ways to Overcome the Novelty Effect
  1. Test for long enough.
  2. Pay attention to what time of year you are testing.
  3. Look at different customer cohorts.
11 Mar 2021


1. How Not to Land an Orbital Rocket Booster
2. Scalping $100 to $10k - Day 18 (Liquidated + NEW plan)
(Joel on Crypto)
3. do NOT age-shame Erica #StrangerThings #Netflix
(Still Watching Netflix)
4. How Cocaine and Depression Affect Your Brain | With Karl Friston
(Jordan B Peterson Clips)
5. Green Day - Good Riddance (Time Of Your Life) [Official Music Video]
(Green Day)
6. Ethical Hacking 101: Web App Penetration Testing - a full course for beginners
Top Articles
Latest Posts
Article information

Author: Edwin Metz

Last Updated: 14/03/2023

Views: 6120

Rating: 4.8 / 5 (78 voted)

Reviews: 85% of readers found this page helpful

Author information

Name: Edwin Metz

Birthday: 1997-04-16

Address: 51593 Leanne Light, Kuphalmouth, DE 50012-5183

Phone: +639107620957

Job: Corporate Banking Technician

Hobby: Reading, scrapbook, role-playing games, Fishing, Fishing, Scuba diving, Beekeeping

Introduction: My name is Edwin Metz, I am a fair, energetic, helpful, brave, outstanding, nice, helpful person who loves writing and wants to share my knowledge and understanding with you.