Why You Should Be A/B Testing Your Website Ahead of the Holidays

The 2021 holiday season is almost here, and along with that comes a flurry of potential customers coming to your website in search of the perfect gift for their loved ones. But how do you guarantee that you actually get the sale for your digital marketing agency or company? The answer: through a seamless and enjoyable customer shopping experience. And one of the best ways to ensure an unforgettable shopping experience is through holiday A/B testing sooner rather than later.

In this blog post, we will cover:

• What A/B holiday testing is and why it’s so important
• Data-backed reasons why NOW is the time to start A/B testing elements on your website
• A list of things you should A/B test ahead of the holiday season
• Important dates to keep in mind this holiday season

What is A/B Testing? And Why Is it So Important? 

A/B testing, also commonly referred to as split testing, is the process of comparing two different versions of a web product  page, app, email, ad campaign, etc. And by measuring the difference in performance between the two versions, we are able to get better insight into what resonates with our customers/users.

In essence, A/B holiday testing is an experiment. Based on a statistical analysis of this experiment you are able to determine which version performs better for a given conversion goal, and in turn build your website accordingly. These data-informed decisions allow us to make UX changes grounded in results rather than “hunches,” essentially removing risky decisions and saving you time AND money.

Clearly, taking out the guesswork can be a huge boost to anyone’s business, so let’s cover some of the most important reasons why you might want to consider setting a split test for your business.

Lower Bounce Rates  

As soon as users end up on your site’s landing page, they are seeing a site that has been perfected through data-driven tests in our case a split test. Therefore, they are more likely to have a positive experience and to convert.

Reduced Cart Abandonment 

There are many reasons for cart abandonment, but tweaking when and where costs are displayed, check-out page messaging and even color scheme may help you find a winning variation that helps reduce cart abandonment and encourage more of your target audience to complete the checkout process.

Personalized Targeting 

You may also learn that one version of a page is working better for one demographic, and a second version works better for a separate demographic. This would also allow you to use personalization to show users the version that is proven to convert better for their demographic.

Increase Revenue/Leads 

The most important benefit this powerful CRO marketing tool can provide for your business is clearly increased sales. Fewer bounce rates, better customer engagement, a higher customer conversion rate, and other aforementioned benefits all lead to increased sales/lead volume for your business, which in turn has the added benefit of increasing your ROAS for your paid marketing campaigns.

Solve Internal Disputes 

Yes, this isn’t a typo. The benefits of split  testing are not just limited to your website’s KPIs. From our experience, there are times where there will be a division in opinion on the client’s internal team about how to approach the design or structure of the website. When that occurs, it sometimes feels like a lose-lose situation since siding with one side will upset the other. Additionally, you’ll never be sure if you picked the right side… well that is if you didn’t run multiple tests of A/B. Running an A/B test eliminates having to pick a side and instead puts both sides against each other head-to-head (if the control isn’t one of them) and lets the test result determine the winner. This way there are no hurt feelings, unless, of course, the control beats out both of them in the test data.

Why NOW is the Time to Start A/B Testing

Now that you know the benefits of testing, let’s discuss why it’s important to start A/B testing now (End of August & Early September).

End of November Code Freeze (Black Friday) 

The reason we are focusing on that final week of November is because Black Friday this year (for the US) takes place on November 27th, and all tests on a website will want to go on a code freeze the week of Black Friday to ensure that no unexpected glitches or poor UX hurt what is projected to be the biggest Black Friday for online retail ever. In other words, you want your website to be buttoned up prior to Black Friday. 

Note: We suggest against testing during Black Friday because we believe that the shopping user behavior of your target audience that occurs during this time is not normal and people will persist through bad UX if it means getting a good deal (people are willing to camp in front of stores and deal with shopping stampedes at brick and mortar stores – the online shopper is not much different during BF&CM). This means there is an outside variable affecting your test result, putting the integrity of the data into question. That said, we are not 100% against testing during the holiday season, but we suggest running tests that are isolated to the week of (11/22 – 12/1) Black Friday & Cyber Monday. During that time, you should also have someone live monitoring tests, ready to pull the plug on one if a test variant is performing incredibly poorly to ensure you are not hurting your site during the biggest sale of the year. If you were able to test like this during BF&CM, then you would gain valuable insight into how to optimize your site for big sale events for the future, but you should be wary of letting these results inform the optimization of your site year round.

A/B Tests Take Time 

The rule of thumb for A/B testing is that a test will take a minimum of 2 weeks to run to completion. Working under this best case scenario, that leaves you around 12 weeks to run tests, or effectively 6 rounds of tests before you hit the final week of November. That, again, is the best case scenario. Most tests will not hit statistical significance in 14 days, therefore, you’ll be running a test 3-5 weeks before you can launch the next test in the same testing track (tests that aim to improve the same goal on the site, hence, why they can’t be run simultaneously). The reason why most tests don’t reach statistical significance in this 14 day window is because there is either not enough traffic spike to the page where the test is taking place or that the test is further up the funnel (i.e. a navigation menu test) where engagement goals typically take longer to hit statistical significance. From our experience, the average site will likely get through 3-4 rounds of testing (2-4 tests per round) prior to Black Friday & Cyber Monday if they start testing at the beginning of September, which is just enough time to make some considerable optimizations to your site prior to Black Friday.

Things you Should A/B Test

Since Black Friday and Cyber Monday are right around the corner, you want to ensure that you test things in the correct order and also test things on your site that will have the greatest impact on your site. 

Navigation Tests

Navigation tests are very important because they can help users find products and information on your site more efficiently, plus bring greater visibility to your entire product catalog. That said, these tests are going to typically take a long time to hit statistical significance since the measurable impact will usually be small (unless the current navigation is very poor). 

These tests will usually take 4-6 weeks to hit statistical significance. Make sure you start them no later than 10/16.

PDP Tests 

The PDP is going to be one of the most important page templates to test on your site prior to the holiday season since they live further down the shopping funnel. The main goal while testing your PDPs is to increase your Add To Cart (ATC) rate, which typically will result in an increase in the website’s revenue and CVR. 

PDP tests generally will hit statistical significance in 2-4 weeks. If you have healthy traffic, you should have no problem squeezing in a PDP test at the start of November.

Cart & Checkout Tests 

These tests, with healthy holiday traffic, can reach statistical significance in 2-4 weeks. Websites that don’t get a lot of  holiday traffic reaching the cart page should likely focus higher up the funnel; otherwise, they run the risk of a test running for 6+ weeks without hitting statistical significance. So while you can see huge improvements from optimizing the cart & checkout (we just recently increased a client’s revenue by an estimated $248k a year from a simple mobile cart test), it is something you should only focus on if enough of your site’s traffic is making it to that stage of the shopping funnel.

Important E-Commerce Holiday Dates to Keep in Mind

The holidays’ are a time in which your buyers’ moods and shopping user behavior likely to change drastically from the norm. They may have a higher sense of urgency to purchase, or they may be more concerned with factors like delivery time than with savings. So it would be a good idea to look for Ecommerce holidays worth A/B testing for your business goals.

Here are some common ones we would suggest:

• Wednesday, January 1 – New Year’s Day
• Thursday, November 26 – Thanksgiving Day

• Friday, December 25 – Christmas Day

• Black Friday

• Cyber Monday

• Halloween

Wrapping Up

The holidays are a make-it or break-it time for many businesses. By following the steps above and planning ahead, we are confident you’ll land in the “make it” category. Happy testing!