A/B tests in marketing
A/B Tests in Marketing: Basics and Comparison of Tools
What is an A/B test?
In an A/B test, two versions of a page are compared in terms of a specific criterion or page element. The target page is duplicated and an element, such as the color of a button, is changed. Variants A and B are then randomly shown to a predefined target audience. The goal is to measure which of the variants achieves better results, for example by comparing the click-through rates. A/B testing is a fundamental element of data-driven marketing, allowing for continuous and data-based evaluation of user interactions.
There are various ways to set up an A/B test for a landing page or a shop:
A/B/n Test: Comparison of one or more variants with the original. Exactly one element is adjusted, for example, the color of the CTA button.
Multivariate Test: Comparison with multiple changed elements, such as color and position of the CTA button, in the form of test combinations.
Split URL Test: Comparison of different versions of entire websites, for example, to test different designs.
What are the advantages of an A/B test?
Initially, companies can continuously optimize their websites or shops by analyzing user interactions to make them more accessible to visitors. Additionally, it is possible to test landing pages against each other through an A/B test before promoting them. This way, alternatives with a low conversion rate can be excluded, and the marketing budget can be used more efficiently.
In the long run, A/B tests provide deep insights into user behavior, thereby enhancing the understanding of the needs and expectations of customers. Through regular testing, companies can improve the customer experience of their websites, increase conversion rates, and ultimately achieve a higher return on investment (ROI).
A/B Testing Tools Compared
There are a variety of tools available for conducting A/B tests. In addition to a wide selection of standalone solutions, some tools are part of larger digital marketing suites (e.g. Adobe). To provide a bit of an overview of the large number of offerings, we have taken a closer look at a few tools.
VWO
The VWO testing tool is part of VWO DXP, but is also available as a standalone solution. The prices of the license plans depend on the features and website traffic and can be viewed at the provider.
The GDPR compliance of the tool is ensured, among other things, by hosting data in the EU.
Try it out and edit.
A/B tests, split URL tests, and multivariate tests
Comprehensive, visual editor (drag-and-drop, WYSIWYG)
Variety of customization options and on-page personalizations
Code editor for server-side changes
Targeting based on attributes such as browser, operating system, cookies, visitors (new and returning); additional attributes are available in the Pro Plan.
Evaluation and analysis of the A/B test
Integrated dashboard with diverse reporting functions and advanced analyses (e.g. funnel analyses and on-page surveys).
Target group filter for reports
Key performance indicators such as sessions, transactions, revenue, bounce rate, and session duration.
Integration with Google Analytics as well as with other analytics tools.
AB Tasty
AB Tasty is a standalone solution. The provider specializes in AI-based support for setting up tests. The prices of the license plans depend on the feature set and website traffic and are available on request at the provider. GDPR compliance is ensured here, among other things, through data hosting in the EU.
Try it out and edit.
A/B/n tests, split URL tests, multivariate tests, and predictive tests
Comprehensive, visual editor (drag-and-drop, WYSIWYG)
AI-based personalizations
Code editor for server-side changes
Targeting based on attributes such as browser, operating system, cookies, visitors (new and returning), location.
Evaluation and analysis of the A/B test
Integrated dashboard with many reporting functions (e.g. campaign evaluations and cumulative annual overviews)
Target audience filter for reports
Key performance indicators such as sessions, transactions, revenue, bounce rate, session duration, and ROI.
Various integration options, including integration with suite solutions.
ABlyft
ABlyft offers a single solution for conducting and evaluating A/B tests. The tool impresses with a very comprehensive code editor, therefore requiring a deeper technical understanding. In addition to individual pricing, Ablyft offers a free version. Use the tools to familiarize yourself with the platform. Data hosting in the EU ensures the GDPR compliance of the tool. In addition to the cloud solution, ABlyft also offers the option for self-hosting.
Try it out and edit.
A/B tests, split URL tests, and multivariate tests
Very easy-to-use visual editor
Comprehensive code editor for server-side changes
Targeting based on attributes such as browser, operating system, cookies, and visitors (new and returning).
Creating target groups and pages is done based on code.
Evaluation and analysis of the A/B test
Integrated dashboard
Evaluation is done based on selected target metrics (views, clicks, revenue, custom).
Integration possibilities with a variety of other analysis tools enable advanced evaluation.
Varify.io
Varify.io is a relatively new German provider of an A/B testing tool on the market. Despite being a work in progress, Varify.io already impresses with a comprehensive visual editor. The roadmap for developing additional features of the editor can be found on the website. However, Varify.io does not have an integrated dashboard and relies on Google Analytics integration for evaluating A/B tests.
At Varify.io, there are no licensing plans based on traffic and scope of performance. Companies that want to use the tool for their website pay 99€/month with no traffic limitations. GDPR compliance is ensured through data hosting in the EU.
Experiments and Editor
A/B tests, split URL tests, and multivariate tests
Visual editor with extensive features, such as inserting campaign boosters like information bars and personalization.
Targeting based on attributes such as browser, operating system, cookies, visitors (new and returning), and language.
Creating target groups is done based on code.
Evaluation and analysis of the A/B test
No integrated dashboard
Evaluation is only possible via Google Analytics 4 or your own tracking solution.
How do I find the right A/B testing tool for my company?
First of all, all tools have identical basic functions regarding the testing procedures themselves, covering a range of A/B tests, split URL tests, and multivariate tests. Some tools also offer advanced testing options such as multipage experiments or A/B/n tests. In addition to these standard functions, many solutions have additional features such as (partially AI-supported) personalization options, the insertion of campaign boosters, and an audience filter for reporting.
Even though creating tests often does not require deep programming knowledge, some tools at least require a basic understanding of programming languages such as CSS and JavaScript. If your team does not have the necessary expertise in this area, this should be taken into account when selecting the tool.
| Requirements | VWO | ABlyft | ABTasty | Varify.io |
|---|---|---|---|---|
| Editor Features | visual (Drag-and-Drop) and code-based | visual and code-based | visual (Drag-and-Drop) and code-based | visal and code-based |
| Personalization | Features depend on plan | adequate (based on target groups) | highly comprehensives, AI-powered assistance | adequate (based on target groups) |
| Integration Capabilities | Google Analytics 4, additional Analytics Tools (e.g. other VWO solutions) | Google Analytics 4, additional Analytics Tools (e.g. Heatmap Tools) | Google Analytics 4, additional Analytics Tools (e.g. Adobe) | Google Analytics 4, custom Tracking Tools |
| KPI Coverage | covered through Tools | partly covered through Tools, can be extended via GA4 | covered through Tools | covered through GA4 Integration |
| Analysis | integrated Dashboard with comprehensive Reporting Features | integrated Dashboard for analyzing selected target metrics | integrated Dashboard with comprehensive Reporting Features | no integrated Dashboard, Analysis via GA4 |
| GDPR Compliance | Data hosted within the EU | Data hosted within the EU, Options for Self-Hosting | Data hosted within the EU | Data hosted within the EU |
Therefore, users should clearly define the requirements that need to be covered by the A/B testing tool before selecting one. The requirements profile should cover the following points:
Feature scope of the editor and on-page personalization options
Structure of the analysis dashboard and requirements for reporting functions
Definition of the relevant KPIs and metrics to be measured.
GDPR compliance of the solution
Integration possibilities with other tools
The majority Provider of A/B testing tools scale their pricing models according to the scope of the solution's services, the number of website visitors, and the associated traffic. Accordingly, the traffic of the website on which the tool is to run should also be estimated in advance. This can help avoid unexpectedly high license prices. Furthermore, the user-friendliness of the dashboard and editor, as well as the practicality of the tool itself, are also important. However, as this is a rather subjective assessment, we recommend making a preliminary selection of possible providers based on the requirements profile. You should then take advantage of the free trial versions that are often offered. This allows companies to see how the tool works in practice and test whether all requirements at the application level are implemented as desired. At the same time, it becomes clear whether all team members can use the tool without any problems and whether the user-friendliness is considered suitable.
Share this post