A/B tests in marketing
A/B Tests in Marketing: Basics and Comparison of Tools
What is an A/B test?
In an A/B test, two versions of a page are compared in terms of a specific criterion or page element. The target page is duplicated and an element, such as the color of a button, is changed. Variants A and B are then randomly shown to a predefined target audience. The goal is to measure which of the variants achieves better results, for example by comparing the click-through rates. A/B testing is a fundamental element of data-driven marketing, allowing for continuous and data-based evaluation of user interactions.
There are various ways to set up an A/B test for a landing page or a shop:
A/B/n Test: Comparison of one or more variants with the original. Exactly one element is adjusted, for example, the color of the CTA button.
Multivariate Test: Comparison with multiple changed elements, such as color and position of the CTA button, in the form of test combinations.
Split URL Test: Comparison of different versions of entire websites, for example, to test different designs.
What are the advantages of an A/B test?
Initially, companies can continuously optimize their websites or shops by analyzing user interactions to make them more accessible to visitors. Additionally, it is possible to test landing pages against each other through an A/B test before promoting them. This way, alternatives with a low conversion rate can be excluded, and the marketing budget can be used more efficiently.
In the long run, A/B tests provide deep insights into user behavior, thereby enhancing the understanding of the needs and expectations of customers. Through regular testing, companies can improve the customer experience of their websites, increase conversion rates, and ultimately achieve a higher return on investment (ROI).
A/B Testing Tools Compared
A wide variety of different tools are available for conducting A/B tests. In addition to a large selection of standalone solutions, some tools are part of larger digital marketing suites (e.g., Adobe). To gain a little overview of the large number of offers available, we have taken a closer look at a few tools.
VWO
The VWO testing tool is part of the VWO DXP, but is also available as a standalone solution. The prices of the license plans depend on the range of features and website traffic and can be viewed from the provider. The tool's GDPR compliance is ensured in part by data hosting in the EU.
Die DSGVO-Konformität des Tools wird unter anderem durch ein Datenhosting in der EU gewährleistet.
Experiments and Editor
A/B tests, split URL tests, and multivariate tests
Extensive visual editor (drag-and-drop, WYSIWYG)
Numerous customization options and on-page personalizations
Code editor for server-side changes
Targeting by attributes such as browser, operating system, cookies, visitors (new and returning); additional attributes are available in the Pro plan
Evaluation and Analysis of A/B Tests
Integrated dashboard with a variety of reporting functions and advanced analyses (e.g., funnel analyses and on-page surveys)
Audience filters for reports
Metrics such as sessions, transactions, revenue, bounce rate, and session duration
Integration with Google Analytics and other analysis tools
AB Tasty
AB Tasty is a standalone solution. The provider specializes in AI-based support for setting up tests. The prices of the license plans depend on the range of features and website traffic and are available on request from the provider. GDPR compliance is also ensured here, in part by data hosting in the EU.
Experiments and Editor
A/B/n tests, split URL tests, multivariate tests, and predictive tests
Extensive visual editor (drag-and-drop, WYSIWYG)
AI-based personalizations
Code editor for server-side changes
Targeting by attributes such as browser, operating system, cookies, visitors (new and returning), location
Evaluation and Analysis of A/B Tests
Integrated dashboard with many reporting functions (e.g., campaign evaluations and cumulative annual overviews)
Audience filters for reports
Metrics such as sessions, transactions, revenue, bounce rate, session duration, and ROI
Various integration options, including with suite solutions
ABlyft
ABlyft offers a standalone solution for conducting and evaluating A/B tests. The tool impresses with a very extensive code editor, which requires a deeper technical understanding. In addition to individual pricing, Ablyft offers a free version of the tool to familiarize yourself with the platform. Data hosting in the EU ensures the tool's GDPR compliance. In addition to the cloud solution, ABlyft offers the option for self-hosting.
Experiments and Editor
A/B tests, split URL tests, and multivariate tests
Very easy-to-use visual editor
Extensive code editor for server-side changes
Targeting by attributes such as browser, operating system, cookies, and visitors (new and returning)
Creating audiences and pages is done based on code
Evaluation and Analysis of A/B Tests
Integrated dashboard
Evaluation is based on selected metrics (views, clicks, revenue, custom)
Integration with a variety of other analysis tools enables advanced evaluation
Varify.io
Varify.io is an A/B testing and personalization tool. The platform deploys test variants while measurement runs through the analysis tool already in use. This ensures that the existing analytics environment remains the central data basis for evaluating the tests. The prices are traffic-independent and currently start at €149/month when paid annually. According to the provider, Varify.io works without cookies; hosting is done in Frankfurt.
Experiments and Editor
A/B tests, split URL tests, multivariate tests, and personalizations
Visual editor with drag-and-drop and WYSIWYG functions
Code editor for JavaScript and CSS
AI-supported creation of test variants as well as AI-CRO auditing for analyzing pages and deriving possible optimization ideas
Targeting by criteria such as page, device, geo, referrer, language, events, and custom conditions
Campaign boosters like information bars and audience-based personalization
Evaluation and Analysis of A/B Tests
Evaluation through the analysis tool already in use
Integrated reporting with significance calculation, trend charts, CSV export, and API access
Integrations with other analytics and qualitative testing tools
KPI coverage depends on the connected analysis tool and the goals defined there
How do I find the right A/B testing tool for my company?
First of all, all tools have identical basic functions regarding the testing procedures themselves, covering a range of A/B tests, split URL tests, and multivariate tests. Some tools also offer advanced testing options such as multipage experiments or A/B/n tests. In addition to these standard functions, many solutions have additional features such as (partially AI-supported) personalization options, the insertion of campaign boosters, and an audience filter for reporting.
Even though creating tests often does not require deep programming knowledge, some tools at least require a basic understanding of programming languages such as CSS and JavaScript. If your team does not have the necessary expertise in this area, this should be taken into account when selecting the tool.
| Requirements | VWO | ABlyft | ABTasty | Varify.io |
|---|---|---|---|---|
| Editor Features | visual (Drag-and-Drop) and code-based | visual and code-based | visual (Drag-and-Drop) and code-based | visual and code-based, with AI-assisted variant creation |
| Personalization | Features depend on plan | adequate (based on target groups) | highly comprehensives, AI-powered assistance | adequate (based on target groups) |
| Integration Capabilities | Google Analytics 4, additional Analytics Tools (e.g. other VWO solutions) | Google Analytics 4, additional Analytics Tools (e.g. Heatmap Tools) | Google Analytics 4, additional Analytics Tools (e.g. Adobe) | Google Analytics 4, additional analytics and qualitative testing tools |
| KPI Coverage | covered through Tools | partly covered through Tools, can be extended via GA4 | covered through Tools | covered through connected analytics tools |
| Analysis | integrated Dashboard with comprehensive Reporting Features | integrated Dashboard for analyzing selected target metrics | integrated Dashboard with comprehensive Reporting Features | integrated reporting and analysis via connected analytics tools |
| GDPR Compliance | Data hosted within the EU | Data hosted within the EU, Options for Self-Hosting | Data hosted within the EU | cookie-less approach, data hosted within the EU |
Therefore, users should clearly define the requirements that need to be covered by the A/B testing tool before selecting one. The requirements profile should cover the following points:
Feature scope of the editor and on-page personalization options
Structure of the analysis dashboard and requirements for reporting functions
Definition of the relevant KPIs and metrics to be measured.
GDPR compliance of the solution
Integration possibilities with other tools
The majority Provider of A/B testing tools scale their pricing models according to the scope of the solution's services, the number of website visitors, and the associated traffic. Accordingly, the traffic of the website on which the tool is to run should also be estimated in advance. This can help avoid unexpectedly high license prices. Furthermore, the user-friendliness of the dashboard and editor, as well as the practicality of the tool itself, are also important. However, as this is a rather subjective assessment, we recommend making a preliminary selection of possible providers based on the requirements profile. You should then take advantage of the free trial versions that are often offered. This allows companies to see how the tool works in practice and test whether all requirements at the application level are implemented as desired. At the same time, it becomes clear whether all team members can use the tool without any problems and whether the user-friendliness is considered suitable.
Share this post