The ABC of A/B Test

Our previous cohort analysis method “is introduced by comparing different period of specific indicators for time window, and today” A/B testing “is introduced by different groups of users on the same time window for comparing different versions of the reaction.
The so-called A/B test is an experiment to compare two (or more) versions of A web page or APP’s “performance”.The A/B test is essentially an experiment.In this experiment, a page of two or more different solution to show a particular user group (the user based on the possibility of random, there may be specific characteristics), and then according to the statistical data analysis, which on the specified “performance” better.We often face multiple choice of design solutions or marketing strategies, and A/B testing is A great way to solve these problems.
Through A/B testing, we can compare the different versions of the user experience, and in view of the updated version of the web page or APP to ask questions, and then collect the relevant data, analysis and update web pages or APP to our established metrics.
Currently, most of the A/B test tools on the market do not support the allocation of traffic to the allocation of traffic (generally random).As a result, the results of the test can not only reflect the preferences of the target’s core users, but also may mislead the product manager and misjudge the direction of product improvement.In this recommendation user behavior analysis tools Cobub Razor, it can help us to accurately identify user attributes, thus to provide A/B testing flow allocation strategy, to ensure the scientific nature of the flow distribution, the reliability of test results.In addition, Cobub also provides real-time monitoring and data analysis of experimental target indicators. We can optimize the flow strategy in real time according to the analysis results, and help to form the closed loop of product optimization.

The A/B test will help us to generate hypotheses from the interface optimization, which will also help us to make the right decisions based on sufficient data analysis.With the support of A/B test, we can make decisions from the “I think…Turning to data analysis results, “we know…”.By measuring the impact of the updated version on each metric, we can ensure that every update has a positive result.

How does A/B test work?

In A/B test, we can create an updated version of the same page or APP interface.The differences between versions can be very simple, such as changing a single icon or button, or a complete redesign of the page.During the testing process, we showed the original version of the page (call it the control group) to half of the users, and half of the page’s updated version (called the test group).

In the A/B test, we collected user behavior data from the control group and the test group, and then analyzed the impact of the updated version on user behavior.

Why do we needA/B test？

Both individual and team or company, after the analysis of the A/B testing data results, we can use the data to talk, to optimize product, enhance the user experience, let the user behavior towards the direction we want. The A/B test also verifies our assumptions. Sometimes the product changes we make from experience don’t get the results we expect (because our customers don’t let us arrange them). Through A/B testing, we can get away from empiricism and shift to data-driven product development.
A/B test can continuously improve our products and improve user experience, so as to help us achieve various goals, such as registration rate, conversion rate, etc.
Operations team, for example, want to A landing page to improve sales target marketing activities, in order to achieve this goal, we will try to title, visual images, forms, action buttons, and the overall layout of the page for A/B testing.
Each time a change is tested, it helps us determine which changes have an impact on the user’s behavior. Over time, our products will get better and better because of these successful improvements in testing.

A/B testing enables us to optimize our products in marketing campaigns and to let users take action based on our goals.
By testing the AD copy, we can see which version has attracted more users’ clicks.By testing the subsequent login page, we can see which layout can facilitate the user’s purchase.If the changes in each step can effectively get new customers, then the cost of marketing will be greatly reduced.

A/B testing can also be used by product managers and designers to demonstrate new features or change the impact of user experience. The product login, user engagement, pattern, and product experience can all be optimized through A/B test. In conclusion, we can achieve the goal through the A/B test, and verify the hypothesis.

A/B测试过程

Here is the A/B test framework, which we can use to run the test:
• Data collection：Data analysis of the product allows us to find the problem and find the direction that needs to be optimized. First, we need to collect data, which can start from the high traffic areas of the site or APP, which can help us quickly identify the key to the problem. At the same time we need to look for pages that can improve low conversion rate and high turnover rate.
• Set goals：Our goal is to measure whether the updated version is better than the original user experience and more successful. Goals can be anything, such as clicking a button, linking to a product or completing a registration.
• Generate the hypothesis：Once the goal is clear, we can generate the hypothesis of A/B test, which is used to explain why we feel the newer version is better than the original version. After having this hypothetical list, we can test it in order of the desired results and implementation difficulties.
• To create change：With the front after a few steps, we can on our website or APP to make the desired changes, design the iterative solution, these changes can be the color of the button, the order of the elements on the page switching, hidden navigation or completely redesign. We create these changes to ensure that they meet our intended goals.
• Running test：Start our experiment and wait for the user to participate. In this step, users of our website or APP will be randomly assigned to the control group and test group, every step of the user operation will be record collection, calculation and comparison, to determine the control group and test group on each change.
• Analysis result：The results were analyzed after the experiment was completed. The A/B test will show the experimental data and tell us whether there are significant differences between the two versions of user behavior.
• Release the best version：If the behavior of the test group meets our intended target, then we can continue to improve the product according to the A/B test results. Instead of being discouraged, we can take this test as an experience and generate new assumptions and continue testing.

Regardless of the test result, we should implement the product optimization closed-loop according to the test experience and continuously improve the user experience.
Here are some of the pitfalls that are common during the use of A/B testing:
• Success in trials does not equal improvement
• Randomly select users to participate in the experiment
• Many tests, a little modification
• Block A/B version of human selection