The Skill Explained
What Are A/B Testing and Experimentation?
A/B Testing and Experimentation means comparing two or more versions of something, such as a feature, a page or a model, by showing them to randomly chosen groups and measuring which works better. Randomisation is what lets you say the change caused the difference.
Why AI Employers Ask for Them
Companies use experiments to decide what to ship based on evidence, not opinion. For AI products, an experiment shows whether a new model or prompt actually helps real users, which offline tests cannot fully prove. Data scientists, product analysts and AI product managers are expected to design and read experiments correctly.
How It Works
You choose one primary metric that reflects the goal (for example task completion or conversion) and guardrail metrics that must not get worse (for example errors or complaints). Users are randomly split into a control group, who see the current version, and a treatment group, who see the new one.
Before starting, a power calculation sets how many users and how much time are needed to detect a meaningful effect. During the test, results should not be checked and stopped early whenever they look good, because that inflates false positives. At the end, a statistical test and a confidence interval show how large the effect is and how sure you can be.
Real experiments face pitfalls: users seeing both versions, novelty effects that fade, seasonal changes and testing many metrics at once. Experimentation platforms help manage assignments and analysis at scale.
What You Do With It at Work
- Design an experiment: hypothesis, metrics, sample size and duration.
- Check that randomisation worked and the groups are comparable.
- Analyse results with confidence intervals and explain them to the team.
- Run online tests of model or prompt changes.
- Build a culture of decisions backed by experiments.
Tools and Terms You Will Meet
- Control and treatment
- The group with the current version and the group with the new one.
- Randomisation
- Assigning users by chance so groups differ only by the change.
- Primary metric
- The single measure that decides the experiment.
- Guardrail metric
- A measure that must not get worse.
- Peeking
- Checking results repeatedly and stopping early, which creates false wins.
Industry Example: Pharma
A pharma company's patient support app tests two reminder messages. Patients are randomly assigned, and after six weeks the team compares medication refill rates between the groups before choosing the message for everyone.
A Common Mix-Up
Comparing users before and after a change is not an A/B test. Many other things change over time; only a randomised control group shows the effect of the change itself.
How Deep Do You Need to Go?
- Know itExplain control groups, randomisation and why they matter.
- Use itDesign experiments with power analysis and analyse results with confidence intervals.
- Master itRun experimentation programmes and handle complex designs and pitfalls.