A/B Testing

A properly run A/B test randomly assigns traffic to each variant, runs long enough to reach statistical significance, and isolates a single change between variants so the cause of any difference in outcomes is clear. Poorly run tests — small sample sizes, multiple simultaneous changes, non-random assignment — produce results that look conclusive but aren't. For AI shopping tools specifically, the relevant A/B test compares shoppers with access to the tool (B) against a genuine random holdout without access (A) — not a comparison between shoppers who voluntarily used the tool and those who didn't, which introduces selection bias.