Data-Driven Persona-Conditioned Agents for A/B Test Simulation
A research paper proposes a simulation framework that predicts A/B test outcomes using LLM-powered agents conditioned on data-driven personas grounded in real user behavioral signals. The framework frames A/B test simulation as a structured question task and studies question design formats, persona data source and domain alignment, trade-off between per-persona behavioral depth and population diversity, and efficient population subsampling. On a benchmark of 40 A/B tests spanning two metric types, the best configuration achieves 0.75-0.90 directional accuracy depending on the test metric.
A/B testing is the gold standard for evaluating product changes, but each experiment requires real user traffic, engineering effort, and weeks of measurement. The proposed simulation framework predicts A/B test outcomes using LLM-powered agents conditioned on data-driven personas grounded in real user behavioral signals. Unlike prior work that relies on synthetic or rule-based personas, these agents are constructed from anonymized behavioral data—activity patterns, engagement signals, and inferred demographics—enabling more faithful population modeling. The framework frames A/B test simulation as a structured question task and systematically studies question design formats, persona data source and domain alignment, trade-off between per-persona behavioral depth and population diversity, and efficient population subsampling. On a benchmark of 40 A/B tests spanning two metric types, the best configuration achieves 0.75-0.90 directional accuracy depending on the test metric, demonstrating that data-driven personas are a viable path toward fast, low-cost experiment pre-screening.
The framework uses LLM-powered agents conditioned on data-driven personas derived from anonymized behavioral data, including activity patterns, engagement signals, and inferred demographics. It frames A/B test simulation as a structured question task and systematically studies question design formats, persona data source and domain alignment, trade-off between per-persona behavioral depth and population diversity, and efficient population subsampling. The best configuration achieves 0.75-0.90 directional accuracy on a benchmark of 40 A/B tests spanning two metric types.
This approach could reduce the need for real user traffic and engineering effort in early-stage experiment screening, potentially accelerating product iteration cycles. The use of data-driven personas grounded in real behavioral signals addresses a key limitation of prior synthetic or rule-based persona approaches, making simulation more faithful to actual user populations.
The framework offers a fast, low-cost method for pre-screening A/B tests, potentially reducing the number of live experiments needed and saving engineering resources and time. It could enable product teams to prioritize high-likelihood-of-success changes before committing to full-scale A/B tests.
Next signals to watch include validation on larger and more diverse A/B test benchmarks, extension to additional metric types beyond the two studied, and integration into product experimentation platforms. Further research may explore the impact of persona data freshness and privacy-preserving techniques on simulation accuracy.