Seesaw Experimentation: A/B Tests with Spillovers

4 Nov 2024  ·  Jin Li, Ye Luo, Xiaowei Zhang ·

This paper examines how spillover effects in A/B testing can impede organizational progress and develops strategies for mitigating these challenges. We identify a phenomenon termed ``seesaw experimentation'', where a firm's overall performance paradoxically deteriorates despite achieving continuous improvements in measured A/B testing metrics. Seesaw experimentation arises when successful innovations in primary metrics generate negative externalities in secondary, unmeasured dimensions. To address this problem, we propose implementing a positive hurdle rate for A/B test approval. We derive the optimal hurdle rate, offering a simple solution that preserves decentralized experimentation while mitigating negative spillovers.

PDF Abstract
No code implementations yet. Submit your code now

Datasets


  Add Datasets introduced or used in this paper

Results from the Paper


  Submit results from this paper to get state-of-the-art GitHub badges and help the community compare results to other papers.

Methods