Notable Quotes
"A good experiment starts from clear goals and clear hypotheses. What product change do you want to create, and how is it expected to impact our end users or our business?"
"We try to really balance the experimentation portfolio to not limit ourselves on the small experiments only."
"Some others are game-changing ones. Those are riskier. They could give us negative results, but it could also change the game and help us win big."
"We're far from replacing real A/B tests with agents. But maybe we can get a cheaper and faster simulation layer just to screen all these hypotheses so we can prioritize the high-impact hypotheses better and run them on the real traffic."
"It is a longstanding business, but we have new missions, we have new challenges."
Takeaways
Use LLM based agents as a cheap simulation layer to screen hypotheses, not as a replacement for real A/B tests.
Connect every experiment to the company North Star by cascading it down to sensitive proxy metrics and controllable inputs.
When a launch is too new to have a success metric, pair short term A/B tests with long term holdouts from day one.
