Experimentation: Putting Research Papers Into Prod

Innovation happens when we apply concepts from research papers and put them into production. Taking complex ideas and applying them to a real world production environment requires one key step: online evaluation. Theory and offline evaluations can only take you so far, you have to evaluate ideas in an online controlled setting. The better your experimentation practices are, the better you will be at evaluating research in production settings.

In this talk, we will zoom in on experimentation strategies from popular research papers that in theory, should really unlock innovation and the speed of experimenting but in practice, are hard to apply, why that's the case and what you can do to put research paper concepts into production.

References

• Debiased Balanced Interleaving at Amazon Search Nan Bi, Pablo Castells, Daniel Gilbert, Slava Galperin, Patrick Tardif, Sachin Ahuja - CIKM 2022
• Overlapping experiment infrastructure: more, better, faster experimentation Diane Tang, Ashish Agarwal, Deirdre O'Brien, Mike Meyer - KDD 2010
• Online short-term reliability evaluation using a fast sorting technique H. Liu, Y. Sun, L. Cheng, P. Wang, F. Xiao - 2008

Leemay Nassery is an engineering leader who has worked at prominent companies such as Spotify, Etsy, Dropbox, and Comcast. From personalizing the homepage at Spotify to leading the A/B testing and personalization engineering teams at Comcast to building the data warehousing teams at Etsy, Leemay is passionate about tying engineering to business outcomes and user impact. In these experiences, she has learned that you do not need the perfect experimentation platform or a large engineering team to start gaining insights as to how users engage and interact with your product.

• LinkedIn: https://www.linkedin.com/in/leemaynassery/
• Practical A/B Testing: https://www.practicalabtesting.com/