Fareed Mosavat is a former Dir of Product at Slack where he focused on growth, adoption, retention, and monetization of Slack’s self-service business. Prior to Slack he worked on product and growth at Instacart, RunKeeper, and Zynga. Fareed recently joined Reforge as an EIR (Executive in Residence), and will be leading the upcoming Experimentation + Testing program developed with Elena Verna (Former SVP Growth @ SurveyMonkey, Malwarebytes and advisor to MongoDB, Miro, and others).
Over the past 10 years as a product leader I’ve shipped hundreds of A/B tests and product experiments to every kind of customer, from social gamers to the most discerning enterprise software buyers. I’ve learned many lessons about building a disciplined and high-impact culture of experimentation.
Great product and growth teams must develop a culture of strategic, rigorous experimentation. When done right, experimentation is a critical tool to move your business and your understanding of customers forward. Unfortunately, many teams use experimentation in misguided ways. I think it’s easier to understand what great experimentation looks like when contrasted with bad experimentation.
So, in the spirit of Ben Horowitz’s classic post, Good Product Manager / Bad Product Manager, here are my lessons learned about Good Experiments and Bad Experiments.
----
Good experiments advance product strategy. Bad experiments only advance metrics.
Good experiments are built around product themes. Every good experiment furthers the product team’s understanding of the customer and her needs. They build on each other, creating insight and depth of understanding in an area. Bad experiments are scattered, one-off tests that stand alone. They come from unfocused brainstorms that result in not being connected to the product vision or a growth strategy.
Good experiments drive impact by solving real user problems. They have strong, well-reasoned hypotheses grounded in data analysis, customer insights, and market research. Good experiments are about understanding true customer behavior around the things that matter. Bad experiments move metrics by confusing or tricking your users. They make things harder for your users, rather than solving underlying problems.
Good experiments are conceived as bets. You know they have a chance to fail, but based on the info you have available, it is a good investment to make. They help you learn about the things that matter, enabling you to take bigger bets over time. Bad experiments are endless optimizations. They adjust things around the edges in an attempt to improve performance in a marginal way. They steal time, energy, and resources from validating more meaningful bets. Businesses aren’t built on optimizations.
Good experiments ask “What do we do next with what we learned?” They dig in on questions like why did it work or not work? Does this prove or disprove our hypothesis? Was it a good test of the hypothesis? If it failed, should we invest more to make it successful? If it succeeded, should we do more to make it even more successful? Bad experiments ask “Should we ship or turn off?” and are quickly forgotten.
Good experiments are used as a tool of humility. They are used to test something we believe in, but we are humble enough to know that we aren’t sure if it will work. Bad experiments are used to defer decision making, settle disagreements, or let the data tell us what the right thing to build is.
Good experiments define success up-front. They properly instrument the metrics associated with the definition of success. Bad experiments add many metrics to the dashboard, then cherry pick the ones that improve. They segment results in endless ways in a search for a positive signal used to justify the experiment idea rather than prove the hypothesis. Bad experiments get restarted because the right metrics weren’t instrumented from the beginning.
Good experiments consider tradeoffs. They understand that the levers of the business work together as a system. Everything is connected. If you move sign-ups, you might decrease activation. If you move monetization, you might decrease retention. Bad experiments focus on their own OKRs and goals. They treat every improvement as a win, even if that win came at a cost elsewhere in the system.
Good experiments use tight exposure groups. Everyone in the test group experienced the change, with an equally sized true control group.. Bad experiments use random assignments to wide audiences. Users in the test group weren’t exposed to the change while those in the control wouldn’t have experienced the change if in the test group. Good experiments are well instrumented at launch. Bad experiments get restarted because of missing data. Garbage in, garbage out.
Good experiments are communicated. They build a narrative around learning. They are written up, shared broadly, discussed in groups, and documented for the future. Bad experiments run a test but never close the loop.
Good experiments are integrated into the core product process. They incorporate cultural principles and think about long term team success. Bad experiments act as renegades. They create endless friction with other teams and lead to distrust and lack of collaboration.
Good experiments have a plan for success. They invest in making a successful experiment a real part of the product and think about full go-to-market integration. Bad experiments leave MVPs in the product forever.
Thank you to Britt Jamison and Zindzi McCormick for brainstorming on early versions of this post.
---
Subscribe here for additional content on product growth. If you or your team are practitioners with 3+ years of experience, check out one of our upcoming programs: