Recently, a colleague was telling me about a “successful” experiment they were excited about. They hypothesized that the best time to encourage users to upgrade from a free trial was while they were using a certain common feature.
To test this they put a big button on that page and saw a measurable lift in conversion compared to users who didn't see the button. Based on this result, they were attempting to drive more free users to that feature.
I asked if they had experimented with putting the same button on other commonly used features, and they hadn’t. They hadn't controlled for the risk that the lift was simply a result of putting a prominent CTA in front of more users.
This was especially concerning given that they were funneling resources into more projects based on the assertion that this one feature is important for activation. It was apparent they just didn't understand how to design proper controls to prove their hypothesis.
I’ve seen examples like this countless times when talking with other growth professionals at companies from seed to IPO. Many experiments they describe are under-optimized or invalid due to flawed design - and they don’t even realize it.
It's great that companies are using the scientific method for growth. What used to be “billboards and a prayer,” has evolved into repeatable and measurable systems for growth.
But, many teams don’t get as much value out of the scientific method as they could. Even worse, they often apply it incorrectly which produces misleading results.
By now, most practitioners responsible for growth know they should be using the scientific method to guide their efforts, but until recently, many hadn’t actively used it since 6th grade science class. (Luckily, there are some useful resources out there for ramping up on building processes for growth.)
But if being effective in your job is dependent on using the scientific method, you need to study how the best teams in the world use it. And the best funded and most experienced teams that use the scientific method aren't in tech, they're in science. Unlike the growth community, which is still in its first decade of existence, the science community has been refining the scientific method for thousands of years.