Why Retention Is The Silent Killer

This is part two in a series I’m producing on the topic of retention in preparation for the launch of Reforge’s Retention + Engagement Series. Both this blog series and the Reforge Retention + Engagement Series are being created in collaboration with Casey Winters, former growth leader at Pinterest and GrubHub, and Shaun Clowes, VP of Product at Metromile and former Head of Growth at Atlassian. Subscribe to get the rest of the series and check out part one - The One Growth Metric that Moves Acquisition, Monetization, and Virality.

In this post, Brian, Shaun, and Casey dig into the pitfalls companies face when it comes to retention.

In late 2013, Paul Graham tweeted an up-and-to-the-right growth graph from an unnamed startup, disclosing that it was the fastest growing startup YC had funded to date. The mystery company he referenced was later confirmed to be Homejoy. Shortly after this tweet, Homejoy raised a monster round of $38 million. All great news, right?

Not so fast… the company shut down just 18 months later.

This story is not unique. It’s happened over and over again.

In 2011 Fab.com experienced crazy top line growth shooting to 500,000 users in 10 weeks, which helped them raise $150 million at a $1 billion valuation. Eighteen months later, after raising a total of $336 million, they were acquired in a fire sale for a rumored $15 million.

Need a third example? Okay...

BranchOut, another character in startup theater’s high profile cast, grew from 1 million active users to 5.5 million active users over a 2 month span in the spring of 2012. They raised $50 million in funding, with $25 million of that coming in April 2012 on the tail of that crazy user growth. Yet again, 18 months later the company was sold for “pennies” - this time $5.4 million.

These three startup stories are high profile due to the sheer magnitude of funds the companies raised and the compressed time window in which they flipped from hockey stick growth to sudden death. But this is just a tiny sample - there are countless other startups we never hear about that die for the same fundamental reason.

What is that fundamental reason? Each company was founded and grown by a talented team and funded by tech’s preeminent investors - that can't have been it. So, how could this happen? By now you can probably guess what I’m going to say.

Retention - not only does it make companies - but it also breaks them. For this reason, poor user retention has become the silent killer. In this post, I will walk you through the three key ways that companies go wrong when it comes to retention:

They deprioritize retention altogether
They define their retention metrics incorrectly
They don’t measure engagement

Must-Know Updates from Tech’s Growth Leaders

Get our weekly 5-min digest

How Retention Ends Up Getting Deprioritized

Retention is stealthy and can be misleading in the short-term because it takes time, often years, to see its impact on your company’s growth. To truly understand how it drives the health of your product, you need to take a longer term view - at least one year, preferably multiple years.

But taking a long-term view on growth can be challenging. Most companies and teams operate with short-term goals (quarterly, monthly, or even weekly). As a result, organizations and teams tend to deprioritize retention initiatives, as retention doesn’t initially present itself as the burning problem or impactful opportunity that it actually is.

Digging into the long-term view requires building a basic forward looking quantitative growth model. Let’s take a look at an example comparing two competitors - Company A and Company B.

What you need to know:

Company A is acquiring 1 million new users per month.
Company B is acquiring 2 million new users per month (2X!).

Hearing those numbers, it’s easy to assume Company B will be more valuable than Company A. But what about retention?

What else you need to know:

Company A has a monthly retention rate of 85%.
Company B has a monthly retention rate of 65%.

A -20% difference doesn’t seem so bad, right? Given Company B has 2x the number of new users per month, at first glance it seems Company B will probably still win.

Let’s test this by taking a look at what happens over time. After 6 months, Company A has approximately 4.2M MAUs. Company B has approximately 5.3M MAUs, which is over 1M more than Company A.

Told you Company B was going to win.

Let’s keep going just to make sure. What do we find after extrapolating out 3 years? Company A has approximately 6.6M MAUs and Company B has approximately 5.7M MAUs - Company A now has almost 1M MAUs more than Company B, despite acquiring half the number of new users per month!

The tides have turned - Company A is winning, and by a large margin too.

From this example you can see how higher retention impacts growth over the longer term, simply by reducing the number of users lost over time. But there are multiple growth advantages of higher retention not reflected in this example that would make the difference even larger in a real life scenario.

As I explained in my last post in this series, a boost in retention:

Increases new user acquisition
Improves monetization
Strengthens acquisition muscle so you can push competitors out of channels and explore new channels
Accelerates payback period so you can reinvest in acquisition sooner

Even without these additional advantages of high retention factored in, the company with higher retention wins. The actual outcome, which would include these additional advantages, would likely be that Company A would crush Company B, even if Company B has 2x the acquisition and only somewhat lower retention.

The Risk of Defining Retention Incorrectly

Besides ignoring retention, another common mistake I see companies make is defining their retention metrics incorrectly. Getting this definition wrong, which leads to inaccurately measuring retention, can kill your product. Let’s walk through a few examples to illustrate this point.

Choosing the Wrong Unit of Measurement

A critical part of accurately measuring retention lies in choosing the right unit of measurement to give you the best read on your product’s retention. When choosing your key retention metric - ask yourself - do I want to measure in terms of usage? Revenue? Transactions? Something else? The answer to this question is one of the most important decisions you’ll make when it comes to retention, and getting it wrong can have dire consequences.

Of all of the different types of companies I advise, I see SaaS companies get this wrong the most. When I ask someone from a SaaS business, or another subscription model business, about their retention, I almost always get an answer involving monthly or yearly revenue retention.

This is a red flag for me. I’m far more interested in how retention is reflected in the breadth and depth of product usage...why, you ask?

Revenue retention is the output of engaged users. The usage is the input, and looking only at revenue retention as a proxy for usage retention has two big problems:

1. Revenue can hide what is going on under the hood with product usage, and shield you from signals about your product’s health over the longer term. You may earn a month or a year’s worth of revenue from a paying subscriber, but if that person isn't using the product, they will churn when that month or year is up.

2. If you are trying to improve retention but only tracking revenue retention, the game is over before you’ve even had the chance to play. Once a paying subscriber has churned, winning them back is almost impossible. If you want to improve retention you need to look at usage retention first.

Choosing the Wrong Frequency

A second part of accurately measuring retention lies in choosing the right frequency for product engagement. When considering the right frequency for your product’s retention metric, ask yourself, “Does a user of my product need to engage on a daily, weekly, or monthly basis to be considered active (or possibly longer for certain products)?” Measuring retention on the wrong cadence can trick you into thinking your product has good retention when it really doesn’t.

Let’s consider a social product as an example. In the graph below, the blue line shows the growth of Monthly Active Users for this product.

Things look amazing, right?

The issue with this growth graph is that for most B2C social products to retain users they need to build a habit with users that is daily, not weekly or monthly.

Now, let’s compare our hockey stick MAU graph to the graph of Daily Active User growth for the same time period, and see what we learn.

Uh oh. This graph doesn't look quite as promising. But 35M MAU's and 5M DAU's is still pretty good, right?

Yes it is, but once we get the whole story on this product we realize that what it foreshadows isn’t pretty. The product, revealed below, was Viddy, a social video app famous for playing out the dreaded shark fin graph in record time.

What we learn from Viddy is that it’s possible for companies to manipulate their retention and top line metrics to look good, just by expanding the frequency. When the same user data is looked at on a realistic frequency, it actually tells a story of impending doom. Though these companies can trick themselves and their investors for a short time with this kind of metric manipulation, the chickens of retention invariably come home to roost, turning a bad situation that could possibly be salvaged into a fatal one.

Choosing the Wrong Core Action

Choosing a retention metric involves defining the core action your user takes in your product. In other words, what qualifies a user as active? In his post Don’t Become a Victim to One Key Metric, Casey Winters, former Growth @ Pinterest and co-creator of Reforge’s Retention + Engagement Series, tells a story about getting the core action wrong.

"This led to the creation of a WARC, a weekly active repinner or clicker. A repin is a save of content already on Pinterest. A click is a click through to the source of the content from Pinterest. Both indicate Pinterest showed you something interesting. A weekly event made it impossible to optimize for marginal activity.

There are two issues at play here. The first is the combination of two actions: a repin and a click. This creates what our head of product calls false rigor. You can do an experiment that increases WARCs that might actually trade off repins for clicks or vice versa and not even realize it because the combined metric increased. Take that to the extreme, and the algorithm optimizes clickbait images instead of really interesting content, and the metrics make it appear that engagement is increasing. It might be, but it is an empty calorie form that will affect engagement in a very negative way over the long term.

The second issue is how it ignores the supply side of the network entirely. No team wants to spend time on increasing unique content or surfacing new content more often when there is tried and true content that we know drives clicks and repins. This will cause content recycling and stale content for a service that wants to provide new ideas. Obviously, Pinterest doesn’t use WARCs anymore..."

Casey’s story highlights how choosing the wrong core action can lead to a variety of problems. You could end up ignoring an important part of the user base, thinking you are optimizing for the right behavior when, in reality, you are covering up problems hiding underneath the surface.

The Impact Of Defining Your Retention Metric Incorrectly

To summarize, there are two main problems that arise when you define your retention metrics incorrectly:

1. You end up covering up big problems in your product and faking yourself out. By the time problems show themselves in the metrics you’re tracking, it’s likely too late to fix them.

2. You incentivize your team to solve for the wrong problems because they focus on optimizing the wrong metrics. If you choose a monthly frequency for your retention metric when it should be daily, your team will build features, notifications, and solutions to encourage monthly engagement with your users.

Beware… 1 + 2 = silent death.

You Might Have Breadth, But No Depth

Maybe you defined your retention metric perfectly, you took the long-term view and spent time and resources improving your retention. Because of that your retention curves flatten, rather than going to zero. But despite all of that you could still have a problem if your product has breadth of engagement, without depth.

Retention measures breadth - the percentage of users who are still active in a given time period. But it doesn’t answer the critical question - how active are they within that time period? It’s the measure of your product’s engagement that will answer that question.

Shaun Clowes, former Head of Growth at Atlassian and co-creator of Reforge’s Retention + Engagement Series, explains:

"The problem with retention as an overall metric is that it's binary - your users are either retained or they’re not. Whereas, depth of user engagement is the strongest indicator of long term-value for both the user and the company, because deeply engaged users don't just stay, they stay forever.

With JIRA, we could show that more engaged users would stay retained and installations of the software where the majority of users were engaged would practically never churn. Put a different way, LTV is highly driven by CPA - but it's actually more driven by life expectancy. Marginal increases in life expectancy have an outsized impact because CPA is fixed, while retention is unlimited."

Retention and engagement are yin and yang. Retention without engagement tells only half the story. It doesn't matter if you have breadth, if you don't have depth - I’ll walk you through the reasons why below.

Engagement Drives your Revenue Model

The success of a product’s revenue model correlates with its users’ depth of engagement. Your product could have great retention (breadth), but without depth of engagement it probably won’t become a big business. It is the function of the two combined that makes a great business.

Let’s dig into a few basic examples with different revenue models to understand this point more fully.

Ad Based Models:

Let’s assume Company A has 1 million DAU's, Company B has 2 million DAU's, and both have similar retention rates. Now let’s look at depth of engagement for each company’s product. Say Company A's 1 million DAUs visit 4 times per day, while Company B’s 2 million DAUs visit only 1 time per day.

In this case, Company A has 4 million visits per day, while Company B has 2 million. If they both make money from ads, it becomes clear that Company A will make a lot more money because it has 2 times more ad inventory to sell than Company B. I would much rather be Company A.

Transactional models:

For this example, we’ll compare the impact of engagement for two restaurant delivery companies. Assume an active user is someone who places one or more orders per week, that both companies have 100K Weekly Active Users (WAUs), and that Company A receives an average of 2 orders per week per active user and Company B receives 1 order per week per active user.

With these inputs, Company A will receive 200K orders per week, while Company B will receive only 100k orders per week. If both have the same average order size, then it’s clear that Company A will make 2X more money than Company B.

SaaS Models:

Most SaaS companies build depth of engagement into their pricing structure. For example, email service providers usually charge by the number of contacts. Likewise, companies that offer premium integrations, charge by number of actions, like the number of Zaps fired for Zapier or the number of calls to the Google Maps API. In all of these cases, depth of engagement is built right into the revenue model and is the key driver of revenue.

Engagement Builds Defensibility

Not only does deeper engagement drive monetization, but it also helps a product defend its place in the market. Again, let’s walk through a few examples with different business models to understand how engagement builds defensibility.

Transactional Models:

Consider the same example above with the two restaurant delivery companies that have the same number of WAU's. Again, Company A receives 2 orders per week and Company B receives 1 order per week. In this case, Company A has double the orders per week, which drives two positive outcomes:

The restaurants in Company A’s network will be more likely to “stick”
Company A will better utilize its delivery network

Both of these factors are key for fueling network effects that help Company A build a moat that strengthens defensibility.

SaaS Models:

Defensibility in SaaS is typically driven by three dynamics that increase switching costs as engagement deepens.

1. Product mastery - When end users engage deeply with a SaaS product, over time they begin to master it. They learn the esoteric features and the keyboard shortcuts, and this deep knowledge allows them to be drastically more productive using the software. When this mastery is combined with easy access to their data, the cost to the user of switching products becomes even higher.

2. Organizational embedding - This happens when companies embed a SaaS product into their product or internal workflow. Through embedding, the product becomes an integral piece of how these companies operate and produce results, making it expensive to “rip out.”

HubSpot, offers a great example of a SaaS product that has used this strategy. As it brought more of its marketing automation customers onto its sales automation tool also, it deepened customer engagement and increased the pain customers felt in switching to a new tool.

3. Network effects - These occur when adding more users improves the product experience, and, again, make it more painful to switch to a competing product. Slack is a good example of a SaaS product with network effects that create switching costs. If an entire team is actively using Slack, the administrator is much less likely to switch to another chat product.

One thing to keep in mind here is that, in general, SaaS products have weaker network effects than B2C products like Facebook or Instagram. It would be much harder for a Facebook user to move their whole network than for an administrator to choose a competing SaaS product.