How Product Managers Make Metric-Informed Decisions | by Jack Moore | Jan, 2023

How to fill in the gaps of a problem where the numbers don’t tell you everything

Welcome to ProdCo, the creators of fine imaginary solutions. Today you’re taking the seat of the head product strategist at ProdCo, at the helm of a product wherein hungry pedestrians can use a mobile application to summon one of an army of be-scootered ProdCo contract sandwich delivery people to their location with a fresh sandwich. UberEats will be dead by next Thanksgiving.

You, the head strategist at ProdCo, are considering making a change to your user interface that’s going to make it easier for your users to order sandwiches — turning the “buy” button from red to green.

Unfortunately, this is only one of many potential strategic priorities that you’re considering.

How do you know that the change you’re making is the optimal use of your teams’ bandwidth?

Fret not! We’re going to arm you with a tool that is going to simplify your life immensely.

Net Value = Revenue — Cost

Ok, maybe you already knew that. What if we start breaking it down a bit? For our product, profit consists of the value per unit sold & the number of units we can sell

Revenue = (revenue per sandwich) * (number of sandwiches sold per week)

Either of these numbers could be broken down further:

Number of sandwiches sold per week = (# of weekly active app users) * (% of active users that order sandwiches in any given week)

This method of modeling is called Key Metric Decomposition, and it makes it easy to craft an ROI model that allows you to express product impact in terms of relevant business metrics.

In this case, this simple model will allow us to forecast the impact that changing conversion rates might have on profit.

By running this calculation once with your current figures and comparing it with reasonable estimates based on the impact your project could have, you can use this equation to project the net impact that your change might have on your business’s bottom line.

Your colleagues are going to be impressed at your ability to draw a line from product decisions to business outcomes, these decisions sometimes appearing somewhat far apart.

Let’s do some math. For now, we’ll make reasonable assumptions for any variable we don’t care about, and that we don’t expect to change as a direct impact of our work, will not change.

Of course, we’ll state these assumptions explicitly so that they can be challenged by people who know more than we do.

Revenue = (revenue per sandwich) * (# of weekly active app users) * (% of active users that order sandwiches in any given week)

Revenue (baseline) = ($5) * (100,000) * (0.05) = $25,000

So, before we consider implementing this change, 5% of our 100k users are converting per week at a revenue of $5 per sandwich, resulting in $25k per week in

Let’s say that we’ve done some in-depth UX research, and we know that there are some UX improvements we could make that would improve our weekly conversion rate to 6%, up from 5%:

Revenue (after) = ($5) * (100,000) * (0.06) = $30,000

$30k — $25k tells us that this feature change is worth $5k per week. If we wanted to take that further, we might consider annualizing this number so as to make it more comparable to other business initiatives (other than product) that are being considered.

But. What if we didn’t have such rock-solid UX research? After all, research resources are sparse in our startup environment. For this, start with a range. Let’s say that you estimate that your change could have anywhere from a 1% to 2% impact on the rate of sandwich conversion, and rerun the calculations using those aggressive and conservative estimates:

Profit (after, conservative) = ($5) * (100,000) * (0.06) = $30,000

Profit (after, aggressive) = ($5) * (100,000) * (0.07) = $35,000

Comparing these values to our baseline, we see that the range of outcomes is +$5k to +$10k per week, compared to the $25k baseline value we computed earlier. If this range of outcomes includes something that we might find interesting, perhaps a study of the conversion rates using prototype screens might help us tighten the range.

You’re feeling pretty pumped. You think you’ve identified a solid win, and you’re about to phone in the next couple of weeks that you had allocated to strategic planning, when your head of customer service sends you some interesting feedback:

“Spanish-speaking are giving us feedback that our interface is difficult to navigate. Is this a feature we could add?”

You know that there are a solid proportion of Spanish-speaking users in the market who are looking for a sandwich-ordering platform, but you’re not sure whether this new feature will increase conversion of those users enough to change your priorities.

You lament over this decision for only a brief moment, though, before you realize you can augment your model to target this specific question!

Profit = (revenue per sandwich) * (# of Spanish-speaking users in target market) * (conversion rate for Spanish-speaking users)

Your research tells you:

• There are 20k users that you believe would be active users if they had access to a Spanish-language interface
• Every week, 0.5% of these users order sandwiches now every week

You know that your current opportunity presents, at minimum, a $5k per week opportunity, so you reconfigure your model to determine the minimum effect that this feature would need to be competitive with the current roadmap alternative.

Minimum conversion rate = Minimum Profit / ((revenue per sandwich) * (# of Spanish-speaking users in target market)) = ($5k) / (($5) * (20,000)) = 0.05

So, this feature would have to increase conversion to 5%, up from the current 0.5%. Sitting down with your research department, you determine that the range of likely outcomes is a resultant conversion rate of closer to 3%, so you decide to pass on the opportunity from now, all the more confident that the path you’ve chosen for your product is sound.

Key Metric Decomposition is a helpful tool for breaking down complex problems & comparing otherwise-unalike opportunities. It can tell you what areas of research are going to most clarify your ROI projections, and can allow you to present such projections in ways that everyone in your organization can understand.