Decoding Metrics as a Product Manager 📊 | Lessons from an Intern and a Senior PM @ Microsoft
Breaking down how metrics are used from a beginner, interview, and on the job context! I also bring a close mentor and friend, Sammi Li, to show how she values metrics on her job as a PM @ Microsoft!
Hey everyone! Thanks for checking in with this month’s issue of Tyler Talks Product! Before getting into the article, I wanted to celebrate a pretty big milestone for the newsletter which is that we’ve reached 350 readers. 🎉
For those of you that are new here, the following are some issues you may have missed that I think are worth the read:
Why Metrics are SO IMPORTANT ✍🏽
Look at any job posting today and you’ll find that ‘Data Proficient and Comfortable with Metrics’ are some of the most sought-after skills for Product Managers. Day-to-day on the job, PM’s are asked to help establish success metrics for a new feature, understand why a specific metric went down, and find ways to optimize metrics for the business.
Google Product Manager Responsibility:
Yelp Growth Product Manager Responsibility:
Toronto-Based Start-Up Responsibility:
At my current internship, I’m responsible for not only setting the key metrics for my experiments, but for the features that I decided to release as well. It can be dizzying at first. You may ask yourself questions such as; “What part of the funnel do I optimize for?,” “Is this a vanity metric?,” and “Is this metric industry specific?”. Don’t worry though, it takes time to understand metrics. Although it’s a quantitative field, it really is an art. The first step to getting better at metrics is understanding their importance.
How to View Metrics for Beginners 👓
Let’s start from the beginning and ask: what is a metric? A metric is like a pulse check on the health of your body (or in this case your product). Each metric you establish should tell you if the product is succeeding, struggling, or failing in a specific way.
Just like how each of our bodies are different, different companies value different metrics. Here’s how some popular companies establish important metrics for their products:
So now that you know that metrics are important, the hard part now becomes where you decide to implement them in your product. Typically, product managers will use their product funnel to get a general sense of what is important to track. For those unfamiliar, a product funnel expresses the journey a user will take to find out, use, and share your product. The most popular of these funnels is the AAERRR funnel, popularized by Lewis Lin.
Each one of these areas refers to the typical processes that all tech industries should follow. Here is a quick breakdown of each section:
Acquisition - How do you acquire users? (E.g. Metrics - SEO ranking, # of new users per acquisition channel, cost per acquisition)
Activation - How many users successfully sign up for your product? (E.g. Metrics - Drop off % during sign-up, time spent per page, time to sign-up)
Engagement/Retention - How do users interact with your product and do they wish to return? (E.g. Metrics - 90-day return rate, Daily/Weekly/Monthly Active Users, and session time)
Revenue - What business value is derived from your user? (E.g. Metrics - Average Basket Size, Revenue Per User, Conversion % from free to paid)
Referral - What is the virality of your product? (Eg Metrics - Average time to share, # of users who share, reviews)
Lastly, here are some other funnel frameworks that may be of interest to you when learning about metrics:
AIDA (Pre-purchase funnel) - awareness, interest, desire, and action.
REAN (Post-purchase funnel) - reach, engage, activate, nurture.
Using Metrics in the Interview Room 📝
Now that you’re familiar with metrics and where to implement them, you’ll probably have to flex your metric muscles in a product manager interview. Product metric questions take different forms at different companies, for example, Meta calls metric questions product execution questions. Regardless, the focus is on the ability to measure and define the success of a product.
There are three types of questions you’ll encounter in this type of question and they are:
Measuring the success of a product
Root cause analysis
Tradeoff between two metrics
Due to the length of the article, I will not go into the ideal framework of each problem, but instead, talk about example questions, what the question is trying to test, and my favourite video example of a great answer.
Measuring the Success of a Product
Example questions may include:
“Can you define what success looks like for the Airbnb Check-in experience?”
“Define the North Star Metric for Instagram Reels.”
“If you had a dashboard for Uber, what would be on it?”
The interviewer is seeing if you are able to understand the company mission, understand the core user journey, and take that insight and translate it into measurable metrics. A pro-tip for this question is that although it may seem like the correct response is to list as many applicable metrics as possible, the best way to approach this problem is picking 4-6 high-value metrics and having a discussion about each one’s value, edge case, and measurement logistics.
Good Video Example -
Root Cause Analysis
Example questions may include:
“MAU for Facebook Messenger has gone down 10%, why?”
“You’re the PM for Uber and there is a stark drop off during 7pm, why?”
The interviewer is testing your ability to ask meaningful questions and to re-frame the problem in a more approachable form. Although some interviewers may have a definitive root cause, your goal here is to show a logical set of steps to break down a problem and hypothesize a plausible reason for the change in metrics.
Good Video Example -
Trade-Off Between Two Metrics -
Example questions may include:
“You implement a new feature on IG reels, and DAU goes up but session time goes down.”
“There’s a new feature on Youtube comments which makes comments go up but it makes watch time go down. Why?”
The interviewer is trying to find out if you can: 1) understand the impact of each of these metrics, 2) prioritize how to fix this based on a logical criterion, and 3) establish if you know enough about experimentation to test your solution.
Great Video Example -
Where to User Metrics (Real-Life) 😱
Using metrics on the job as a product manager can be daunting. So I reached out to a friend of mine, Sammi Li. Sammi is a senior PM at Microsoft and works on the AI for Teams devices, such as Cortona. I asked her some questions below and she provided these insights on using metrics in an actual product setting.
How are metrics approached in the real world?
How PMs approach which metrics matter varies greatly, but I think being data-driven and being good at goal-setting go hand in hand. Without setting up ambitious yet realistic goals, you don't know what's important to track, and really, what's the purpose of looking at a sea of data when you don't have a north star? At Microsoft, we use OKRs (Objectives and Key Results) to plan what product features we're prioritizing and how we track the results. Simply put, the Objective is the "what" and the Key Results are the "how". Key results are metric-driven which could be revenue, engagement, usage, retention, market share, etc. and they are also time-bound and measurable. After setting up the OKRs, we can then begin to dig in the right spot to get our nuggets of insights about our users (eg. how users are adopting the product, how are they engaging with the product, etc.).
What’s more important; qualitative or quantitative data?
Telemetry data can only tell you part of the story, and oftentimes, it sheds light on the "what" and "when" but not the "why". I think a good data-driven PM is out on the field, rolling up their sleeves, and really building empathy with their users. By talking to your users, you learn more about the intrinsic motivation or the psychology behind the data. In my org, we have customer-driven events and office hours where we get to hear feedback about our product firsthand. This is where we have the opportunity to understand user pain points, ask questions to validate our hypothesis and get ideas for our product roadmap. Whether or not these opportunities are available in your org, a PM should seek out ways to do user interviews, user surveys, etc. to pair the qualitative data with the quantitative data.
What does it mean to be proactive with data as a PM?
Being action-oriented Data is only useful if you do something with it, and I think a good data-driven PM can effectively articulate data, process it, and nurture it into a plan of action. Looking at data is not a one-time thing, and there's a cycle to it; think data -> insights -> storytelling -> action. One of my favourite quotes from Steve Jobs is "To me, ideas are worth nothing unless executed... Execution is worth millions." And I think this goes for how we use data. It's one thing to read it and understand it, but how can you translate it to your business partners or cross-functional organization? How do you use the data to execute a new feature proposal?
I hope that by discussing these three areas of growth, you have enough confidence to begin your data journey! Thanks again to Sammi for taking the time to answer these questions :)
Resources for Growth 📕
Recent Tech News That Might Interest You
Gaming vets promise to make blockchain games fun and sustainable
Daily Crunch: Breaking down Apple’s quarterly numbers
Facebook risks ban in Kenya for failing to stop hate speech
On General Product Prep
Try Exponent: A PM website home to questions, company guides, and helps facilitate 1:1 interviews with other aspiring PMs.
Tech Talk for Non-Developers: A course dedicated to demystifying technical concepts for product managers in a more digestible way. (Fun Fact: I did this course and it helped a ton with my first PM job!)
APM List: APM season is here, keep up to date with PM job postings!
Request a Topic 🎙
Click the button below if you have a topic in mind that you want to explore in product management! It may be the next issue :)
Coffee Program and Thank You ☕️
Thank you for taking the time to read this month’s issue! I’ll see you next month :)
Personal thank you to my editors (Cat, Anne, and Lucus) for taking the time to read each issue before it goes out to the amazing readers!
I’m going to go make some coffee. You should too! Feel free to use the scan below to buy yourself some coffee on me at Starbucks. As always, please be respectful as to the amount you’re purchasing <3
~ Tyler