As a product owner, you want to understand how users interact with your product. This will improve adoption and engagement but will also align the product’s success with business objectives.
Your role is to shape a product to fit a market’s need. And it’s a race to get there. Then you want to scale that product, and it’s also a race to get there. Needless to say, that speed is of the essence.
That’s why we believe that analytics is essential in any product-to-market strategy. Not only does it orient your decision making, but it allows you to accelerate your development cycles.
In this article, we’ll go over four key topics:
- Understanding your product
- Improving your product
- Measuring your product
- Empowering users
This article is in continuous progress, as it is a reflection of Lantrns Analytics‘ approach to product analytics. You may want to subscribe to PAN, our Product Analytics Newsletter, to keep track of our thought process and to avoid missing any important updates on our approach.
Understanding Your Product
How do customers use your product? What does their buying journey look like? How do you fit in that journey? Why do they behave the way they do? What keeps them engaged? Or what’s pushing them out?
As a product owner, knowledge is key. You may ask yourself some questions: what are the forces at play regarding your product? What are the dynamics and outcomes?
That knowledge can be accessed through various means, such as customer interviews. We think of product analytics as a necessary component in tightening your feedback loop.
That kind of knowledge dictates your product strategy. So let’s take a look at the big picture of knowledge with regards to product management and how this information constitutes the core of product analytics.
No matter what sort of digital product (platform, marketplace, internal BI platform, community, etc.) you manage, there is a dynamics that is fundamental to your business model and that drives growth.
Of course, this dynamics is driven by the product itself and its users. And solely focusing on those two forces can get you pretty far. But there are other, more subtle, forces that contribute to your overall dynamics and that shouldn’t be overlooked.
Product forces are the building blocks to understanding your product.
Think of marketplaces. Part of their dynamics is to increase desirability and value on both sides (supply and demand) in an equilibrium that makes it enticing for both sides to engage more, based on the health of the other side’s engagement. The forces here are not only the users but groups of users that are attractive to the other side.
Another example: Upwork. This is a pretty cool marketplace, where freelancers are bidding on projects from contract givers. Imagine if only one side was active, while the other one was lethargic. There wouldn’t be any market. Simple as that. Contract givers turn to Upwork because of the number and quality of freelancers, and freelancers turn to Upwork for the number and quality of contract givers.
Another important force is pricing. How is this contributing to the type of users who engage with your product and stick around? How well these users fit in your overall strategy?
Each product has its own set of forces and contributes to your product’s dynamics. The ones we’ve talked above are just a fraction of what might be contributing to the overall dynamics of your product.
Having mapped out the forces at play for your product, you need to start describing explicitly what you already intuitively know.
Of course, you have a good grasp of the connections within your product. You now understand what dynamics helped you realize your product’s mission and how to nurture those dynamics. Even with all this knowledge, scaling a product could become quite a challenge.
If your product is scaling, it often means your team will shortly scale too. Product growth is no longer a solo task, it is a group function.
It’s not enough for you to instinctively know what’s working. What was implicitly known by one person now needs to be explicitly described to a whole team.
Product dynamics are how your building blocks are interacting together.
A very well-known and documented dynamics is user engagement. Some companies focus only on that and create a North Star metric associated with it. How do your users interact with your product? This question, pretty straightforward, has been analyzed/discussed many times to elaborate on product dynamics.
But you would have a narrow vision of your product dynamics if you only focus on this element.
Perhaps, maybe an external force influences your product dynamics. For example, it might be important in your growth strategy to rely on organic referencing coming from social media. The dynamics here is an interaction between social media producers and social media readers. We can summarize their rapport with this question: which keywords or sentences the producer must write to influence the reader (to try the product)? If you’re scaling your product, chances are the answer to this question is vital to increase your number of users without skyrocketing your cost of acquisition.
Another dynamics that could be important to you is how discounts impact renewed business. Discounting in itself is an important force. How you leverage it might have an important impact on the number of transactions from your users. But it might also have a negative effect as it creates an expectation and could lower average profit from transactions as users might wait for discounted items to make purchases.
Again, because there is such a variety of forces at play in your product dynamics. Therefore, mapping them out is again essential to your growth and analytical strategy.
A word of caution though: not all dynamics are created equal. Or, they might not be equally important at each stage of your product’s growth. Knowing about each dynamics is important, but you can’t focus on all dynamics at once. We’ll dive deeper into this later.
Dynamics yield outcomes. They are observable. You can capture those outcomes, monitor them, analyze them, act on them, etc.
Product outcomes are the manifestation of how forces interact with each other.
If you’re a product owner that is in scaling mode, it’s not enough anymore to intuitively know about forces and dynamics. You need tangible information to discuss with your team. You also need this concrete info to pass on to would-be investors who are interested in the potential of your product.
Nothing conveys potential more than quality metrics that demonstrate healthy dynamics.
So, product outcomes are what most of us talk about whenever we start looking into product analytics. And that’s what it should be. But without prior understanding of forces, dynamics and which ones are important in your stage of growth, then all outcomes might seem equally important.
If your product is a media portal, there’s a good chance that attention span is crucial. You want to have content that pulls readers/viewers in and keeps them interested. Core forces here are articles/videos and users. Their dynamics yields attention span as an outcome.
So how would you measure attention span? Of course, you could just look at Time on Page or Pageviews. But is that the full story?
Below is an image taken from “The Lifespan of News Stories”, which looks at how news stories remain in the public eye.
Source: The Lifespan of News Stories
The outcome here is how much a story stays relevant throughout time. So much so, that users are actively searching for that topic on one’s website. How can you leverage that sub-force (the said topics) to stimulate the dynamics between content and readers, and therefore increase the attention span?
There are so many outcomes we could look into. Because you now know which dynamics are important to you, you also know which outcomes you should monitor.
Improving Your Product
Having defined what knowledge is necessary to a product owner and how analytics reflects pieces of that knowledge, we want to act on it.
This is what allows you to accelerate your development cycle and move faster towards winning the market.
Watch Out For Vanity Metrics
At this stage of your product’s growth, knowing the forces, dynamics and outcomes that are important is key to your growth and analytics strategy. As we’ve mentioned, it keeps your team focused and your investors interested.
An analytics strategy that solely focuses on outcomes without asking those core questions is at risk of making bad product improvement decisions. Once you find answers to these core questions, you can define data points and tell a story on how well your product is filling its mission.
What is a vanity metric?
Eric Ries, author of Lean Startup, tells us why we should be careful of vanity metrics:
“This is the curse of vanity metrics, numbers which look good on paper but aren’t action oriented: website hits, message volume, or “billions and billions served.” They look great in a press release, but what do they accomplish?”
Eric Ries, “Entrepreneurs: Beware of Vanity Metrics“
In your knowledge framework, vanity metrics only show a single side of outcomes, but not the underlying dynamics that tells the whole story.
Essentially, analytics should help enlightens the streams of data from your product forces, how they interact with each other and the outcomes that emerge from those dynamics.
- Because you want to monitor your product’s dynamics.
- Because that allows for observations and group discussions.
- Because you can then act on those forces above, see how it impacts dynamics, learn from those experiments, retreat or double down, rinse, repeat.
The Power Of Ratios
Keeping away from vanity metrics leads to multiple strategies. We believe that the Lean Analytics book provides the richest and simplest answers to that problem.
Essentially, it’s about defining metrics as ratios. Because ratios represent the dynamics between forces.
For example, those two metrics taken individually are meaningless:
- Number of Users
- Number of Pageviews
But take them together, and you now have a ratio that gives you insight into a dynamics (interest in your product’s content): Number of Pageviews / Number of Users.
Even if those two numbers rise individually in time, it’s their interaction that tells you how well your product is performing. And it’s their interaction that tells you what to improve.
Focusing On What Matters
Now that you know how to create meaningful metrics, you need to focus on the right one. As we stated previously, not all dynamics are created equal and you want to focus on metrics that are important for you at this moment of your product’s development.
There are quite a few approaches to how to improve a product, but one approach that clearly focuses on all aspects of your product’s ecosystem is the “Product-Market Fit Pyramid” by Dan Olsen.
Source: The Playbook for Achieving Product-Market Fit, by Dan Olsen
If you are unfamiliar with this concept, we highly recommend “Mastering the Problem Space to Achieve Product-Market Fit” at Mind The Product SF 2018
Here’s a quote from it:
“There is a set of universal conditions that needs to hold true if you want to have product-market fit.”
So, what does each layer of the Product-Market Fit Pyramid represent?
From the foundation to the top of pyramid:
- Target customers: whom are we trying to build value for?
- Understand their needs: what are those customers’ unserved needs we want to focus on?
- Value proposition: how are we filling those needs and why we’re better than anyone else?
- Feature set: what are the functionalities that will convey benefit to our users?
- User experience (UX): how will users interact with the product to get the said benefits?
To get towards Product-Market fit, you need to get those five layers right. It’s about formulating and validating each layer, from the foundation (target customer) to the top (UX).
But, why do products fail? You probably guessed it (or watched it!), it’s because product owners tend to focus too much on the product layers. But what is really important at first are the market layers.
We do this because we live in a solution space. We’re more interested in the solution than the problem.
Depending on your current stage of growth, you need to measure accordingly. And this is what the One Metric That Matters (a similar idea is the North Star metric) is all about in Lean Analytics.
The idea is to laser focus to learn, shape and measure fast. To go about it one key metric at a time, based on your growth stage.
Learn, Shape and Measure
Frameworks, such as Lean Startup, has speed as the core tenet of their approach. How can you go through the cycle’s build, measure and learn as fast as possible… and even accelerate the process?
As you gain insights on your users’ behaviours, you go through the cycle, incrementally and intelligently improving your product.
Having strong knowledge of the forces at play and their dynamics is the fundamental requirement here. Your metrics are only there to validate hypotheses you’ve formed, or invalidate them, or reveal something you hadn’t suspected.
That’s what learning is about. It’s about reinforcing an understanding you already have of your product. It’s about raising questions and forming hypotheses that you want to test out in a future development cycle.
To shape a product is to manipulate its dynamics and to try to influence outcomes.
If experiences are about refuting a null hypothesis, it has to be that those hypotheses exist within a theoretical framework. The product strategy is just that, with underlying assumptions that need to be invalidated or confirmed.
Again, going back to Lean Startup and Lean Analytics, this is one of the core ideas behind those frameworks. By mapping out your product’s business model on a Lean Canvas, you are in fact mapping out the main assumptions behind your strategy towards product-market fit.
Here’s a template for that Lean Canvas, taken from Leanstack.com.
Source: Lean Canvas from Leanstack.com
If you’re familiar with the Lean Framework, then you’re most probably familiar with the Lean Canvas. This is a quick way to map out the important components of your product’s business model. Underneath it are the forces at play, their dynamics and the desired outcomes.
This is your theoretical model of how you think your product will win the market. But learning is about refuting or confirming your hypothesis, the underlying assumptions. This is where experiments are crucial to accelerating your Learn – Shape – Measure cycle.
Want to test out an idea?
- Elaborate the test – Formulate a question, a hypothesis, set up treatment, identify the metric that should be impacted by that treatment
- Perform the test – Split your users for an A/B testing, apply treatment to one group, let it run.
- Evaluate the test – See how the metric’s results differ between groups; validate that the test was statistically significant, etc.
With that experimental cycle, you can then revisit your business model and its underlying assumptions.
Have an analytics infrastructure and a solid knowledge of your product’s forces, dynamics and outcomes is crucial to experiment frequently and improve your product quickly.
Next up, we’ll dive into the heart of what an analytics infrastructure is and how to build it.
Measuring Your Product
With your strategy defined, you can standardize data collection by capturing, cleaning up, transforming, enhancing and ultimately storing your events in a data warehouse.
First thing first, you need to define what you’ll track. That’s where knowledge of our product will be helpful. The goal is not the track everything, but only what’s important.
A tracking plan helps you understand the motivation behind why an individual becomes a customer, the process of how he becomes one, stays one, promotes your product and prevents data excess tracking. Such a tracking plan becomes the analytics backbone to your online product’s growth strategy.
It is necessary to understand what are the most important events in your user’s journey? To do so, there are quite a few approaches, but arguably the most popular one is the one defined by the Pirate Metrics (original slides by Dave McClure).
Why “Pirate” metrics, you ask? Because AARRR 🙂 Here’s a visual to explain.
Soùrce: AARRR Framework- Metrics That Let Your StartUp Sound Like A Pirate Ship, by Melanie Blake
Yes, another pyramid. Again, there’s linearity in how you should proceed when deciding on which metric to focus on. Whereas the previous pyramid we saw (product-market fit pyramid) mapped out the health of your product, this pyramid focuses more on the health of the user’s journeys. We tend to think of those AARRR metrics as supporting indicators for the health of your overarching metrics.
So what does a tracking plan look like? There are a couple of takes on this, but our preferred method right now is the simple template provided by Segment.
Having a tracking plan defines a unique way of tracking user events throughout multiple touchpoints. Again, the goal is not to have many events to track, but only a core set of events that will allow you to measure the health of the user’s journeys and overall product’s health.
The building block of your product analytics infrastructure is user behaviour, or otherwise known as events. It should aim to provide quality data and insights about your users so you can improve your entire funnel: from acquisition to activation, to retention, to referral and to revenue.
There are many attributes on how users interact with your product (type, length, time, relationships with other interactions, etc.), and that should be the main structure of each datum.
An analytical infrastructure feeds itself on those interactions. There are transformation + enhancement + aggregation + etc. layers on top of it, but we essentially grab all product outcomes and push those towards a consumable data repository (usually in the form of a data warehouse) that can then be queried and used by many analytical services. We prepare the road for data analytics.
Here’s how we like to set up our product analytics architectures at Lantrns Analytics.
This stack accomplishes a few things:
- It defines user events so that we always have a single definition throughout your analytics (Tracking Plan).
- It captures data from sources and moves it (Stitch and Segment) to a staging area in your data warehouse (Snowflake).
- It transforms that data (dbt) directly in your data warehouse and creates entity tables consumable by multiple analytical tools.
We have a set of principles that leads our approach to the development of analytical stacks:
- Agility – We work with product owners that want to learn and act fast. It is thus important to build analytical stacks that could be transformed quickly to adapt to a product owner’s needs.
- Scalability – Your aim is to grow your product and we want our analytical stacks to grow with you. Our stacks are built to scale effectively and effortlessly.
- Modularity – We don’t work with big enterprise software. We think modern analytics stacks should be modular as each piece plays a specific role in the overall architecture and does it well.
- Boldness – We know which modules to select when it comes to product analytics. There are endless possibilities, but we have our preferences. This allows us to work with fewer modules, but work better with them, and pass those benefits to our own clients.
Single Source of Truth
Your analytics architecture’s role is to grab streams of user data and transform it into a single source of truth.
The data warehouse becomes that single source of truth and can be consumed through various means: analytical SaaS tools, BI software, programming languages, etc.
You may like to stick with R or Python for custom analysis, do all your visualizations in Tableau, Power BI or Looker, or just plainly send your data to your favourite SaaS tool(s). Whatever the UI, you will always analyze the same single source of truth, with all the flexibility you require.
In the end, whatever the tool you prefer to gain knowledge from that source of truth, all the hard work you’ve put into building that source of truth should allow you to focus on a few metrics that are key to guide the development of your product.
Now that we’ve gone through all the foundational work to create a product analytics strategy and infrastructure, we want to take the time to go through a few remaining topics that are not directly associated to product analytics but are as important in our opinion (remember, we are opinionated!)
Let’s first look at this graphic that provides a high-level view of what we’ve seen so far, ie. the bottom two layers.
We think of this as the product ecosystem. And at the top of it, we, of course, want to improve our products to improve business value. But product improvement shouldn’t only have an impact on business value. Product improvement should in fact also be about empowering your users.
As Jesse Weaver said in “It’s Time for Digital Products to Start Empowering Us“:
“Utility alone won’t assuage us. We want empowerment. We want to be better people. We want technology to enhance our capabilities and increase our sense of agency without dictating the rhythm of our lives.”
In any business, a very important topic is privacy. We might only talk about it at the end of our guide, but it should be central to product improvement.
The reason why respecting the privacy of your users is so important is that it is foundational to empowering them.
Rethinking user empowerment requires rethinking how we collect and manage data. It goes further than just complying with GDPR (General Data Protection Regulation). It raises the question of how privacy should be at the core of product management.
Regarding user empowerment, all product owners should consider the concept of privacy by design. As defined by Wikipedia:
“Privacy by Design is about embedding data protection controls into systems that process personal data at all stages of system development, including analysis, design, implementation, verification, release, maintenance and decommission.”
In practice, that means adopting certain data protection practices, such a pseudonymization or de-identification, as well as providing users with mechanisms that give them control over their data.
If the GDPR is anything, it’s an opportunity for all organizations to be more respectful of their users’ privacy. It’s a worthy objective to give ourselves. And it can only contribute to empowering your users.
To make it short, the General Data Protection Regulation wants to give personal data back to individuals. This regulation states key obligations to be put into practice by all affected organizations. Among the list of obligations:
- Get explicit consent from individuals to collect profile and behavioural data about them;
- apply pseudonymization to collected data;
- provide a right to erasure to individuals;
- provide a right to gain access to all personal data collected by an organization.
Better understanding your product’s users should have for purpose to engage intelligently and help them move forward on their journey.
Guiding users on their own journey is also empowering users.
Remember that tracking plan? It is an over-simplification of what customers’ journeys are, it’s also a blueprint of what their journeys look like. It’s way more than just what’s mapped through AARRR metrics, and as such, you should eventually seek to use product analytics to orchestrate your users’ journeys.
It’s to act on the insights you have at your disposal. The art of acting when needed – to the point of automating personalization with such a tool as Segment Personas or Snowplow React.
Product management is not a science, but there is a method to it. And as with any other concept, the right process with the right tools ensures higher quality. Analytics is not to be considered the holy grail, but if you can use it efficiently in your development cycle, it has the benefit of providing clean feedback quickly to guide you towards your next cycle of development.
What are the first steps you should take? It all depends on the size of your organization and the resources at your disposal.
If you have the time, the interest and the technical know-how, you can definitely take on the challenge yourself. If you have available resources, the best approach is definitely to dedicate a person within your team. If that’s not an option, you can hire freelancers or consulting agencies.
At Lantrns Analytics, we embrace a somewhat novel approach to this. By providing a mix of SaaS and personalized service, we believe that this is how we can best serve product owners who may not have time to do it themselves, the team to support the change internally, nor the resources to mandate an expensive agency.
We like to think of ourselves as your dedicated team of product analytics experts.
We believe that product analytics is central to winning the market. It allows product owners to have a mechanism to understand how individuals are using their product, how they should shape and improve their products, and what dynamics should be enhanced to achieve product-market fit and further empower users.