As a product owner, you want to understand how users are interacting with your product, in order to improve adoption and engagement, but also align the product’s success with business objectives.
Your role is to shape a product to fit a market’s need. And it’s a race to get there. Then you want to scale that product, and it’s also a race to get there. Needless to say that speed is of the essence.
That’s why we believe that analytics is essential in any product-to-market strategy. Not only does it orient your decision making, but it allows you to accelerate your development cycles.
In this article, we’ll go over 4 key topics:
- Understanding your product
- Improving your product
- Measuring your product
- Empowering users
This article is in continuous progress, as it is a reflection of Lantrns Analytics‘ approach to product analytics. To not miss any important updates on this article, or just to keep track of our thought process, subscribe to our Product Analytics newsletter.
Understanding Your Product
How are customers using your product? What do their journeys look like? How do you fit in that journey? Why are they behaving the way they do? What keeps them engaged? Or what’s pushing them out?
As a Product Owner, knowledge is key. What are the forces at play within your product, what are the dynamics and their outcomes.
That knowledge can be accessed through various means, such as customer interviews. We think of product analytics as a necessary component in tightening your feedback loop.
That kind of knowledge informs product strategy. So let’s take a 10,000 view of what is knowledge in regards to product management and how that constitutes the core of product analytics.
No matter what sort of digital product (platform, marketplace, internal BI platform, community, etc.) you’re managing, there is a dynamic that is at the core of your business model and that drives growth.
Of course, the product itself and its users are core to this dynamic. And solely focusing on those 2 forces can get you pretty far. But there are other, more subtle forces that contributes to your overall dynamic and that shouldn’t be overlooked.
Product forces are the building blocks to understanding your product.
Think of marketplaces. Part of its dynamic is to increase desirability and value on both sides (supply and demand) in an equilibrium that makes it enticing for both sides to engage more, based on the health of the other side’s engagement. The forces here are not only the users, but building groups of users that are attractive to the other side.
Think of Upwork. This is a pretty cool marketplace, where freelancers are bidding on projects from contract-givers. But imagine if only one side was active, while the other one was lethargic. There wouldn’t be any market. Simple as that. Contract-givers turn to Upwork because of the number and quality of freelancers, and freelancers turn to Upwork for the number and quality of contract-givers.
Another important force is pricing. How is that contributing to the type of users that are engaging and sticking around with your product. Are those the ones that fits well in your overall strategy?
Each product has their own set of forces that interacts with each other and contributes to your product’s dynamics. The ones we’ve talked above are just a fraction of what might be contributing to the overall dynamic of your product.
Having mapped out the forces at play in your product, you have to start describing explicitly what you already, intuitively know.
Of course you have a good grasp of what are the dynamics within your product. Especially if you are now scaling that product. It means you understood what dynamic helped you realize your product’s mission and how to nurture that dynamic.
But the trouble now is that, well, you’re scaling. It’s not enough for you to instinctively know what’s working. If your product is scaling, your team probably is also. And that means that growing a product is no more the work of one person. Growth strategy is now a group function. What was implicitly known now needs to be explicitly described.
Product dynamics are how your building blocks are interacting together.
A very well known and documented dynamic is user engagement. Some companies focus only on that dynamic and have a north star metric associated to it. How are your users interacting with your product – that’s a pretty straightforward and much analyzed/discussed dynamic.
But it’s having a narrow vision of your product’s dynamics to only focus on this.
Maybe a force that impacts your product dynamic is external to it. For example, it might be important in your growth strategy to rely on organic referencing coming from social medias. The dynamic here is an interaction between social media producers and social media readers. How is the first one influencing the second to consider trying your product. If you’re scaling your product, chances are that this is a dynamic that is really important to you as it would allow you to increase your number of users without skyrocketing your cost of acquisition.
Another dynamic that could be important to you is how discounts impact renewed business. Discounting in itself is an important force. How you leverage it might have an important impact on the number of transactions from your users. But it might also have a negative effect as it creates an expectation and could lower average profit from transaction as users might wait for discounted items to make purchases.
Again, because there is such a variety of forces, there is also an even more important number of dynamics at play within your product. Explicitly mapping them out is again essential to your growth and analytical strategy.
A word of caution though, and we’ll dive deeper into this later, not all dynamics are created equal. Or, they might not be equally important at each stage of your product’s growth. Knowing about each dynamic is important, but you can’t focus on all dynamics at once.
Dynamics yields outcomes. They are observable. You can capture those outcomes, monitor them, analyze them, act on them, etc.
Product outcomes is the manifestation of how forces interact with each other.
As we’ve discussed above, if you’re a product owner that is in scaling mode, it’s not enough anymore to intuitively know about forces and dynamics. You need tangible information to discuss with your team. And you also most probably need tangible info to pass on to would-be investors who are interested in the potential of your product.
Nothing conveys potential more than quality metrics that demonstrates healthy dynamics.
So, product outcomes are what most of us talk about whenever we start looking into product analytics. And that’s what it should be. But without prior understanding of forces, dynamics and which ones are important in your stage of growth, then all outcomes might seem equally important.
If your product is a media portal, there’s a good chance that attention span is crucial. You want to have content that pulls readers in and keeps them interested. Core forces here are articles and users. Their dynamics yields attention-span as an outcome.
So how would you measure attention span? Of course, you could just look at time on page per user. Number of hits. But is that the full story?
Below is an image taken from “The Lifespan of News Stories”, which looks at how news stories are staying in the public eye.
The outcome here is how much a story stays relevant throughout time. So much so, that users are actively searching for that topic on your website. How can you leverage that sub-force, which are topics, to stimulate the dynamic between content and readers, and therefore increase attention span?
There are so many outcomes we could look into, but at this point, once you know which dynamics are important to you, it’s a small leap to know which outcomes you should monitor.
Improving Your Product
Having defined what knowledge is to a product owner and how analytics reflects pieces of that knowledge, we want to act on it.
This is what allows you to accelerate your development cycle and move faster towards winning the market.
Watch Out For Vanity Metrics
At this stage of your product’s growth, knowing the forces, dynamics and outcomes that are important is key to your growth and analytics strategy. As we’ve mentioned, it keeps your team focused and your investors interested.
An analytics strategy that solely focus on outcomes without asking those core questions is at risk of making bad product improvement decisions.
Only when those core questions are asked and answered can data points tell a story of how well a product is filling its mission.
A special kind of metrics that we can be fooled into focusing on without foundational knowledge are vanity metrics.
Eric Ries, author of Lean Startup, tells us why we should be careful of vanity metrics:
“This is the curse of vanity metrics, numbers which look good on paper but aren’t action oriented: website hits, message volume, or “billions and billions served.” They look great in a press release, but what do they accomplish?”
Eric Ries, “Entrepreneurs: Beware of Vanity Metrics“
In our knowledge framework, vanity metrics only shows a single side of outcomes, but not the underlying dynamics that tells the whole story.
Essentially, analytics should help illuminate the streams of data from your product forces, how they interact with each other and the outcomes that emerges from those dynamics.
Why? Because you want to monitor your product’s dynamics. Because that allows for observations and group discussions. And because you can then act on those forces above, see how it impacts dynamics, learn from those experiments, retreat or double down, rinse, repeat.
The Power Of Ratios
Keeping away from vanity metrics has led to multiple strategies. We believe that the Lean Analytics book provides the richest and simplest answers to that problem.
Essentially, it’s about defining metrics as ratios. Because ratios represent the dynamic between forces.
For example, those 2 metrics taken individually are meaningless:
- Number of Users
- Number of Pageview
But take them together, and you now have a ratio that gives you insight into a dynamic (interest in your product’s content): Number of Pageviews / Number of Users.
Event if those 2 numbers rise individually in time, it’s their interaction that tells you how well your product is performing. And its their interaction that tells you what to improve.
Focusing On What Matters
Now that you know how to create meaningful metrics, you need to focus on the right one. As we stated previously, not all dynamics are created equal and we want to focus on metrics that are important for us at this moment of our product’s development.
There are quite a few approaches to how to improve a product, but one approach that clearly focuses on all aspects of your product’s ecosystem is the “Product-Market Fit Pyramid” by Dan Olsen.
If you want a quick intro to his ideas, we highly recommend “Mastering the Problem Space to Achieve Product-Market Fit” at Mind The Product SF 2018
Here’s a quote from it:
“There is a set of universal conditions that needs to hold true if you want to have product-market fit.”
So, what does each layer of the Product-Market Fit Pyramid represent?
From foundation to top of pyramid:
- Target customers – who are we trying to build value for?
- Understand their Needs – what are those customers unserved needs we want to focus on?
- Value proposition – how are we filling those needs and why we’re better than anyone else
- Feature set – what are the functionalities that will convey benefit to our users
- User experience – how will users interact with product to get the benefits
To get towards Product-Market fit, you need to get those 5 layers right. It’s about formulating and validating each layer, from foundation (target customer) to top (UX).
Now, why do products fail? You probably guessed it (or watched it), it’s because product owners tend to focus too much on the product layers. But the market layers are really what’s important to understand first.
We do this because we live in a solution space. We’re more interested in the solution than the problem.
The main point here is that depending on your stage of growth, you need to measure accordingly. And that is what the One Metric That Matters (a similar idea is the North Star metric) is all about in Lean Analytics.
The idea is to have laser-focus to learn, shape and measure fast. To go about it one key metric at a time, based on your growth stage.
Learn, Shape and Measure
Frameworks such as Lean Startup has speed as the core tenet of their approach. How can you go through the cycle’s build, measure and learn as fast as possible… and accelerate?
As you gain insights on your user’s behaviours, you go through the cycle, incrementally and intelligently improving your product.
Having strong knowledge of the forces at play and their dynamics is the fundamental requirement here. Your metrics are only there to validate hypothesis you’ve formed, or invalidate them, or reveal something you hadn’t suspected.
That’s what learning is about. It’s about reinforcing an understanding you already have of your product. It’s about raising questions and forming hypothesis that you want to test out in a future development cycle.
To shape a product is to manipulate its dynamics and to try to influence outcomes.
If experiences are about infirming a null hypothesis, it has to be that those hypothesis exist within a theoretical framework. The product strategy is just that, with underlying assumptions that needs to be invalidated or confirmed.
Again, going back to Lean Startup and Lean Analytics, this is one of the core ideas behind those frameworks. By mapping out your product’s business model on a Lean Canvas, you are in fact mapping out the main assumptions behind your strategy towards product-market fit.
Here’s a template for that Lean Canvas, taken from Leanstack.com.
If you’re familiar with the Lean framework, then you’re most probably familiar with the Lean Canvas. This is a quick way to map out the important components of your product’s business model. Underneath it are the forces at play, their dynamics and the desired outcomes.
This is your theoretical model of how you think your product will win the market. But learning is about infirming or confirming your hypothesis, the underlying assumptions. This is where experiments are crucial to accelerating your Learn – Shape – Measure cycle.
Want to test out an idea? Formulate a question, an hypothesis, set up a treatment, identify the metric that should be impacted by that treatment, split your users for an A/B testing, apply treatment to one group, let it run, see how the metric’s results differ between groups, validate that the test was statistically significant, etc. With that experimental cycle, you can then revisit your business model and its underlying assumptions.
Have an analytics infrastructure and a solid knowledge of your product’s forces, dynamics and outcomes is crucial to experiment frequently and improve your product quickly.
Next up, we’ll dive into the heart of what an analytics infrastructure is and how to build it.
Measuring Your Product
Your strategy defined, you can standardize data collection by capturing, cleaning up, transforming, enhancing and ultimately storing your events in a data warehouse.
First thing first, you need to define what you’ll track. That’s where knowledge of our product will be helpful. The goal is not the track everything, but only what’s important.
A tracking plan helps you understand the motivation behind why an individual becomes a customer, the process of how he becomes one, stays one and promotes your product.
Such a tracking plan becomes the analytics backbone to your online product’s growth strategy.
So what are the most important events in your user’s journey? There are quite a few approaches to this, but arguably the most popular one is the one defined by the Pirate Metrics (original slides by Dave McClure here).
Why “Pirate” metrics, you ask? Because AARRR 🙂 Here’s a visual to explain.
Yes, another pyramid. Again, there’s a linearity in how you should proceed when deciding on which metric to focus on. Whereas the previous pyramid we saw (product-market fit pyramid) mapped the health of your product, this pyramid focuses more on the health of user’s journeys. We tend to think of those AARRR metrics as supporting indicators for the health of your overarching metrics.
So what does a tracking plan look like? There are a couple of takes on this, but our preferred method right now is the simple template provided by Segment.
Having a tracking plan defines a unique way of tracking user events throughout multiple touchpoints. Again, the goal is not to have many events to track, but only a core set of events that will allow you to measure the health of user’s journeys and overall product’s health.
The building block of your product analytics infrastructure is user behaviour, or otherwise known as events. It should aim to provide quality data and insights about your users so you can improve your entire funnel: from acquisition, to activation, to retention, to referral and to revenue.
There are many attributes to how users interact with your product (type, length, time between, relationships with other interactions, etc.), and that should be the main structure of each datum.
An analytical infrastructure feeds itself on those interactions. There are transformation + enhancement + aggregation + etc. layers on top of it, but we are essentially grabbing all product outcomes and pushing those towards a consumable data repository (usually in the form of a data warehouse) that can then be queried and used by many analytical services.
Here’s how we like to set up our product analytics architectures at Lantrns Analytics.
This stack accomplishes a few things:
- It first defines user events so that we always have a single definition throughout your analytics (Tracking Plan)
- It captures data from sources and moves it (Stitch and Segment) to a staging area in your data warehouse (Snowflake)
- It transforms that data (dbt) directly in your data warehouse and creates entity tables consumable by multiple analytical tools.
We have a set of principles that leads our approach to the development of analytical stacks:
- Agility – We work with product owners that want to learn and act fast. It is thus important to build analytical stacks that be transformed quickly to adapt to a product owner’s needs.
- Scalability – Your aim is to grow your product and we want our analytical stacks to grow with you. Our stacks are built to scale effectively and effortlessly.
- Modularity – We don’t work with big enterprise softwares. We think modern analytics stacks should be modular as each piece play a specific role in the overall architecture and does it well.
- Opinionated – And we are opinionated in the selection of those modules. There are endless possibilities, but we have our preferences. This allows us to work with fewer modules, but work better with them, and pass those benefits to our own clients.
Single Source of Truth
Your analytics architecture’s role is to grab streams of user data and transform it in a single source of truth.
The data warehouse becomes that single source of truth and can be consumed through various means: analytical SaaS tools, BI softwares, programming languages, etc.
You may like to stick with R or Python for custom analysis, do all your visualizations in Tableau, Power BI or Looker, or just plainly send your data to your favorite SaaS tool(s). Whatever the UI, you will always analyse the same single source of truth, with all the flexibility you require.
In the end, whatever the tool you prefer to gain knowledge from that source of truth, all the hard work you’ve put into building that source of truth should allow you to focus on a few metrics that are key to guide the development of your product.
Now that we’ve gone through all the foundational work to create a product analytics strategy and infrastructure, we want to take the time to go through a few remaining topics that are not directly associated to product analytics, but are as important in our opinion.
Let’s first look at this graphic that provides a high-level view of what we’ve seen so far. That is, the bottom 2 layers.
We think of this as the product’s ecosystem. And at the top of it, we of course want to improve our products to improve business value.
But product improvement shouldn’t only have an impact on business value. Product improvement should in fact also be about empowering your users.
As Jesse Weaver said in “It’s Time for Digital Products to Start Empowering Us“:
“Utility alone won’t assuage us. We want empowerment. We want to be better people. We want technology to enhance our capabilities and increase our sense of agency without dictating the rhythm of our lives.”
Another very important topic is privacy. We might only talk about it at the end of our guide, but it should be central to product improvement.
The reason why respecting the privacy of your users is so important is that it is foundational to empowering them.
Rethinking user empowerment requires rethinking how we collect and manage data. It goes further than just complying with GDPR (General Data Protection Regulation). It raises the question of how privacy should be at the core of product management.
In regards to GDPR, this is an important topic that cannot be thoroughly covered here as it is always-evolving and pretty complex. But in essence, the GDPR wants to give personal data back to individuals. That’s through key obligations to be put into practice by all affected organisations. The non-exhaustive list of obligations is to first get explicit consent from individuals to collect profile and behavioural data about them; apply pseudonymisation to collected data; provide a right to erasure to individuals; provide a right to gain access to all personal data collected by an organisation; etc.
But one important concept that GDPR and other such laws are promoting is the one of privacy by design.
As defined in Wikipedia:
“Privacy by Design is about embedding data protection controls into systems that process personal data at all stages of system development, including analysis, design, implementation, verification, release, maintenance and decommission.”
In practice, that means adopting certain data protection practices, such a pseudonymisation or de-identification, as well as providing users with mechanisms that gives them control over their data.
If the GDPR is anything, it’s an opportunity for all organisations to be more respectful of their user’s privacy. It’s a worthy objective to give ourselves. And it can only contribute to empowering your users.
Better understanding your product’s users should have for purpose to engage intelligently and help them move forward on their journey.
Guiding users on their own journey is also empowering users.
Remember that tracking plan? It is an over-simplification of what a customer’s journey is, their journeys being more complex than this. But it’s a blueprint of what their journeys look like. It’s way more than just what’s mapped through AARRR metrics, and as such you should eventually seek to use product analytics to orchestrate your user’s journeys.
It’s to act on the insights you have at your disposal. The art of acting when needed – to the point of automating personalization with such a tool as Segment Personas or Snowplow React.
Product management is not a science, but there is method to it. And as with any other profession, the right process with the right tools ensures higher quality. Analytics is not to be considered the holy grail, but if you can use it efficiently in your development cycle, it has the benefit of providing clean feedback quickly to guide you towards your next cycle of development.
What are the first steps you should take? It all depends on the size of your organization and the resources at your disposal.
If you have the time, the interest and the technical know-how, you can definitely take on the challenge yourself. If you have available resources, then having a dedicated person or team is definitely the best approach. If that’s not an option, you can hire externally with freelancers and consulting agencies.
At Lantrns Analytics, we are taking a somewhat novel approach to this. By providing a mix of SaaS and personalized service, we believe that this is how we can best serve product owners who either do not have time to do it themselves, do not have the team to support them, nor the resources to mandate an expensive agency.
We like to think of ourselves as your dedicated team of product analytics experts.
If you’d like more info on how our Product Analytics Core packaged service works, have a look here.
In conclusion, we believe that product analytics is central to winning the market. It allows product owners to have a mechanism to understand how individuals are using their product, how they should shape and improve their products, and what dynamics should be enhanced to achieve product-market fit and further empower users.