In the world of digital products, none is as widespread and as easily understood as newsletters. They’re usually not an end in themselves, but just a touchpoint to an overarching business. Their effectiveness can be immensely beneficial to an organization.
Our goal here is to go through how we’ve set up an analytical strategy and infrastructure for Lantrns Analytics newsletter. We’ll go through the following 3 questions:
- What to measure?
- How to measure?
- What to do with the results?
It should be noted that our approach for Lantrns Analytics has been documented in our Guide to Product Analytics. This post on newsletters is just one use case of doing product analytics inspired by our approach.
With that in mind, let’s first define what we want to measure.
What to measure?
What to measure assumes the basic knowledge of a product’s ecosystem. Newsletters might be simple digital products in their structure, but their dynamics can be complex.
Let’s just first map out what we understand of that ecosystem, which will orient us towards what we want to measure for our current stage of growth. Here’s what we believe are the core forces, dynamics and outcomes of our newsletter.
- The circles represent the entities within the ecosystem.
- The inner attributes are the forces.
- The boxes represent the dynamics generated from the relationship between forces.
- The listed items underneath the dynamics are examples of the dynamics’ outcomes.
It’s far from a perfect nor complete model of what this newsletter’s ecosystem is, but at our current stage of growth, it’s more than enough to focus on our OMTM (One Metric That Matters; from Lean Analytics’ lingo).
So what do we care about at our stage of growth?
For us, as we are still in the early phase of product-market fit, we are interested in how attractive content is for readers who opened our email. We don’t yet care about overall open rate, nor clickthrough rate. We focus on the value of our content and track those who take the time to read what we share.
Our KPI / North Star / OMTM is the Click to Open Rate. Here’s a taste of what it looks like…
But we’re still a long way from producing that KPI. We first need raw data. And not any kind of data and not an overabundance of data. Just the right amount of behavioural data that will allow us to produce the graph above and dig further to understand the underlying story behind these results.
This is a high-level view of the tracking plan we’re using for our newsletter (there is a lot to be said on building a tracking plan, implementing it and enforcing it, but we will keep that for a future post).
As you can see, we do not need to go all out with the number of events to track. We have core events that are essential to understand our subscriber’s journey with us, and some additional events that would allow us to dig further in the ad-hoc analysis.
How to measure?
Alright, we know what to measure, but how do we do that?
At Lantrns Analytics, we have our very opinionated approach to building our analytical stacks. We encourage you to refer back to Guide to Product Analytics, as well as our section on how our stacks work to get a sense of how we build our data infrastructures.
For quick reference, the high-level view of our stacks look like:
Here is the point-form description of that graph:
- Users interact with a product.
- These behaviours are captured by scanners.
- This data is sent to an ETL to be cleaned up, transformed, enhanced, etc.
- The resulting data is stored in a data warehouse.
- The structured data can then be consumed by BI software, digital analytics tools and more.
We’ve been very fortunate to be a part of Stitch’s beta integration for Mailchimp, which gave us automated access to very rich data about our subscribers and their actions.
Once the raw data is in the staging area of our data warehouse, we can run our ETL which cleans up, augment and transform our data to have it in a shape that can be easily consumable by Tableau, R, or other visualization tools.
There is a bunch of models in this ETL that is outside the scope of this newsletter’s analytics, but it gives you an understanding of how that same stack can be combined with other products to get an even more complete picture of your users and their journeys.
For now, our final data warehouse has 2 entity tables: dim_user and fct_event. This means that whenever visitors subscribe to our newsletter, subscribers open the newsletter, click on a link or visit our website to view a page or trigger some other kind of event, this data is being mined and transformed to ultimately populate those two final entity tables.
All that data taken individually would be chaotic if we would like to create sense. This is where dbt does its magic by facilitating the whole ETL process (and much much more) and get to a state where that data is consumable. If you’re not familiar with this tool, I think this post we published a while back will help you get started with dbt.
What to do with the results?
Now that you have the data in the right shape and ready to be consumed, the next step is to evaluate your results, explore and act on those insights.
As we’ve pointed in our guide, not all metrics are created equal. It all depends on your stage of growth and what you believe should be your focus at this point.
For Lantrns Analytics, our goal with the newsletter is to build brand awareness by creating an audience of product owners who are interested in what we love: using data to drive product development. Also, as we are still in an early phase of our newsletter, what’s important for us now is that our subscribers find value and engage with our content.
We’ve shown the results of our click to open rate above, but we mostly care for progress. We experiment with every newsletter we send out. Our goal at this point is not to look for marginal improvements, but really to find the right format that will drive engagement. The graph below tells us how each issue performed in relation to our baseline metric.
Owning our data means that we can dig into the underlying data behind that KPI’s results. Who are our most engaged subscribers? Who seems to be drifting away? What have they been clicking on? How many times did they open one newsletter? Have they shared it?
The type of analysis you do can be quite extensive. For example we’ve created a score to classify subscriber’s engagement and see which ones have the most impact on our click to open rate.
But no matter how we explore and interpret the data, one fact remains: we started by identifying a very personalized way to evaluate the performance of our newsletter, came up with a definition of our KPI, put the stack in place to track the relevant data and, at the end, find a way to measure our results. We can run experiments, see their impacts on that KPI and explore our data further to get to the full story.
Interested in implementing an analytical stack for your own newsletter? Drop me a line at email@example.com.