MPKnowledgeDrop

How should I analyze user engagement?

  • 8 May 2020
  • 9 replies
  • 846 views
How should I analyze user engagement?
Userlevel 5
Badge +2
  • Mixpanel Product Manager
  • 87 replies

Understanding your product’s engagement is important because of the broad impact that user engagement in your product has on your business. The more engaged your users are, the more likely they are to find value in your product, retain for longer periods of time, encounter monetization opportunities, and refer additional users. 

 

Since engagement is so vital to success, understanding it is critical. But how should we look at engagement?

 

When you measure engagement, what you want to know is how active your user base is over a given time period. Contrary to conversion analysis, where we want to understand if a user has a) converted or b) dropped off, and retention, where we want to see if a user has a) retained or b) gone dormant, engagement is not a binary metric. Engagement of a user base lies on a spectrum. That means, to effectively measure engagement, we need to break up the user base into different buckets along this usage spectrum and examine the different groupings along this stretch. This is because your goal is to move users from a less engaged state to a more engaged state, and by breaking the engagement spectrum off into buckets, you are creating a tangible way of measuring progress to and from each sub-state.

 

But that brings us to another important question: how should you create these buckets?

 

One common way of doing this is plotting a distribution of users with activity in your app, and then craft three engagement groupings (power, core, and casual) based on a fixed percentage. 

 

  • For example, if I worked on a music streaming service called MusicFinder, I would plot users doing any activity. The bottom 10% of users are classified as casual, the middle tier as my core users, and the top 10% as my power users. 

 

l5RkVlKYB8GUGnI1qK8CfCzT98EIdB_XZQ_-VOxPMH6UFQTryS3fDdOEGalMo7QzjmM0Wo3GtXwdO_AplNA9IwDUzm9tkP58hg1FLrHnhcjH3QjfLASx69PQsEATB39d9rCkCZv1

 

While this seems logical, it falls short of the mark for a few reasons:

  1. Any activity does not necessarily exemplify good activity for your business. It’s important to choose an event that aligns with value creation. Otherwise, you’re adding a lot of noise. 

    • In the case of the streaming service, I want to make sure that I’m looking at events like starting to play a song or creating a playlist because they’re correlated with my value proposition—listing to music. If a user foregrounds the app but does nothing of substance, that should not be considered an engaged user. 

  2. The current state of the engagement distribution of the user base does not necessarily reflect the ideal distribution.

    • If the majority of MusicFinder’s user base plays a song once a quarter, that doesn’t mean the core base should be on a quarterly cadence. That probably means we have a big problem to solve.

 

Rather, choosing the right core engagement bucket relies on critical thinking about your product. Ask yourself two questions: 

  1. What problem does my product solve?

    • This dictates the event behind the value for your core user persona. This is the event (or events if doing analysis for multiple personas) whereby we should be measuring engagement.

  2. How often does my target audience encounter this problem?

    • This dictates how often the core user group should be coming back. If users solve this problem at a monthly rate, the core engagement group should be monthly. 

 

Now that we have the core user bucket, we can create hypotheses around what the power and casual user buckets could be—with the casual bucket being a step down and power being a step up. 

 

  • For example, if MusicFinder is meant to entertain users at a weekly cadence, the power group would use biweekly/daily, and casual users would be on more of a monthly basis. 

 

The last step is to confirm our analysis with data and make sure our qualitative hypotheses line up. To do this, hop into your Signal report and run a correlation analysis between your engagement event and retention.

 

Here, I’ve run a query to see how my Starting Play event correlates with 2nd Week Retention

 

S7b2AHulaGx1ey7ovPj_RhSJfnUIP22HW5j7dNzOCdzhSeNX-nglQXE-w2MzNE8BMt3E8dLycrVw0M9zGn5H5SQqdyQl7796bSlDoo2g1QKzEq-N9pcRFQaDdigLypAWpB_jhsJD

 

We can see that starting to play a song optimally occurs 4 times within 7 days, and done by the most engaged minority of our users. This fits nicely with our hypothesized power user bucket at a biweekly/daily cadence. 


Now that you have your user engagement buckets that are aligned with your product and business objectives, you can start digging into ways you can influence movement between these engagement states. Stay tuned for next week’s article to learn how


9 replies

@michael I assume the X-axis on the first image is how often users did X and on the Y-axis how many users are in that group right? How did you create this report? 

Userlevel 5
Badge +2

Hi @mbruschi ! This is a feature that is currently in closed beta - a linear distribution graph of the Frequency visualization in the Funnels Report.

It should be available to all Mixpanel users very soon!

@michael looks nice, is there a way to beta test this?

I should have read the comment before searching in the application for one hour 🙂 Is it possible to get access to the beta or is there another way of easily replicating this in MP?

Userlevel 5
Badge +2

Hi @Irfan and @mbruschi , 

Apologies for the delay here, but the linear distribution graph of the Frequency visualization in the Funnels Report is GA.

Now all users can find a line and bar option it in their Funnels report, under Frequency.

 

8.1.1

Thanks @michael, this is great news. I am trying to now replicate what you described in the article and jumped over another obstacle: “

  • “For example, if I worked on a music streaming service called MusicFinder, I would plot users doing any activity. The bottom 10% of users are classified as casual, the middle tier as my core users, and the top 10% as my power users. “

In my current view, I need to select in the funnels report a step 1 and step 2 before applying the frequency view. In comparison to the “insights” report, I do not have the option to select “Any event”.  The Frequency would only display how often step 1 was done before doing step 2. If I get it right, it excludes the cases where step 2 was done without step 1. Would be great to guide me in the right direction here.

Userlevel 5
Badge +2

Hi @irfan

Personally, I recommend that you do not do it this way. While it’s conventionally done, I cover a few reasons why you should opt for a different method:

While this seems logical, it falls short of the mark for a few reasons:

  1. Any activity does not necessarily exemplify good activity for your business. It’s important to choose an event that aligns with value creation. Otherwise, you’re adding a lot of noise. 

    • In the case of the streaming service, I want to make sure that I’m looking at events like starting to play a song or creating a playlist because they’re correlated with my value proposition—listing to music. If a user foregrounds the app but does nothing of substance, that should not be considered an engaged user. 

  2. The current state of the engagement distribution of the user base does not necessarily reflect the ideal distribution.

    • If the majority of MusicFinder’s user base plays a song once a quarter, that doesn’t mean the core base should be on a quarterly cadence. That probably means we have a big problem to solve.

 

However, if you are looking to do a distribution without a Funnel (users needs to do Step 1 and then Step 2) and rather on a particular event, you could use Insights. 

For example, if you wanted to know the count of how many distinct days in the last 30 that your user did a particular action then you could create a frequency as shown in this linked gif.

Here is an example report in one of our sample data projects: https://mixpanel.com/s/AKQDQ

 

But I hope you opt for using a key event that conveys user value rather than “any event”. 

Hi @michael. Nice post. Can you please share the link to the funnel report? I mean the one you are using to create your user buckets. If yes, please do that with the “Music Finder” sample dataset. Thanks.

Userlevel 5
Badge +2

Hi @Shishi 

I’d advise against creating these user buckets- 

In theory your bucketed users would be tailored to your specific product. When your user engagement buckets are aligned with your product and business objectives, you can start digging into ways you can influence movement between these engagement states.


Start by asking yourself two questions: 

1.  What problem does my product solve?

  • This dictates the event behind the value for your core user persona. This is the event (or events if doing analysis for multiple personas) whereby we should be measuring engagement.

2 How often does my target audience encounter this problem?

  • This dictates how often the core user group should be coming back. If users solve this problem at a monthly rate, the core engagement group should be monthly. 

From there, you can use a Signal report to see the optimal action that correlates with 2 week retention.

 

If you wanted an example report in one of our sample data projects that depicts a frequency distribution I can help with that.  We’ve made some strides in Insights since this post was published to allow for distinct property and property value counts. I like to use this to see how many days users have performed the core action in a given time period https://mixpanel.com/s/AKQDQ

 

But if you wanted to know how many times a user did a core action before moving on to convert to a different action- you could use a funnel like this: https://mixpanel.com/s/7HY80

 

Reply