Solved

How do I get retention for unique users only?

A
Badge +2

Hey guys,

I’m looking to get retention in 2 month intervals. For example, how many users that completed event x in March came back and completed event x again within 2 months (by the end of May).

Since there is no 2 month increment on the retention table, I don’t want to count a user that completed event x twice (one 20 days after and again 40 days after) to count twice. If this would be counted increasing my absolute or relative return rate, it would skew my retention to appear higher.

Please let me know how to measure this (I’m assuming the standalone retention doesn’t accomplish what I’m looking for). Thanks!

icon

Best answer by mattsmith 1 July 2020, 10:43

Hi @arshia93 

Rather than being common practice, it is just simply how the Retention report works, as it cannot be broken down in 2 monthly buckets.\

 

For Cohorts it would be slightly different as you would be setting the criteria as “Did Event x at least 1 time between March 1st and March 31st, AND, Did Event x at least 1 time between May 1st and May 31st” and you can then see how many people are in this Cohort. To do this, once it is created you can select the 3 dots on the right hand side and hit “Visualize Trend” to see it in Insights as a line chart, or View Users to see the volume of users within Explore.

 

If you create multiple Cohorts for January > March, February > April, March > May etc you can add them all to the same Insights report and see how many users are in each, and how they compare. Or you can simply record the volume of these cohorts yourself to see how many users are in each one.

 

Thanks a lot,

Matt

View original

13 replies

mattsmith
Rank
Userlevel 2
Badge +2

Hi @arshia93 

 

This is a good question, and can definitely be achieved in Retention.

 

You are already doing the right thing (Event x, came back and did Event x), but please note that you will need to choose “N Day Retention” under where it says “Advanced Query Settings”. This is important because in this format Retention is taking your adhoc Cohort (Size column) of users who perform your Event x on your specific time range (within a month, or day) and then sees whether they came back and did it again.

After the first period (<1mth) here you can see that 10,642 of the original 10,877 users came back and did my event again, and then 8,209 came back after 1 month (between 1 and 2 months), etc. This *does not* mean that these 8,209 also came back within 1 month and were part of my 10,642 in that column. The same goes for every subsequent column. The numbers here are not subsets of the column before it, they are only subsets of the original “Size” cohort you have created.

 

Then all you need to do is stretch out the date intervals enough so you get the date range you need. You can then toggle whether to look at Days, Weeks, or Months on the right hand side (in my example above I have 6 month date range split by Months). For your requirement you can simply look at column 2, column 4 and see how users were retained up to 2 months, and 4 months, and 6 months etc.

 

I hope this all makes sense, let me know if you have any other questions.

Matt

A
Badge +2

Hi @mattsmith ,

Understood. Here’s the part I don’t understand yet, let’s look at this:

 

 

If I only look at column 3 (labeled 2 for those retained by the end of the second month), it shows 2 were retained. The thing is that 4 were retained during the initial month, 5 on the first month, and 2 on the second month.

Because my product varies based on usage (some need it less than 1 month some less than 2 months - 2 months is the threshold though), it feels like I can’t assume only 2 of 21 were retained. Actually it’s rare column 1 is larger than column 2 (like most examples). This is why my hunch was to combine them, but only if it was unique users.

What do you think?

mattsmith
Rank
Userlevel 2
Badge +2

Hi @arshia93 

I’ll admit this is where the column headings can be slightly misleading.

The third column, although marked 2, actually means “Users who came back and did my Event after 2 months, but before 3 months”. In this instance you’re looking at users who did your event after May 25th, but before June 25th.

 

Apart from that though you are reading the report correctly. You had 21 users in your initial sample period (March 25th for 1 month I’m guessing) who did Event x in that period. 4 of them came back and did it again in that same period March - April, 5 then between April and May, and then 2 between May and June. This is also definitely unique users, as this is how the Retention report works.

 

One way to test what has happened here is to click on each number within the report and hit “Create Cohort”. This will create a Cohort of users specifically who did your Event x in your initial period, and then did it in your chosen period after that. You can then load this cohort as a filter in Insights, select the same event you are using as Event x here, and then change the values showing from Total to be Unique. This will show you how many Unique users did your event, within the time period of your Retention report. You can do this in Insights in general too. Simply select your Event x, set the date range to match your retention report month, and then change the value to be Uniques.

 

Thanks a lot,

Matt

A
Badge +2

Hi @mattsmith,

Thanks for clarifying the columns and the fact that they are unique users.

 

Apart from that though you are reading the report correctly. You had 21 users in your initial sample period (March 25th for 1 month I’m guessing) who did Event x in that period. 4 of them came back and did it again in that same period March - April, 5 then between April and May, and then 2 between May and June. This is also definitely unique users, as this is how the Retention report works.

 

I’m looking at the full month of March and doing so to say I have x number of users retained or x% retention within frequency of every 2 months.

If Matt completed the event again 5 weeks after the first event, wouldn’t he be placed in column 2 and skip column 1? If so, when you’re saying 7 of 8 or 7 out of 29 were retained it doesn’t seem like that is always the case. Maybe I would complete the event every 3 weeks and stop doing so between month 1-2, but you may only show up on column 2, 4, 6 etc or someone else would only show in column 1, 3, 5 etc.

 

 

mattsmith
Rank
Userlevel 2
Badge +2

If Matt completed the event again 5 weeks after the first event, wouldn’t he be placed in column 2 and skip column 1?

Yes correct.

For example, if I do event x on March 1st I count as one of your 29. Then 5 weeks later I do the event again I am counted in column 2 (1 month) because I didn’t do it within 1 month, but did do it between 1 and 2 months.

In your example here you have 7 users who did your first event in your first month, then came back between 1 and 2 months later and did it again. If I completed the event every 3 weeks I would be in each column, as I kept coming back at least once every month, so will keep counting in your columns.

A
Badge +2

Got it. To sum it all up, can you tell me if I’m reading this correctly? The goal is to measure retention in 60 day intervals:

 

March: 42.86%

April (if it was to end with the existing data): 48.27%

Average: 39.55%

 

Assumptions I'm making based on my understanding:

  • To ensure I'm counting the retention rate regardless of when a user re-books within the 60 day window I'm combining the two columns because there is unique users in each column. If user a,b,c,d book in March, user b rebooks in March, users a and c rebook in April, therefore I retained 75% (3/4)
  • April could contain users who first booked in March but this is still a correct reading of the report

 

mattsmith
Rank
Userlevel 2
Badge +2

Yes your assumptions are correct, and how I would read the report.

Remember when you’re looking to combine the column % values, you might have users who count in both (or all) columns as they keep coming back and doing the same event over and over. Apart from that you should be good to go.

A
Badge +2

Thank you.

 

 

Remember when you’re looking to combine the column % values, you might have users who count in both (or all) columns as they keep coming back and doing the same event over and over. Apart from that you should be good to go.

 

Exactly, this is what I was referring to in the title of the question when I said “unique”. Just to be clear, is there a better way to prevent that from happening? (perhaps it’s common practice to keep it as is?)

mattsmith
Rank
Userlevel 2
Badge +2

Hi @arshia93 

 

You were right in your message to me, I did not see your response.

 

I apologise for the confusion throughout this exchange, as you are correct that if a customer does your Event both within 1 month and between month 1 and 2 they would count in both columns, meaning if you combined the % volumes a user may count twice.

 

If this is a compromise you can live with then it wont be a problem. It would be common practice to either live with this compromise, or to simply run monthly retention figures (which is why the report is default to this). For 2 month retention you could create Cohorts of users who joined, or did Event X, within a defined month, and you can then add these cohorts as filters within the Retention report. I will need to check whether there is a better way to do this, as I can’t think of one right now, outside of exporting the data and running the analysis manually.

 

Thanks a lot,

Matt

A
Badge +2

Hi @mattsmith,

Ah yes, cohorts could help partially. Though the counting in both columns issue wouldn’t go away.

If it’s common practice to not break it down any further then that could be fine. It’s very important that I’m presenting these numbers to stakeholders with confidence, so I’d love to hear what insights you have here.

Thanks

mattsmith
Rank
Userlevel 2
Badge +2

Hi @arshia93 

Rather than being common practice, it is just simply how the Retention report works, as it cannot be broken down in 2 monthly buckets.\

 

For Cohorts it would be slightly different as you would be setting the criteria as “Did Event x at least 1 time between March 1st and March 31st, AND, Did Event x at least 1 time between May 1st and May 31st” and you can then see how many people are in this Cohort. To do this, once it is created you can select the 3 dots on the right hand side and hit “Visualize Trend” to see it in Insights as a line chart, or View Users to see the volume of users within Explore.

 

If you create multiple Cohorts for January > March, February > April, March > May etc you can add them all to the same Insights report and see how many users are in each, and how they compare. Or you can simply record the volume of these cohorts yourself to see how many users are in each one.

 

Thanks a lot,

Matt

A
Badge +2

Thanks @mattsmith ,

I believe you mean:

“Did Event x at least 1 time between March 1st and March 31st, AND, Did Event x at least 1 time between April 1st and May 31st”

Then create →

“Did Event x at least 1 time between April 1st and April 31st, AND, Did Event x at least 1 time between May 1st and June 31st”

Since most users complete event x between 2-8 weeks after doing it the first time.

 

I’ll use the above method for internal analysis. However, if that’s the right way to say I retained x% of March users, then this should be good just wanted to make sure.  

mattsmith
Rank
Userlevel 2
Badge +2

Hi @arshia93 

 

Yes you’re correct, this is definitely the way I would do it.

And yes I would classify it as “x% of March users retained for 2 months”.

 

Thanks,

Matt

Reply


Mixpanel