Question

Is there a difference between filtering and breaking down retention report?

  • 2 February 2021
  • 7 replies
  • 134 views

Badge +2

Hey guys, I’m getting two different retention rates when filtering or breaking down an event. They are both the same people property. Any clue why this may be the case? Thanks


7 replies

I think the retention reports using filters are correct but the reports using breakdowns are incorrect. The report using breakdown calculates the retention for a given month = (number of users that performed event B in given number of months after event A) / (total number of users who performed event A). The numerator is correct here but the denominator is wrong as it includes users who performed event A but the given number of months haven’t passed yet to see if they perform event B. This means the retention curve is a severe underestimate of the true retention curve. This is a pretty bad mixpanel bug and from other posts on this forum I see mixpanel have been told about it quite some time ago but they still haven’t fixed it. The breakdown feature would be really useful to me as well 😞. Mixpanel please fix it!

Badge +2

Thank you @matty , are you certain of this?

It’d be nice to have confirmation on this from Mixpanel as well (if they read these)

Userlevel 2
Badge +2

Hey @arshia93 ,

Great question -- when doing a filter, our retention report behaves normally in that it buckets the users based on the time unit you select.  However, when breaking down by a property, the buckets of the report will be broken down by that cohort or property where each bucket shows the average number of users in each cohort who performed the event, or the number of users with that property who performed the event within the date period.  As such, the calculations will slightly differ.

With regards to the calculation that @matty brought up, I’m not sure I follow the part about the number of months not having passed yet.  We have the n-day and unbounded retention options to calculate the rate.  You can read more about it here.

Best,

Ian

Hi @ian @arshia93 ,

Thanks for responding. I think there is a problem with the retention breakdown feature. The retention for month N of event A followed by event B is calculated = (# users who did event B N months after event A) / (# users who did event A). This is incorrect it should be = (# users who did event B N months after event A) / (# users who did event A at least N months ago). You should calculate the retention at month N only for people where at least N months have passed after doing event A.

Let me give an example with a music streaming service where we look at the retention of users registering then coming back to stream a song. Say there are 1000 users total and only 10 users registered at least 18 months ago. Say 5 of those 10 users have stayed active so the 18 month retention should be 5/10=50% but using the current breakdown method it would be 5/1000=0.5%. This is a clear error as it is dividing by the total number of registered users instead of only the number of users who registered at least 18 months ago.

 

This error can be clearly seen if you e.g. filter for a certain country, look at the retention curve, then do a breakdown by country and see how the retention curve changes even though it should not change since it’s the exact same data.

 

Hopefully this clarifies why the retention breakdown feature is incorrect and needs to be fixed. Please let me know if anything isn’t clear.

Thanks, Matt

Badge +2

Hi @ian , I’m not sure I understand the difference. If the filter is the right one I’ll do it, but to be clear, they’re the same exact property (not breaking down by cohort)

Userlevel 2
Badge +2

Hello all,

Sorry for the delay here.  We’ve just revamped our segmented retention report such that we intervalize each segment rather than throwing all into one entire date range.  You can read more about it here:

https://help.mixpanel.com/hc/en-us/articles/360060422811

Best,

Ian

 

That’s great, thanks a lot @ian! Really helpful

Reply


Mixpanel