IN REVIEW

Retention Breakdown - Relative Retention Rates

  • 4 March 2020
  • 7 replies
  • 130 views

Userlevel 2
Badge
  • New Participant
  • 3 replies

Hi there,

I’ve been looking into the detail of the retention reports when using the Breakdown feature and I’ve come to realise that the numbers that these reports churn out are dangerous and may not be what one would expect to see. I understand what the numbers are showing but I feel that these need to either come with a significant health warning or a more sophisticated calculation.

The key issue is that, say if looking over the recent 52 week period (as such most of the weekly cohorts wont have data for the latter weeks), the wk 52 retention for the week 1 cohort maybe 50% but when aggregated in the way introducing a breakdown does, this would appear as a number much smaller than this (maybe 1/52 if the weekly cohorts are consistent). Similarly, if comparing cohorts over a particular time period, the comparison is influenced by the phasing of those cohorts e.g. if you recently significantly invested in expanding across android devices, then the large influx of new users would make the latter week retention figures, when aggregated, look bad.

The solution is adding an option to make a relative retention calculation that would only include cohorts in the denominator if they also appear in the numerator.

No idea if that makes sense but wanted to share my thoughts.


7 replies

Second this!

To be honest, the current way Mixpanel calculating retention when breaking down by segment is plainly wrong - it should not even exist as an option.

Userlevel 2
Badge

Thanks for the support @Ayang.

If the period selected had data for the full date range for all cohorts, then it would be fine. But most of the time we are looking at recent data to identify changes in trends or behaviours which, in my experience, renders the use of this nearly obsolete.

In order for me to get this to work properly, I’d need to download the data for each country (for example) so I can have the weekly breakdown, and reaggregate it myself.

Nonetheless, an incredibly powerful tool if used correctly and even more powerful if MP could get it completely right. Would be great to view multiple breakdowns too, if it worked in the first place.

 

Badge

Third this!

Userlevel 5
Badge +4

@marc @blazzz 

Thank you for submitting this feedback on our Retention report! We filed a product gap on your behalf with our Engineering and Design team for review.

 

If you are not automatically subscribed, please make sure you do by clicking on the subscribe star button  - at the bottom of the original idea post - to get emailed updates on its progress.

 

@Ayang I noticed that you signed up for Community with a different email address than your Mixpanel Account. 

In order to submit the product gap, would you mind sending me a private message with your Mixpanel email? Ill be here!

Userlevel 4
Badge +2

hey @marc

I’m the PM at Mixpanel for our Retention report. I’m going to reach out for additional clarification and to discuss your thoughts on this, so please keep an eye out for an email from me  :slight_smile:

 

I was misled by the report in the same way that was described above… I do believe the way to use breakdowns in retention report has to be re-think as this can in the best case lead to confusion and in the worst case to bad decisions making. 

Userlevel 4
Badge +2

Hi @pbriod, @blazzz , @marc , and @Ayang ,

I’m not quite sure I understand the issue as described- would one of you mind elaborating so I have more context on the problem you have with how segment breakdowns are calculated in Retention?

Looking forward to hearing back!

Reply