Hi there,
I’ve been looking into the detail of the retention reports when using the Breakdown feature and I’ve come to realise that the numbers that these reports churn out are dangerous and may not be what one would expect to see. I understand what the numbers are showing but I feel that these need to either come with a significant health warning or a more sophisticated calculation.
The key issue is that, say if looking over the recent 52 week period (as such most of the weekly cohorts wont have data for the latter weeks), the wk 52 retention for the week 1 cohort maybe 50% but when aggregated in the way introducing a breakdown does, this would appear as a number much smaller than this (maybe 1/52 if the weekly cohorts are consistent). Similarly, if comparing cohorts over a particular time period, the comparison is influenced by the phasing of those cohorts e.g. if you recently significantly invested in expanding across android devices, then the large influx of new users would make the latter week retention figures, when aggregated, look bad.
The solution is adding an option to make a relative retention calculation that would only include cohorts in the denominator if they also appear in the numerator.
No idea if that makes sense but wanted to share my thoughts.
