Retention rate doesn't consider first action's lifetime when breakdown by property?

  • 4 March 2020
  • 1 reply

I noticed a weird thing earlier when looking at first-time retention report. 

  1. When breaking down by time-based cohort (based on date of first action, this is the default view), the retention curves flatten before dropping directly to 0 in month-N (side note: it should discontinue instead). This is normal because the first event simply haven’t passed for N months yet – hence the data don’t exist.
  2. However, when breaking by a property, the retention curves will gradually decrease to 0, and all segments will reach 0 around the same time.

After some digging, I suspected this is the formula system is using:
(# of users with X property who did action B in month-N) / (# of user with X property who did action A at any time in reporting period)

This present a big problem because some users didn’t perform action-A that long ago, but they’re part of the denominator. In my opinion, a user should only be included in the denominator for month-N retention when he performed action at least N months ago.

Is my interpretation correct?

1 reply

Hi Ayang,

Thanks for this post - I’m a support engineer here at Mixpanel. Just wanted to say that I see your point here and that your interpretation is correct; for retention reports with breakdowns in them (meaning the rows correspond with the property values, rather than date buckets), you do see new users count in the denominator when it comes to those later retention months they’ve not technically ‘qualified’ for yet. 

We’ve surfaced this as a product/usability gap and will be discussing it in more detail with the retention team sometime soon. In the meantime, you might consider using our new “First Event Time” user property to filter out newer (or older) users from the retention report overall; that might help you get a little more control over your denominators when it comes to analyzing those segments long-term retention.

Hope that helps, and again, thanks for bringing this up.