Question

Mixpanel Retention "Incomplete Buckets" Coloring and Labeling is Incorrect

  • 8 September 2020
  • 2 replies
  • 37 views

Userlevel 1
Badge
  • Frequent Contributor
  • 5 replies

Apologies if this has been raised before, I could not find an example.

 

Problem:

  • Mixpanel labels “incomplete buckets” at the “edge” of the retention report (the right-most square in any given row)
  • Incomplete bucket labeling, combined with labeling like “Users will continue to qualify until X” create the false perception that some conversion windows are closed, when in fact they are not closed for many users, and returning to the same retention report days later will show larger conversion figures in those ranges as users continue to convert.

Proposal:

Apply the white and grey dashed-line styling to all blocks in the retention report which are still “open”, and adjust the labels to reflect conversion windows which are true for all of the users who were bucketed in the given month - not just those who were bucketed on the first day of the month.

 

Example of the problem here:

If a user is bucketed on July 31:

  • their <1 month or 30-day range ends August 30th
  • Their “1 month later” or 30-59 day range ends September 28th. If today’s date is September 7th, this box should be white with grey dashes, and the label should read “Users continue to qualify until September 28th”
  • Their “2 months later” or 60-89 day range ends October 28th, the box should be white with grey dashes, and the label should read “users will continue to qualify until October 28th” 

This coloring system appears to be in place to support users who want to easily answer these questions: 

  • “what is COMPLETE and INCOMPLETE data?” 
  • “when I can report on the performance of X-month in retention terms with final numbers?”
  • “when have I missed my chance to ‘win back’ some dropped users?”

But the coloring/labeling in place seems borderline dangerous for anyone trying to answer these questions without carefully reading beyond what is written in the help docs (and beyond the mis-positioned labels). 

Hope this helps!

 

 

 


2 replies

Userlevel 1
Badge

If you wanted more proof, here’s the same chart some hours later 

 

21.29 → 21.55% in a time window that Mixpanel’s styling and labeling suggests is closed. Figure in other blue windows near the edge of the report have also grown.

 

 

Badge

Hi Adam,

 

Thanks for raising this issue with us! I have forwarded it to our engineering team who will fix this. It seems that the labelling of the retention buckets is a bit off at the moment. While I am not sure how this happened, our engineering team is aware and will revert back to how it used to be.

 

Have a great day :)

Reply