We are seeing issues with A/B experiments targeting our apps, and are curious if other customers have the same issue that we see. Here's the overview.
- Target an experiment. In our case we have a user property that defines a user as a "subscriber".
- So we target our experiment to that property- show this experiment only to "subscribers".
- Fine- a list of subscribers appears in the A/B experiment.
- Start the experiment.
- Subscribers start seeing it.
Now the issue:
- A person who is a free user subscribes- they are now a subscriber. Yet the experiment does not appear for them, even though they now meet the criteria for inclusion in the experiment.
- A person who is a subscriber cancels their subscription- yet the experiment continues to show for them, even though they no longer meet the criteria for inclusion in the experiment.
Intended effect: as a person enters or exits the experiment criteria, it consistently adapts to either show or end the experiment for that user.
Please advise if you've seen this issue. And if anyone from Mixpanel is reading this, please advise on the intended effect. Do experiment cohorts a) get defined as a one-time batch process when setting up the experiment (thus explaining this issue- it is then not a bug, but a product reality) or b) does the experiment cohort update in real-time. If "b" please provide the time frame- is it truly real-time, or in batches x-times per min/hour/day/week.
This will save us a lot of time debugging.