I have a pre-experiment bias alert but it doesn’t say what’s causing it
I have a pre-experiment bias alert but it doesn’t say what’s causing it
I have a pre-experiment bias alert but it doesn’t say what’s causing it
The alert you're seeing is likely from Mixpanel's Retrospective-AA (Retro-AA) health check, which flags pre-existing differences between your experiment groups before the experiment started. It doesn't pinpoint a single cause, but here are the most common reasons it fails:
Bugs in randomization or bucketing logic — users aren't being assigned to variants truly randomly
Exposure events not firing consistently across variants
Users being reassigned to different variants mid-experiment
What to do next
Investigate your assignment logic to check for any bias in how users are bucketed into control vs. treatment groups.
Enable CUPED — even if Retro-AA fails, CUPED can help reduce the impact of pre-experiment bias by adjusting for prior user behavior, potentially allowing you to still analyze the results. [CUPED docs]
If the bias is significant, consider restarting the experiment after fixing the root cause.
Note: Retro-AA failing doesn't automatically invalidate your experiment, but it does mean any observed lift could be partially explained by pre-existing group differences rather than your treatment. [Health Checks]
ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
"Bugs in randomization or bucketing logic" refers to implementation errors that cause users to be unevenly or incorrectly distributed across your experiment variants. Here are specific examples drawn from Mixpanel's documentation:
1. Static string passed instead of a dynamic user ID
A common coding mistake is accidentally passing a hardcoded string to your identity/assignment function instead of the actual user's ID. For example:
function authenticate_user(user_id) {
mixpanel.identify('user_id'); // ❌ static string, not the variable
mixpanel.track('user_authenticated');
}
This causes all users to share the same ID, completely breaking how they're bucketed into variants. [Hot Shard Limits]
2. Sample Ratio Mismatch (SRM)
If you configured a 50/50 split but ended up with, say, 60% in control and 40% in treatment, that's a sign something went wrong in the assignment logic. Mixpanel uses a Chi-squared test to detect this. Common causes include:
Differences in the start times of each variant
Broken redirects or technical issues affecting one variant
Untested or complex user assignment logic [Health Checks]
3. Exposure events not firing consistently
If your $experiment_started exposure event doesn't fire reliably for all users in a variant, some users' pre-experiment behavior gets attributed incorrectly, skewing the group composition before the experiment even begins. [Health Checks]
4. Users reassigned mid-experiment
If a user is moved from one variant to another during the experiment, their behavior gets split across groups, introducing bias into both. [Health Checks]
In short, any of these issues can result in your control and treatment groups not being truly comparable before the experiment starts — which is exactly what Retro-AA is designed to catch.
ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.