Calculating Z-score for A/B Test Results in Mixpanel

Is there a way to calculate the Z-score / statistical significance in an Insights report or somewhere else in Mixpanel? We are running an AB test, and the two samples are exposed to two different designs in our app. We are measuring their avg number of certain events tracked per week, and the number of events are close in both versions. I'd like to see if there's a statistical significance but I'm missing the standard error to calculate a Z-score

2 comments

kapa.ai
APP
·
·
Hi Marina V., Please stand by as I search the Mixpanel knowledge base for an AI-powered answer. I normally take about 15 seconds or so. In case AI cannot answer your question, feel free to reach out to Mixpanel support at mixpanel.com/get-support. If you have follow-up questions for this thread, please tag me again with your question: @kapa.ai <follow-up question> 👋
Feel free to ask a follow-up question once I've answered by mentioning me with @kapa.ai
kapa.ai
APP
·
·
While Mixpanel's Insights report doesn't directly calculate Z-scores, Mixpanel does offer statistical significance calculations in the Funnels report, which could be helpful for your A/B test analysis.In Funnels, Mixpanel calculates statistical significance using a hypergeometric distribution to determine if a segment's conversion rate differs significantly from the overall conversion rate. This is shown as a p-value in the overview table when you group by a property or cohort. Mixpanel considers results with a confidence level (1-p) greater than 0.95 as statistically significant, indicating the variation is likely not driven by random chance.Funnels: Measure conversions through a series of eventsFor a deeper understanding of how Mixpanel calculates statistical significance: > "To determine whether a particular segment's conversion rate is significantly different from the overall conversion rate, we use a hypergeometric distribution to calculate statistical significance. The hypergeometric distribution is used to model the probability of picking k items of a particular type in n attempts without replacement from a population of size N having K items of the same type."How does Mixpanel calculate statistical significance?For your specific A/B test scenario, you might want to: 1. Set up a funnel that includes the events you're tracking 2. Use the A/B test variation as a breakdown property 3. Examine the statistical significance values provided in the resultsAlternatively, Mixpanel has integrations with dedicated A/B testing tools like AB Tasty that might provide more comprehensive statistical analysis: Mixpanel + AB Tasty: A match made in data privacy heavenYou can also learn more about A/B testing with Mixpanel in this introductory video: Introduction to A/B testing with MixpanelIf these options don't meet your specific needs for calculating Z-scores with standard error, you might need to export the data and perform the calculations externally.
kapa.ai is still learning and improving, please let me know how I did by reacting below.
ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.