It’s the first time I’m posting here so I hope that’s the right place :)
I’d like to submit my two cents on the current ab testing evaluation feature, which I think it has a huge potential!
What I think are necessary features:
1. An integrate planning tool that let us plan the experiment and obtain a target sample size, maybe with some dynamic adaptive strategy, to avoid peeking. (It’s fundamental to help inexpert users in running ab test consistently)
2. The possibility to select a primary KPI and get an alert when the experiment is finished
3. The possibility of saving an history of executed experiments with labels (Running, Failed, Succeeded, Ready to Start)
4. A space to describe the experiments (Title and description might be fine)
What I think that would be nice to have:
1. Bayesian testing for non binomial variables
2. Overall statistics on experiments done in the past
I was starting to program a custom report to assolve some of those tasks.
Let me know if I can help somehow.
CPO @ Loopsie