Nullable booleans
We are relying heavily on mixpanel for evaluating A/B tests, but have found it to be hard to trust mixpanel on a number of occasions.

Imagine the below scenario:
we set-up an A/B test -> “has some button” true/false

we would then like to evaluate the effect of “has some button” on our metrics, e.g. messages per user
The feature flag is only set once the user opens the screen where that very button exists… but mixpanel treats all null or not set values as false … this is wildly misleading.

When evaluating the messages per user, the denominator “per user” by definition becomes deflated on whatever group we call true as this group ONLY contains users who have been on the screen, while the denominator becomes extremely inflated for whatever group we call false as these contain not only users to whom we have assigned the false flag, but also all other users who have not been given a flag at all.
Therefore, all true tests will outperform false on all per user metrics.

If we revert the test, we ofc. just get the opposite situation “does not have some button” true/false. Now “does not have some button” = true will outperform.

We are overcoming this by creating obscure metrics to evaluate our A/B tests e.g. messages per messager where we are sure that both the numerator and denominator have been given true/false flags. But this is an annoyingly cumbersome process - one which I assume the whole purpose of mixpanel is to overcome. If not, we might as well query data directly from our DB anyways.

Could you please consider having some option at a global level, whether to treat “not set” values as false or as null - we would at least opt for “secure” analytics in this regard, if we have not explicitly set a bool value, we expect mixpanel to not treat it as one.

⚠️ This should also stand as a warning to other folks who use A/B testing + mixpanel, you might be always going for the true outcome on a boolean test ⚠️

//thanks ❤️ :mixpanel-new:

Product Ideas

Product Ideas