Mixpanel Community Icon

Addressing Mixpanel's Handling of Nullable Booleans in A/B Testing

·
·

Nullable booleans We are relying heavily on mixpanel for evaluating A/B tests, but have found it to be hard to trust mixpanel on a number of occasions. Imagine the below scenario: we set-up an A/B test -> “has some button” true/false we would then like to evaluate the effect of “has some button” on our metrics, e.g. messages per user The feature flag is only set once the user opens the screen where that very button exists… but mixpanel treats all null or not set values as false … this is wildly misleading. When evaluating the messages per user, the denominator “per user” by definition becomes deflated on whatever group we call true as this group ONLY contains users who have been on the screen, while the denominator becomes extremely inflated for whatever group we call false as these contain not only users to whom we have assigned the false flag, but also all other users who have not been given a flag at all. Therefore, all true tests will outperform false on all per user metrics. If we revert the test, we ofc. just get the opposite situation “does not have some button” true/false. Now “does not have some button” = true will outperform. We are overcoming this by creating obscure metrics to evaluate our A/B tests e.g. messages per messager where we are sure that both the numerator and denominator have been given true/false flags. But this is an annoyingly cumbersome process - one which I assume the whole purpose of mixpanel is to overcome. If not, we might as well query data directly from our DB anyways. Could you please consider having some option at a global level, whether to treat “not set” values as false or as null - we would at least opt for “secure” analytics in this regard, if we have not explicitly set a bool value, we expect mixpanel to not treat it as one. ⚠️ This should also stand as a warning to other folks who use A/B testing + mixpanel, you might be always going for the true outcome on a boolean test ⚠️ //thanks ❤️ mixpanel-new

  • Avatar of Ingmar A.
    Ingmar A.
    ·
    ·

    This behaviour is also weirdly inconsistent with other data-types: int , float and string are all null or not set in cases where the value has not been set

  • Avatar of Christopher C.
    Christopher C.
    ·
    ·

    You can change the type of your boolean property to a string as a bit of a workaround (which will then give you not set as well as true and false), but agreed that this is an issue, I've had too many co-workers get confused by not realising that false and null are grouped together.

  • Avatar of Vlad S.
    Vlad S.
    ·
    ·

    types in mixpanel are pretty loose (e.g. you can send something as a string and then as an int and really confuse the interface about it) and I'd recommend against relying on anything being a certain type that said, we run out A/B tests by firing an Experiment Started event with the experiment name and variant as properties on that event. That way, anyone who is not placed in the experiment doesn't have the event, and you can set up your boards like funnel reports or retention reports where the first event is the experiment placement event and the rest is whatever you want to measure, and you can break it down by variant.

  • Avatar of Vlad S.
    Vlad S.
    ·
    ·

    Incidentally, that's also how Mixpanel's experiments product works, although it makes it a lot easier to check a million metrics at once