Question

Possible bug when setting up AB testing

  • 9 April 2019
  • 1 reply
  • 88 views

Badge

I tend to find when I set-up developer tweaks for a new AB test that when I go back later, some of the tweaks are gone from Mixpanel. And thus the AB test is set-up incorrectly. I then have to reset them up again. As I know this happens now, I always check for it, so I catch it. Is this a known issue?


In case it's not very clear, let me give an example.

I set up a couple of MPTweakValue in my iOS code.

I fire up the simulator, get the mixpanel website to see the simulator.

Set an AB test up that uses the tweak values.

If I go back, in a day or so, and review the AB test, it won't show all of the tweak values I'm using.

Is this just me?




1 reply

Userlevel 3
Badge

@appstoreveteran-

A few things that might help you in your situation. It sounds like you are noticing that what is reflected in the A/B test builder is not how you originally designed the A/B test.

Firstly, it is important to note that no default values are stored within the Mixpanel UI - so if some of your tweak values are actually default values they will not be saved in the AB test builder when you go back to review it.

Secondly, the A/B test editor reflects the state of the device that it is connected to. Since it uses a web-socket connection, the device connected to the A/B test editor will display any variants for A/B tests that it has already qualified for; however, if the test device or emulator is newly installed or you are working with a new tweak that has not been deployed in an existing an A/B test, the editor will display the default values instead, and changes in the UI will persist.

To help, you can try the following:

A. Create a testing device and filter out the profile for that device each time you target users in an A/B test. — This will prevent your testing device from qualifying for an A/B test, and each time you go to edit your A/B tests there will not be existing variants applied to the device that could interfere.

B. Use a new testing device, or uninstall and reinstall the app before you connect to the A/B test editor. — This will clear the local storage and when we hit the Mixpanel /decide API there will not be an applicable A/B test to deploy on the device. Just remember that A/B tests are applied on the second page of the app, so if you uninstall and reinstall the device to edit the A/B test, you will need to stay on the first page, or a splash page.


I hope this helps!


Reply