Solved

Advice and best practices on a tracking plan for a PaaS product

  • 26 March 2020
  • 5 replies
  • 166 views

Hi there,

First and foremost, hope everyone is safe at home!

We’re currently starting to implement Mixpanel to track our key product metrics and understand user behaviours in their first steps on the product.

In a nutshell, we develop a platform-as-a-service that allows users to create and deploy AI and data applications, going from the collection of raw data, through data preparation and transformation, up to the development of a front-end interface or the deployment of APIs for machine-to-machine consumption.

We’re currently in the process of wrapping up our tracking plan and as we’re still pretty new to Mixpanel concepts, we’d love to have a more expert person having a look at it. The tracking plan can be accessed here

Our goal is to track users’ actions in their first steps on the platform and better understand their behaviours to increase early adoption and improve the “self-service-ness” of our product, from a data source creation to the deployment of an application.

We also have a question regarding the fact that each user can access several “DataPlants”, i.e. dedicated and fully-fledged environments to build and deploy data projects. How can we track the fact that a user can take actions in different environments? (DataPlant as super property?)

Thanks a lot for your help. 

Take care

icon

Best answer by MP_Rich 28 March 2020, 00:22

Hey There!

 

My name’s Rich. I’m an implementation manager at Mixpanel. I’ve taken a look through your spec. For a first pass, it’s pretty good! A few quick notes below:

  • You’ve done this in a lot of places, but where possible, add IDs in for the entity you’re tracking. I’ve noticed you’ve done it for Workflows and Dataplants but haven’t done it for the user’s organization. Within Mixpanel, the IDs aren’t going to be super useful. However, if you ever need to export that data and join it with other data (i.e. in your data warehouse), it’s generally a lot easier to join on IDs
  • You can use Mixpanel to track success rate of certain tasks as well as their latency. For example, with deploying and building an app you can have two events -- “deploy app start” and “deploy app complete.” On the “deploy app complete” event you can add properties like the amount of time it took to deploy, whether the deploy was successful, an error reason if it failed, and properties on the size of the app to correlate that with deploy times/errors.
  • Incremental event properties generally don’t work very well. Mixpanel can only increment super properties. However, because super properties are stored within the device’s local storage, if a user were to switch devices, the super property value wouldn’t persist. So, a user may have a value of 4 for the “attributes set” super property on one device and if they were to switch devices, the super property would start at 0 resulting in some inconsistent data. The way around this is to either pass the correct value (e.g. pass “4” instead of “+1”) or to use incremental user properties as user properties are stored within Mixpanel’s database and not on the user’s local device.
  • Similar to the above point, when a user logs in, you should set all relevant user and super properties again (such as their company, org, etc.). In the event that a user clears their cookies/cache or is logging in from a new device, the super properties won’t exist and you can use the point of login as a great time to set them and ensure they’ll exist going forward.
  • With the “Explored URL” event, if you’re sending in the destination URL, you should ensure that they’ll be standardized in some kind of way. For example, if the URL is url.com/module/page/dynamic_string_123/dynamic_variable_456/, then it may be best to parse that URL so that only url.com/module/page is sent and the dynamic pieces are dropped off. The reason is that Mixpanel aggregates properties on an exact match. So, if the URLs are the exact same except one character is lowercase and one is uppercase, Mixpanel will treat them as two different values and they won’t be aggregated in the reporting. 

Hopefully that helps! Like I said earlier, for a first pass, this is a pretty good spec!

View original

5 replies

Hey There!

 

My name’s Rich. I’m an implementation manager at Mixpanel. I’ve taken a look through your spec. For a first pass, it’s pretty good! A few quick notes below:

  • You’ve done this in a lot of places, but where possible, add IDs in for the entity you’re tracking. I’ve noticed you’ve done it for Workflows and Dataplants but haven’t done it for the user’s organization. Within Mixpanel, the IDs aren’t going to be super useful. However, if you ever need to export that data and join it with other data (i.e. in your data warehouse), it’s generally a lot easier to join on IDs
  • You can use Mixpanel to track success rate of certain tasks as well as their latency. For example, with deploying and building an app you can have two events -- “deploy app start” and “deploy app complete.” On the “deploy app complete” event you can add properties like the amount of time it took to deploy, whether the deploy was successful, an error reason if it failed, and properties on the size of the app to correlate that with deploy times/errors.
  • Incremental event properties generally don’t work very well. Mixpanel can only increment super properties. However, because super properties are stored within the device’s local storage, if a user were to switch devices, the super property value wouldn’t persist. So, a user may have a value of 4 for the “attributes set” super property on one device and if they were to switch devices, the super property would start at 0 resulting in some inconsistent data. The way around this is to either pass the correct value (e.g. pass “4” instead of “+1”) or to use incremental user properties as user properties are stored within Mixpanel’s database and not on the user’s local device.
  • Similar to the above point, when a user logs in, you should set all relevant user and super properties again (such as their company, org, etc.). In the event that a user clears their cookies/cache or is logging in from a new device, the super properties won’t exist and you can use the point of login as a great time to set them and ensure they’ll exist going forward.
  • With the “Explored URL” event, if you’re sending in the destination URL, you should ensure that they’ll be standardized in some kind of way. For example, if the URL is url.com/module/page/dynamic_string_123/dynamic_variable_456/, then it may be best to parse that URL so that only url.com/module/page is sent and the dynamic pieces are dropped off. The reason is that Mixpanel aggregates properties on an exact match. So, if the URLs are the exact same except one character is lowercase and one is uppercase, Mixpanel will treat them as two different values and they won’t be aggregated in the reporting. 

Hopefully that helps! Like I said earlier, for a first pass, this is a pretty good spec!

Thanks a lot @MP_Rich, really useful. 

Cheers!

Hi @MP_Rich, I’m Thomas from ForePaaS work with Marie on the Product team at ForePaaS. We integrated thoroughly your feedback but I have additional questions for you. See below the list of questions:

  • We have one particularity in our product is that users can join multiple organizations (think of it as user groups). At the moment I’ve updated the tracking plan such as on events which affect organization we update the profile properties of list type. Is that a good practice?

  • Why do most tracking plan samples track sign-up / log-in events only through profile properties and not with event properties as well? Should we track create / join organizations with profile properties only as well?

  • Should we set super properties concurrently to event properties?

    • First situation, should we update all three super, profile and event properties on trial activation? It feels a little redundant but I’m still unclear on how super properties are used.

    • Second situation, in our product users are asked to perform a certain number of tasks within specific “environments” called DataPlants. In other terms, the user is expected to Login, navigate to the right organization, choose a specific DataPlant and then start working. How should I go about tracking actions “per DataPlant”? Is this what super properties are for?

Please find our updated Tracking Plan here. Thanks a lot for your help. I’d be happy to also jump on a quick call to review all pending questions if that’s easier for you.

Best,

Thomas

Badge

Hi ForePaas Team, 

I’m Andrea, an Operations Specialist with the Customer Education Team. I’ve had some experience looking at customer tracking plans and hopefully I can provide you with some of the answers you’re looking for. 

Super Properties are useful for values that you would like to track how they change over time. Event properties are only indicative of the value when the event occurred. 

For example, for a subscription based payment structure, you could register a Super Property with all of your Events that indicates whether they were done by a free or a paid user. Then, if a user changes from free to paid, you can see over time which Events led up to that conversion and which specific Events they did as each type of user. 

My advice to customers is to track values that change over time and that you’d want to analyze over time as a Super Property. For values that you don’t expect to change and isn’t usually part of business analysis such as name, email or company name would suffice as User Properties - however if they do change updating their values on the user profile would also suffice.  

In your second situation tracking ‘per DataPlants’ would make sense as a Super Property since the actions that users can do are the same and you would need to know which actions were performed in which environment. 

I hope this answer helps! 

-Andrea 

 

Yes that makes sense! Thank you.

Best,

Thomas

Reply