how to curate a list of user properties and event properties while still doing a raw dump of the Mixpanel data using Mixpanel EXPORT DATA API. Play devil advocate and determine if this can be done and the effort required to update this or is a raw dump to a data platform or lake recommended. We want to use a PULL method and not a PUSH method where we establish with Mixpanel data integration sync feature
Hi event, Please stand by as I search the Mixpanel knowledge base for an AI-powered answer. I normally take about 15 seconds or so. In case AI cannot answer your question, feel free to reach out to Mixpanel support at mixpanel.com/get-support. If you have follow-up questions for this thread, please tag me again with your question: @kapa.ai <follow-up question> 👋
Feel free to ask a follow-up question once I've answered by mentioning me with @kapa.ai
To curate a list of user properties and event properties while still doing a raw dump of Mixpanel data using the Export Data API (i.e., using a PULL method), here’s what’s possible and the trade-offs involved:
1. Event Export (Raw Event Export API)
- Filtering Events:
You can filter which events to export using the event parameter (as a JSON array) and further refine with the where parameter for property-based filtering.
Example:
curl --request GET \
--url '[https://data.mixpanel.com/api/2.0/export?from_date=2024-07-23&to_date=2024-07-24&event=["sample_event"]](https://data.mixpanel.com/api/2.0/export?from_date=2024-07-23&to_date=2024-07-24&event=[%22sample_event%22])' \
--header 'accept: text/plain' \
--header 'authorization: Basic YOUR_TOKEN'
This will export only the specified events, but all properties for those events will be included in the output. There is no built-in way to select only a subset of event properties in the raw export; you will need to post-process the JSONL output to remove unwanted properties yourself Export Methods.
2. User Profile Export (Engage API)
- Filtering Properties:
The Engage API allows you to specify which user profile properties to return using the output_properties parameter.
Example:
output_properties=["$last_name", "$email", "Total Spent"]
This lets you curate the list of user properties in your export Exporting Profiles.
3. Effort Required to Update Curated Lists
- Event Properties: Since the Raw Event Export API does not support selecting specific event properties, you must always export all properties and then filter them in your data pipeline or ETL process. This means: - Initial setup: Write a script or use a data processing tool to parse the JSONL and keep only the properties you want. - Ongoing maintenance: If your list of desired properties changes, you only need to update your post-processing logic, not the Mixpanel export configuration.
- User Properties:
The Engage API makes it easy to update which properties you export—just change the output_properties parameter in your request.
4. Devil’s Advocate: Is Curation Worth It?
- Pros of Curation: - Reduces storage and processing costs downstream. - Keeps your data warehouse/lake cleaner and more relevant. - Easier to comply with privacy/data minimization requirements.
- Cons of Curation: - Event Properties: Since Mixpanel’s API does not support property-level filtering for events, you must always handle the full payload and filter after export. This adds complexity and processing time. - Maintenance: You must maintain your own scripts or ETL jobs to curate the data, and update them as your schema evolves.
- Raw Dump Recommendation: - If you want the simplest, most future-proof approach, a raw dump (exporting all events and all properties) is recommended. This ensures you never miss a property you might need later, and you can always curate downstream as requirements change. - If you have strict storage or compliance needs, invest in a robust ETL process to curate after export.
5. Summary Table
In conclusion: You can curate user properties at export time, but for event properties, you must export all and filter after. This means a raw dump is often the most practical approach, especially if you want to minimize ongoing maintenance and ensure you capture all data for future needs. If you need to curate, be prepared to handle this in your data pipeline, not in the Mixpanel export step Export Methods Export Methods.
If you want a fully managed, property-level curation at export, Mixpanel’s APIs do not currently support this for events—you must use a PULL method and handle curation yourself.
kapa.ai is still learning and improving, please let me know how I did by reacting below.
ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.