Issues Exporting Large Data with Mixpanel API Pagination | Mixpanel Community

Issues Exporting Large Data with Mixpanel API Pagination

Eric N. kapa.ai Hello Team I was tying to export data where the data is huge and per day event count exceeds. That's why i was trying to do it using pagination at day level and for that i trued using where query param of export API but as soon as whre condition is specified it doesn't return any data

curl --location --globoff 'https://data.mixpanel.com/api/2.0/export?project_id=XXXXXX&from_date=2024-05-01&to_date=2024-05-14&time_in_ms=true&where=properties[%22time%22]%20%3C%201715449195&limit=1' \
--header 'Authorization: ••••••'

5 comments

kapa.ai
APP
·
·
Please stand by Abhishek C. as I search the knowledge base. I normally take about 15 seconds or so. 👋
Feel free to ask a follow-up question once I've answered by mentioning me with @kapa.ai
kapa.ai
APP
·
·
Thank you for your detailed question. Based on the knowledge sources, here are some important points and possible reasons why your export API call with the where parameter is not returning data:
1. where Parameter Usage
The where parameter in the Raw Event Export API is designed to filter events based on event properties. The syntax must follow Mixpanel's event selector expressions. For example, to filter by a property, you would use: LBKSwhere=properties["property_name"] == "value"LBKSIn your example: LBKSwhere=properties["time"] < 1715449195LBKSThis is syntactically correct if you are trying to filter events where the time property is less than a specific Unix timestamp.
2. The time Property
- The time property in Mixpanel events is the event timestamp, stored as a Unix timestamp (seconds since epoch). - However, the export API already filters events by the from_date and to_date parameters, which are based on the event's timestamp. - Filtering again on properties["time"] may not be necessary and could result in no data if the filter does not match any events within the already specified date range.
3. No Data Returned
If specifying the where parameter results in no data: - Double-check that the value in your where clause (1715449195) actually matches the range of event timestamps in your project for the given date range. - If your from_date and to_date already restrict the data to a certain window, and your where clause further restricts it to a non-overlapping or empty set, you will get no results. - Ensure that the property name and value types are correct and that the property exists on the events you are querying.
4. Rate Limits and Pagination
- The export API does not support traditional pagination, but you can break up your export by day or by using different time windows in your requests, as you are doing. - The API has a rate limit of 60 queries per hour, 3 queries per second, and 100 concurrent queries. Make sure you are not hitting these limits, as a 429 error would indicate rate limiting, not a data issue Rate Limits.
5. Example from Mixpanel Docs
Here is an example of a correct export API call with a where clause: LBKScurl --request GET \ --url '[https://data.mixpanel.com/api/2.0/export?from_date=2024-07-23&to_date=2024-07-24&event=["sample_event"]&where=properties["property_name"]=="value"](https://data.mixpanel.com/api/2.0/export?from_date=2024-07-23&to_date=2024-07-24&event=[%22sample_event%22]&where=properties[%22property_name%22]==%22value%22)' \ --header 'accept: text/plain' \ --header 'authorization: Basic YOUR_TOKEN'LBKSExport Methods: Export raw data and reports from Mixpanel
6. Additional Notes
- If you want to export large volumes, consider splitting your requests by smaller date ranges (e.g., one day at a time) and avoid unnecessary where clauses unless you need to filter by a property other than time. - The limit parameter can be used to restrict the number of events returned, but cannot exceed 100,000 per request.
Summary
- Double-check your where clause logic and the values you are filtering on. - The from_date and to_date parameters already filter by event time, so additional filtering on properties["time"] may not be needed and could result in no data if the ranges do not overlap. - If you continue to have issues, try removing the where clause and see if data is returned, then incrementally add filters to isolate the problem.If you need more help, please provide the exact values you are using for from_date, to_date, and the expected range for properties["time"] so we can help debug further.References: - Event Export API - Export - Export Methods: Export raw data and reports from Mixpanel - Rate Limits
kapa.ai is still learning and improving, please let me know how I did by reacting below.
ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
Abhishek C.
·
·
even if i add where condition fo oter feiuld it doesn't work kapa.ai
kapa.ai
APP
·
·
Let me check the knowledge base for you again. One moment! 📚
kapa.ai
APP
·
·
Thank you for clarifying. Based on the knowledge sources, the Mixpanel Raw Event Export API does support the where parameter for filtering events by event properties. The syntax should be: LBKSwhere=properties["property_name"] == "value"LBKSHowever, if you are finding that no data is returned even when filtering on other fields, here are some key points and limitations to consider: 1. The where parameter only works on event properties The documentation specifies that the where parameter is used to filter events by event properties. If the property you are filtering on does not exist on the events in the specified date range, or if there is a typo or case mismatch (Mixpanel is case sensitive), the query will return no data. Double-check the property name and value you are using in your filter.2. Property must exist on the events If you filter on a property that is not present on any events in the date range, the result will be empty. You can confirm which properties exist by inspecting your events in the Mixpanel UI or by exporting a small sample without a where clause first.3. Correct syntax is required The where clause must be a valid JavaScript expression referencing event properties, e.g.:
```LBKS If the syntax is incorrect, the API may not return data.LBKS4. **No explicit mention of known issues or bugs** The knowledge sources do not mention any known issues or bugs with the `where` parameter itself in the Export API. If you are following the correct syntax and filtering on a property that exists, it should work as expected according to the documentation [Event Export API - Export](https://github.com/mixpanel/docs/blob/main/openapi/src/export.openapi.yaml).LBKS5. **Case sensitivity** Mixpanel is case sensitive for property names. Ensure you are using the exact casing as stored in your events.LBKS6. **No authentication or permission issues documented** The documentation does not mention that authentication or permissions would affect the `where` parameter specifically.LBKS**If you are still not getting results:** - Try running the export without the `where` parameter to confirm data is returned. - Inspect a sample of the returned events to verify the exact property names and values. - Add the `where` parameter back, ensuring the property and value match exactly.LBKS**If the issue persists despite following all the above, the knowledge sources do not provide further troubleshooting steps or known limitations for this scenario.** You may need to contact Mixpanel support for deeper investigation.LBKSReferences: - [Event Export API - Export](https://github.com/mixpanel/docs/blob/main/openapi/src/export.openapi.yaml)
- Export Methods: Export raw data and reports from Mixpanel
kapa.ai is still learning and improving, please let me know how I did by reacting below.
ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

Issues Exporting Large Data with Mixpanel API Pagination

5 comments

1. `where` Parameter Usage

2. The `time` Property

3. No Data Returned

4. Rate Limits and Pagination

5. Example from Mixpanel Docs

6. Additional Notes

Summary

5 comments

1. where Parameter Usage

2. The time Property

3. No Data Returned

4. Rate Limits and Pagination

5. Example from Mixpanel Docs

6. Additional Notes

Summary

1. `where` Parameter Usage

2. The `time` Property