Managing Exports for Over 50,000 Rows: Best Practices
what happens if we have more than 50000 rows to export how do we manage that?
what happens if we have more than 50000 rows to export how do we manage that?
Hi event, Please stand by as I search the Mixpanel knowledge base for an AI-powered answer. I normally take about 15 seconds or so. In case AI cannot answer your question, feel free to reach out to Mixpanel support at mixpanel.com/get-support. If you have follow-up questions for this thread, please tag me again with your question: @kapa.ai <follow-up question> 👋
Feel free to ask a follow-up question once I've answered by mentioning me with @kapa.ai
If you try to export a report as CSV from the Mixpanel UI and your report contains more than 50,000 rows (for Insights reports in Bar, Stacked Bar, Pie, Table, or Metric view), only the first 50,000 rows will be exported. For all other views and reports, the export limit is 10,000 rows. If your report exceeds these limits, only the number of rows up to the limit in ascending order is returned. There is no built-in way in the UI to export more than these limits in a single export.
To manage exports larger than these limits, you can:
- Use filters to reduce the number of rows in your report and export multiple filtered CSVs. - For raw event data (not formatted reports), use the Raw Event Export API, which does not have the same row limits and is designed for high-volume exports. This API allows you to export all your event data as JSONL, and you can use parameters to paginate or filter your exports as needed Export Methods: Export raw data and reports from Mixpanel. - For formatted report data, there is no documented method to export more than the UI or API limits in a single request. You would need to break your queries into smaller segments using filters or breakdowns.
If you need to export more than the allowed rows for a report, consider whether you can use the Raw Event Export API or adjust your report to fit within the export limits Reports Overview: Segment Limits and Ordering.
kapa.ai is still learning and improving, please let me know how I did by reacting below.
ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
There is another issue of duplicate values in the insight report, how do I know we passed another unique identifier that allows us to count unique users so there are instances where the same user is counted twice and this is fact checked with the foreign key identifier present in both events for the unique identifier captured and where it is showing a device ID instead How to explain to a client and manage expectation kapa.ai
Let me check the knowledge base for you again. One moment! 📚
Duplicate user counts in Mixpanel Insights reports typically occur when the same user is tracked with different identifiers (such as a device ID before login and a user ID after login) and these identifiers have not been properly merged. This can result in the same user being counted twice—once under their anonymous device ID and once under their authenticated user ID.
How to explain this to a client:
Mixpanel uses a unique identifier called distinct_id to count users. Before a user logs in, events are tracked with an anonymous device ID (e.g., $device_id). After login, events should be tracked with a persistent user ID (e.g., $user_id). If the implementation does not properly merge these two identifiers (by calling identify() at login), Mixpanel will treat them as separate users, leading to duplicate counts in reports—even if both events have the same foreign key or unique identifier in your own system.
Managing expectations and best practices:
- ID Merge is essential: To ensure accurate unique user counts, always call identify() with the user's unique ID at login or registration. This merges the anonymous device ID with the authenticated user ID, so Mixpanel recognizes all events as coming from the same user, regardless of which identifier was used before or after login (Identifying Users (Simplified)).
- Check for consistent identifiers: If you see duplicate users in your reports, it is likely because some events are still being sent with only the device ID, and others with the user ID. Review your implementation to ensure that after login, all events include the user ID and that identify() is called appropriately (Identity Management in Action).
- User Profiles and foreign keys: Mixpanel’s unique user count is based on distinct_id, not on any foreign key or custom property you may pass. Even if your events include a foreign key, Mixpanel will not use it for deduplication unless it is set as the distinct_id or properly merged via the identity management process (Identity Management Overview).
- Expectation: If identity management is not implemented correctly, you may see the same user counted twice—once as an anonymous device and once as an identified user. This is expected behavior unless the IDs are merged.
Summary for the client:
Duplicate user counts happen when Mixpanel receives events for the same person under different identifiers. To avoid this, ensure your implementation merges device and user IDs at login using identify(). Mixpanel counts unique users based on its internal identifiers, not on your foreign keys, so proper identity management is critical for accurate reporting (FAQ: Identity Management in Mixpanel).
kapa.ai is still learning and improving, please let me know how I did by reacting below.
ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
Let me check the knowledge base for you again. One moment! 📚
If you are seeing duplicate users in Mixpanel reports even after login and after calling identify(), and you can confirm this by a secondary unique identifier (such as a foreign key) present in both events, there are a few important points to consider:
1. Mixpanel’s Unique User Logic:
Mixpanel determines unique users based solely on the distinct_id property. Even if you pass a secondary unique identifier as an event property, Mixpanel will not use it for deduplication or user counting. Only the distinct_id (which should be your persistent user ID after login) is used for user uniqueness in reports (Identity Management Overview).
2. Possible Causes of Duplicates After Login:
- If you see the same secondary unique identifier (your foreign key) associated with different distinct_id values in Mixpanel, it means that events are still being sent with different identifiers (e.g., some with device ID, some with user ID) even after login.
- This can happen if identify() is not called consistently, or if there is a delay or issue in the merging process. Mixpanel may also randomly select which ID to display as the canonical distinct_id for a user cluster, but all IDs in the cluster should resolve to the same user (Identifying Users (Original) FAQ).
3. Mixpanel’s Merging Process:
When identify() is called, Mixpanel merges the anonymous device ID and the user ID into a single identity cluster. However, if events are sent with both IDs after login (for example, if the SDK is not properly initialized or if there is a race condition), Mixpanel may temporarily treat them as separate users until the merge is fully processed. The mapping can take up to 24 hours to propagate to all parts of the system, though the Activity Feed updates in real time (Identifying Users (Simplified) FAQ).
4. How to Explain to a Client:
- Mixpanel counts unique users based on its internal distinct_id, not on any secondary identifier you provide.
- If you see the same foreign key under different distinct_ids, it means the identity merge did not occur as expected, or there is a delay in the merge propagating.
- To avoid this, ensure that after login, all events are sent with the correct user ID as the distinct_id, and that identify() is called immediately after login.
- It is also recommended to track your user ID as a super property or user property for troubleshooting, but this does not affect Mixpanel’s unique user counting logic (Best Practices).
In summary:
Even after login, if events are sent with different distinct_ids for the same user (as confirmed by your secondary unique identifier), Mixpanel will count them as separate users. The only way to ensure accurate unique user counts is to implement identity management correctly and consistently, as Mixpanel does not use secondary identifiers for deduplication. If the issue persists after correct implementation, allow up to 24 hours for the merge to propagate throughout the system.
If you need to troubleshoot further, compare the distinct_id values and the secondary identifier in your raw event data to identify where the split is occurring, and review your implementation to ensure identify() is called at the right time and all events after login use the correct user ID (Identity Management in Mixpanel FAQ).
kapa.ai is still learning and improving, please let me know how I did by reacting below.
ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
Let me check the knowledge base for you again. One moment! 📚
Thank you for clarifying. Based on the knowledge sources, here’s how to explain and manage this situation:
Why is Mixpanel using device ID ($device_id) as the distinct_id after login, even when the user is identified?
- Mixpanel determines the unique user in reports based on the distinct_id property. After login, if identify() is not called correctly or if there is a timing issue, some events may still be sent with the device ID as the distinct_id instead of the user ID. This can result in the same user being counted twice: once under their device ID and once under their user ID, even if your own secondary unique identifier (e.g., a foreign key) confirms it is the same person (Identifying Users (Simplified) FAQ).
- Mixpanel does not use any secondary unique identifier you pass (such as a foreign key) for deduplication or user counting. Only the distinct_id (which should be your persistent user ID after login) is used for unique user counts in reports (Identity Management in Mixpanel FAQ).
How to explain this to a client:
- Mixpanel’s unique user counts are based solely on its internal identity management system, which uses distinct_id. If events after login are still being sent with the device ID as the distinct_id, Mixpanel will treat these as separate users, even if your own unique identifier shows they are the same person.
- This can happen if identify() is not called immediately after login, or if there is a delay in the merge process. Mixpanel may take up to 24 hours to fully propagate identity merges across all reports, though the Activity Feed updates in real time (Identifying Users (Simplified) FAQ).
- The secondary unique identifier you pass is useful for your own analysis and troubleshooting, but Mixpanel does not use it for user deduplication.
Managing expectations:
- To ensure accurate unique user counts, always call identify() as soon as the user logs in, and ensure all subsequent events are sent with the user ID as the distinct_id.
- If you see device IDs being used after login, review your implementation to ensure the identity merge is happening as expected.
- Explain to the client that Mixpanel’s user counting is based on its own identity logic, not on any custom or secondary identifier, and that this is a limitation of the platform (Identifying Users (Simplified)).
If you have confirmed that identify() is being called correctly and the issue persists, it may be necessary to allow time for the merge to propagate, or to review the event stream for any implementation gaps.
Summary for the client:
Mixpanel counts unique users based on its internal distinct_id, not on your secondary unique identifier. If you see the same user counted twice (once as device ID, once as user ID), it means the identity merge did not occur as expected, or there is a delay. Ensure identify() is called at login and all events after login use the user ID. The secondary identifier is not used by Mixpanel for deduplication, so discrepancies can occur if identity management is not fully aligned with Mixpanel’s requirements.
kapa.ai is still learning and improving, please let me know how I did by reacting below.
ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.