Solved

Node.Js Data upload taking Forever


Userlevel 2

Been on this for over 2 hours and it just uploaded a litter over 1300 records, am I doing something wrong? Video says 5 mins

 

icon

Best answer by ak-at-mixpanel-se 29 July 2021, 16:58

View original

12 replies

Thought it was only me...:thinking:  Experiencing same difficulty here

 

 

Same here, with exactly the same script form ProductSchool - every time it is stuck at some point. Tried it like 20 times so far. But 2 times it even succeeded to create 5000 user profiles, but then stuck on sending data in batches :joy: Any workaround here? Things to try??

Image for illustration

Same here, with exactly the same script form ProductSchool - every time it is stuck at some point. Tried it like 20 times so far. But 2 times it even succeeded to create 5000 user profiles, but then stuck on sending data in batches :joy: Any workaround here? Things to try??

The Same Problem is here :) 

Userlevel 1
Badge

@Terick @AmirH_Jandaghian @Julia @chuka760 

(EDIT: see https://community.mixpanel.com/sending-data-to-mixpanel-11/node-js-data-upload-taking-forever-5574?postid=9136#post9136)

Hi all!

Thanks for writing in. 2 hours is definitely too long to wait! :cold_sweat:

It seems to me that all of you are running windows (windows7? windows8?) … i’m still trying to find a proper windows environment to test out why it’s running slow.

 

AS A WORKAROUND TO GET YOU GOING:

If you peer down further in the script to lines 60 + 61 (in the DATA CONFIG), you’ll find two lines in JSON which define the total number of users AND the total number of events (per user) in the simulation:

the data config

https://gist.github.com/ak--47/9970ab52ee55723f05999b1ef1a0bc73#file-productschool-js-L60-L61

 

Try using smaller numbers here, but be sure to match the formatting exactly (no quotes!)… so

numberOfEvents: 50000, numberOfUsers: 5000,

might become

numberOfEvents: 5000, numberOfUsers: 1000,

this will generate less data, but should move the process along faster.

 

Sending less data will mean that the numbers in the video series won’t line up directly with the numbers you see in the video

 

If that’s not your preference, I’ll spend some time debugging in windows-land and try to come back with a better solution.

 

Apologies for this snag! It was not anticipated!

Userlevel 1
Badge

@Terick @AmirH_Jandaghian @Julia @chuka760 

I have an alternative for you that might be a bit easier - essentially we’re going to run the script in the cloud through your web browser! No terminal needed!

I found a free service called Repl.IT that allows me to publish code, which you can run in your browser - which means you don’t need to worry about choppy internet connections or navigating your terminal. 

You still will need to modify the script file so that your token/secret match

 

ALTERNATIVE STEPS TO LOAD THE DATA

  • Sign Up for a Free Repl.IT account: 

https://replit.com/signup?from=landing

  • Find the hosted version of my script

https://replit.com/@akTunes/ProductSchool#index.js

  • Click FORK (top right of the main box)

 

  • Now you’ll see a plain text editor on the left pane and a terminal in the right pane. As with before, modify the token secret to match yours project!
  • then click the big green RUN button in the top center of the page…data should load much faster!  (leave your browser tab open while it runs!) 

 

Hope this helps! Let me know! Would love to get you all activated as quickly as possible!

 

Thanks,

AK

This helped! I had been stuck on the error for awhile.Thanks AK! 

I ran the node script for the RandoSando data set on Repl.it and it was much faster than the local script.  It pulled in all 208k events with no problem.  But I’m at the module where it describes how to view events and change the names in the lexicon and I am not seeing the same events as the tutorial, just “event start” and “event end”.  I’ve retraced my steps and I can’t see what I missed.  Did I leave out a step?

 

And never mind….

Reset the data, reran the script and this time it came through.  The hamster powering my computer must not have been running fast enough in his wheel the first time.  

Thanks a lot Ak for this alternative. This has helped a great deal.

Hello.

I am on the ‘Load Data Into Mixpanel section’ trying to get into the settings but am receiving this message. Please advise.

 

‘Unable to view settings for this project’

 

Got it!

Reply