Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I have a funny story I need to tell some day about how I could get a 4GB JSON loaded purely in the browser at some insane speed, by reading the bytes, identifying the "\n" then making a lookup table. It started low stakes but ended up becoming a multi-million internal project (in man-hours) that virtually everyone on the company used. It's the kind of project that if started "big" from the beginning, I'd bet anything it wouldn't have gotten so far.

Edit: I did try JSON.parse() first, which I expected to fail and it did fail BUT it's important that you try anyway.





Curious about which browser and hardware. In my experience browsers often choke on 0.5GB strings, or decide to kill the tab/proccess.

Yes, but I didn't read the full file, I kept the File reference and read the bytes in pages of 10MB IIRC to find all of the line break offsets. Then used those to slice and only read the relevant parts.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: