Memory usage with large JSON

General Usage Performance
1

I was using JSON to parse a large (1.6 GB) JSON file, and the memory usage was surprising. More surprisingly, perhaps, was that even after removing the reference to the parsed object and garbage collecting, the memory usage remained high:

using JSON
# write and parse a 25 byte JSON to compile
# Julia memory usage ~200 MB
d = JSON.parsefile("really_big.json")
# takes a long time
# Julia memory usage 12 GB
d = nothing
Base.GC.gc()
# Julia memory usage 8 GB

Is this pattern expected? Can anything be done to mitigate? Thanks!

2

Have you tried JSON3? From what I know it generally is just better than JSON

1 Like
3

The situation is indeed better with JSON3. Is this a known problem of standard lib JSON? Is JSON3 uniformly recommended instead?

4

As far as I know, it is. The main reason we still have JSON around is that we need something in the STDLIB that can parse JSON for package management stuff.

5

JSON isn’t a standard library, it’s just a normal package; maybe you’re thinking of TOML?

2 Likes
6

Shows what I know. Learned something new tonight–thanks!

1 Like