I always wanted to try JSON3 but I was afraid to do so, JSON3.Object always scared me away. This weekend, I did it as a pet project, extending one our library to be compatible with JSON3. First of all, it was relatively easy, as most of changes corresponds to replacing Vector with Union{Vector,JSON3.Array} and Dict with Union{Dict,JSON3.Object} in method signatures. The speed improvement was massive. The loading time of one simulation was cut down from 13.5 hours to 720s, which is something like 62x improvement. It is probably specific for our project, where we load large JSON files and use just their subset. Never the less I am happy for that. One rarely gets such a speedup for so little work.
17 Likes
if data loading is such a large part of your calculation time, you really shouldn’t be using JSON if at all possible. Binary formats like parquet are often 10s to 100s of times faster to load.
3 Likes
Unfortunatelly, this is outside of my control. We have been looking at various formats three years ago and we did not find a clear winner, therefore we have sticked with JSONs. I think at that time, Parquet was not really well supported in Julia. I have never used it, so I do not know, if I can just take JSONs and store them in Parquet.
One thing I believe helps our use-case is that JSON3 uses Symbols as keys whereas JSON use String.