# Efficiently Read JSON and Create DataFrame

**URL:** <https://discourse.julialang.org/t/efficiently-read-json-and-create-dataframe/41087>\
**Category:** Performance\
**Tags:** json, dataframes\
**Created:** [June 9, 2020, 6:25pm UTC](https://discourse.julialang.org/t/efficiently-read-json-and-create-dataframe/41087 "2020-06-09T18:25:04Z")\
**Posts on this page:** 1\
**Showing post:** 6

<div class="post-metadata">

**Author:** ![lwabeke](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lwabeke/32/4005_2.png) [@lwabeke](https://discourse.julialang.org/u/lwabeke)\
**Post date:** [June 10, 2020, 11:44am UTC](https://discourse.julialang.org/t/efficiently-read-json-and-create-dataframe/41087/6 "2020-06-10T11:44:09Z")

</div>

Hi

There are a few JSON packages available: JSON.jl, JSON2.jl, JSON3.jl, LazyJSON.jl are ones I’m aware of.

See [[ANN] JSON3.jl - Yet another JSON package for Julia](https://discourse.julialang.org/t/ann-json3-jl-yet-another-json-package-for-julia/25625)  
for some details.

My summary:

- JSON.jl - The original and does proper handling, but slow and very memory intensive. Uses parse to parse into a dict
- LazyJSON.jl - Memory efficient, which makes it relatively fast. Parse also gives a “dict” type interface.
- JSON2.jl & JSON3.jl - Claims to be fast. Parses into a provided type. At some point I looked at them and got the impression one (or both?) was cheating a bit: It assumed the JSON fields would match the sequence of fields defined in the structure (to speedup, it didn’t check if label matched). This works if the JSON string is created by the same package, but could break if done by another encoder. I’m not sure if this is still the case and/or if I just misunderstood the code.

I would assume LazyJSON would shine if you only need to access a subset of the JSON, but as part of the testing of the [Unmarshal.jl](https://github.com/lwabeke/Unmarshal.jl) package I got the impression that even if I unmarshal the whole object LazyJSON could still outperform JSON, but it depends on the size and complexity of the structure.

The [Unmarshal.jl](https://github.com/lwabeke/Unmarshal.jl) package can be used to convert from the JSON.jl and LazyJSON.jl dict interface to a Julia type object, which might be an alternative to what your doing in:

```julia
simstates = vcat(DataFrame.(inDict[“simstates”])…)

```

It is however focused on functionality and not really performance, in particular since working with the original JSON.parse, it seemed the JSON.parse dominated timing compared to the Unmarshal step.

---

_[View the full topic](https://discourse.julialang.org/t/efficiently-read-json-and-create-dataframe/41087)._