# Parse a large array of small JSON objects?

**URL:** https://discourse.julialang.org/t/parse-a-large-array-of-small-json-objects/61040
**Category:** Performance
**Tags:** json3
**Created:** [May 12, 2021, 7:01pm UTC](https://discourse.julialang.org/t/parse-a-large-array-of-small-json-objects/61040 "2021-05-12T19:01:40Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![Satvik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/satvik/32/20486_2.png) [@Satvik](https://discourse.julialang.org/u/Satvik)
#### Post date: [May 12, 2021, 7:01pm UTC](https://discourse.julialang.org/t/parse-a-large-array-of-small-json-objects/61040/1 "2021-05-12T19:01:40Z")

</div>

Hi,

I have a DataFrame that contains a column of several million JSON strings, e.g.

```julia
julia> DataFrame(j=""""{a": 1, "b": 2}""")
julia> pretty_table(df)
┌──────────────────┐
│ j │
│ String │
├──────────────────┤
│ "{a": 1, "b": 2} │
└──────────────────┘

```

The simple way to parse them is with something like `df[:, "j"] .= JSON3.read.(df.j)`. However, this appears to create a memory leak when I do it repeatedly on different datasets, even when `df` is no longer reachable, the memory appears to stay used and the program eventually runs into an OOM error. In contrast this doesn’t happen if I don’t do the JSON parsing.

Is there a better practice for parsing a lot of small JSON objects that I’m not aware of? I saw that JSON3 uses a semi-lazy evaluation method, and was wondering if that’s confusing the garbage collector somehow.

---

<div class="post-metadata">

### Author: ![ericphanson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ericphanson/32/215186_2.png) [@ericphanson](https://discourse.julialang.org/u/ericphanson)
#### Post date: [May 12, 2021, 7:12pm UTC](https://discourse.julialang.org/t/parse-a-large-array-of-small-json-objects/61040/2 "2021-05-12T19:12:59Z")

</div>

I think the default `JSON3.Object`’s etc will reference the object, but you can tell JSON3 how to use something else. E.g. `JSON3.read(obj, Dict)` says to parse `obj` into a regular `Dict` (which then does not share memory with the original object). If they are all of a certain structured form, you could also define a `struct` and use StructTypes to say how to (de)serialize to it, then do `JSON3.read(obj, MyStruct)` to parse into that structure.

---

<div class="post-metadata">

### Author: ![Satvik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/satvik/32/20486_2.png) [@Satvik](https://discourse.julialang.org/u/Satvik)
#### Post date: [May 12, 2021, 8:36pm UTC](https://discourse.julialang.org/t/parse-a-large-array-of-small-json-objects/61040/3 "2021-05-12T20:36:29Z")

</div>

`JSON3.read(obj, Dict)` solved the memory leak. Thanks!
