# A way to push to a JSON IO like with a collection

**URL:** <https://discourse.julialang.org/t/a-way-to-push-to-a-json-io-like-with-a-collection/20990>\
**Category:** General Usage\
**Tags:** question, json\
**Created:** [February 19, 2019, 7:55pm UTC](https://discourse.julialang.org/t/a-way-to-push-to-a-json-io-like-with-a-collection/20990 "2019-02-19T19:55:24Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [February 19, 2019, 7:55pm UTC](https://discourse.julialang.org/t/a-way-to-push-to-a-json-io-like-with-a-collection/20990/1 "2019-02-19T19:55:24Z")

</div>

I’m running a somewhat long and slow iteration. I want to incrementally save the result of each iteration in a singular JSON file. The problem is that when the file is closed (because of early termination or because the loop is simply done), and I want to load the content of the JSON file (with `JSON.parsefile`) the resulting object should be an **Array** of `Dict`s. Is there any way of doing that…?

The rational is that in the same way you’d `push!` the result from the iteration into some `Vector` you’re collecting the results in (to later do what you will with), I’d like to “push” into a JSON file.  
MWE that does not work:

```julia
x = rand(10)
open("tmp.json", "a") do io
    for i in x
        y = sin(i)
        JSON.print(io, Dict(:k => i, :v => y), 3)
    end
end

```

---

<div class="post-metadata">

**Author:** ![quinnj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/quinnj/32/11_2.png) [@quinnj](https://discourse.julialang.org/u/quinnj)\
**Post date:** [February 19, 2019, 8:15pm UTC](https://discourse.julialang.org/t/a-way-to-push-to-a-json-io-like-with-a-collection/20990/2 "2019-02-19T20:15:21Z")

</div>

You need to actually specify the array in the JSON, so something like:

```julia
x = rand(10)
open("tmp.json", "a") do io
    print(io, "[")
    for i in x
        y = sin(i)
        JSON.print(io, Dict(:k => i, :v => y), 3)
       i == x[end] || print(io, ",")
    end
    print(io, "]")
end

```

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [February 19, 2019, 8:20pm UTC](https://discourse.julialang.org/t/a-way-to-push-to-a-json-io-like-with-a-collection/20990/3 "2019-02-19T20:20:22Z")

</div>

Thanks! But this won’t work with the `append` flag for the `open`, nor would it work if the iteration abruptly stops midway.  
I guess there’s no real way of accomplishing this with JSON files…

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [February 19, 2019, 8:26pm UTC](https://discourse.julialang.org/t/a-way-to-push-to-a-json-io-like-with-a-collection/20990/4 "2019-02-19T20:26:51Z")

</div>

It’s a pretty cool thought though: Link an array with a file. Any change to the array is reflected in the file…

In this case the array is a `Vector` of `Dict`s and the file is a `JSON` file. But it could be any collection and any file format…

---

<div class="post-metadata">

**Author:** ![bennedich](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bennedich/32/4894_2.png) [@bennedich](https://discourse.julialang.org/u/bennedich)\
**Post date:** [February 19, 2019, 8:28pm UTC](https://discourse.julialang.org/t/a-way-to-push-to-a-json-io-like-with-a-collection/20990/5 "2019-02-19T20:28:09Z")

</div>

As for the abrupt termination, you could wrap it in `try` / `finally`. But the desire to append makes it trickier. As you say, JSON does not seem like a good choice here. What does the data look like for each iteration? How about a CSV file or other simple format where you just add a new line per iteration?

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [February 19, 2019, 8:34pm UTC](https://discourse.julialang.org/t/a-way-to-push-to-a-json-io-like-with-a-collection/20990/6 "2019-02-19T20:34:13Z")

</div>

> [@bennedich](#):
>
> What does the data look like for each iteration?

Currently, it’s a collection of pairs of keys (like a UUID) each pointing at a `Matrix{Float64}` with 3 columns (x, y, and t) and a variable number of lines (anything from 1 line to 1000 lines). Something like this:

```julia
using UUIDs, StructArrays
d = StructArray((k = uuid1(), xyt = rand(rand(1:1000), 3)) for _ in 1:100)

```

So I guess I could flatten it out by printing each row `key, x, y, t` and thus “needlessly” repeating the key multiple times…? Hmm…  
Like this:

```julia
using UUIDs, StructArrays
d = StructArray((k = uuid1(), xyt = rand(rand(1:1000), 3)) for _ in 1:100)
open("tmp.csv", "a") do io
    for xyt in 1:100
        k = uuid1()
        for _ in 1:rand(1:1000)
            println(io, join([k; rand(3)], ','))
        end
    end
end

```

---

<div class="post-metadata">

**Author:** ![bennedich](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bennedich/32/4894_2.png) [@bennedich](https://discourse.julialang.org/u/bennedich)\
**Post date:** [February 19, 2019, 8:50pm UTC](https://discourse.julialang.org/t/a-way-to-push-to-a-json-io-like-with-a-collection/20990/7 "2019-02-19T20:50:49Z")

</div>

For appending data, I think it’s cleaner to just append new lines that way. (Or use a proper database / message queue.)

If you really want to use JSON, one option would be to read the existing JSON file into an array in memory at startup, append to this array, and serialize the entire array to disk for each iteration. Sure, it won’t be the most efficient, but if the iterations are slow anyway, it might not matter.

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [February 19, 2019, 8:53pm UTC](https://discourse.julialang.org/t/a-way-to-push-to-a-json-io-like-with-a-collection/20990/8 "2019-02-19T20:53:15Z")

</div>

> [@bennedich](#):
>
> but if the iterations are slow anyway, it might not matter.

That’s true.

> [@bennedich](#):
>
> use a proper database / message queue

I’m afraid I’m not very SQL savy, is there an option like that in say JuliaDB or any of the other DBverses?

---

<div class="post-metadata">

**Author:** ![pfitzseb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pfitzseb/32/45566_2.png) [@pfitzseb](https://discourse.julialang.org/u/pfitzseb)\
**Post date:** [February 21, 2019, 10:02am UTC](https://discourse.julialang.org/t/a-way-to-push-to-a-json-io-like-with-a-collection/20990/9 "2019-02-21T10:02:35Z")

</div>

How about something like

```julia
using JSON
xs = rand(10)
open("tmp.json", "a") do io
    tmpbuffer = IOBuffer()
    try
        println(io, '[')
        for (i, x) in enumerate(xs)
            y = sin(x)
            i == 8 && error()

            JSON.print(tmpbuffer, Dict(:i => i, :x => x, :v => y), 2)
            print(tmpbuffer, ',')

            print(io, String(take!(tmpbuffer)))
        end
    finally
        skip(io, -1) # get rid of the last comma which JSON doesn't allow
        println(io, ']')
    end
end

JSON.parse(read("tmp.json", String))

```

Should be relatively robust ~~unless your error happens during the `JSON.print` call. If that is possible (e.g. with interrupts) it’d be best to write that to a temp buffer.~~ Still doesn’t help with file system errors, but eh…

---

<div class="post-metadata">

**Author:** ![bennedich](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bennedich/32/4894_2.png) [@bennedich](https://discourse.julialang.org/u/bennedich)\
**Post date:** [February 21, 2019, 10:38am UTC](https://discourse.julialang.org/t/a-way-to-push-to-a-json-io-like-with-a-collection/20990/10 "2019-02-21T10:38:50Z")

</div>

This is the `try` / `finally` solution I referred to, but I would recommend against manually generating JSON this way since it’s easy to create bugs and problems for oneself. This code also doesn’t support appending properly (if I understand OP’s needs). I guess you could start by erasing the last closing bracket, but… no, JSON is just not a good fit for this problem (unless serializing the entire data structure for each iteration, which I think is clean, but inefficient).

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [February 21, 2019, 10:42am UTC](https://discourse.julialang.org/t/a-way-to-push-to-a-json-io-like-with-a-collection/20990/11 "2019-02-21T10:42:38Z")

</div>

I tend to agree with this… JSON is a bad format for this kind of work. CSV would also fail if the process got interrupted mid-row, but it’s easier to deal with. Having said that, it would be cool if there was a way to incrementally write data to a file in a safe way. JuliaDB might want to work on that…?