# HTTP Streaming Chunked JSON

**URL:** <https://discourse.julialang.org/t/http-streaming-chunked-json/82797>\
**Category:** Web Stack\
**Created:** [June 15, 2022, 8:09am UTC](https://discourse.julialang.org/t/http-streaming-chunked-json/82797 "2022-06-15T08:09:25Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![JohnDuffy](https://avatars.discourse-cdn.com/v4/letter/j/f14d63/32.png) [@JohnDuffy](https://discourse.julialang.org/u/JohnDuffy)\
**Post date:** [June 15, 2022, 8:09am UTC](https://discourse.julialang.org/t/http-streaming-chunked-json/82797/1 "2022-06-15T08:09:25Z")

</div>

Hi

I receive streamed new-line delimited JSON data from a REST API.

The data is streamed in chunks and full line of JSON may span multiple chunks. So, I need to somehow buffer the chunks as they arrive until I have a full line to read.

I have been reading the HTTP documentation but I don’t fully understand how to create an IO buffer for the received chunks.

I think the code should look something like the following:

```julia
url = ...
headers = ...

io = ??? # Help Required
with HTTP.open("GET", url, headers) do io
  for line in eachline(io)
    println(line)
  end
end

```

Any suggestions or help would be very much appreciated.

Kind regards

---

<div class="post-metadata">

**Author:** ![quinnj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/quinnj/32/11_2.png) [@quinnj](https://discourse.julialang.org/u/quinnj)\
**Post date:** [June 17, 2022, 5:53am UTC](https://discourse.julialang.org/t/http-streaming-chunked-json/82797/2 "2022-06-17T05:53:42Z")

</div>

I haven’t verified that this works, but I _think_ something like the following should work:

```julia
findnewline(bytes) = something(findfirst(==(UInt8('\n')), bytes), 0)
HTTP.open("GET", url, headers) do io
    conn = io.stream
    for bytes in readuntil(conn, findnewline)
        x = JSON3.read(bytes)
        # so stuff with parsed JSON object `x`
    end
end

```

We could probably define `readuntil` on the `Stream` directly so you don’t have to reach in and get the `Connection` object.

Let me know if you run into issues; I’m working on improving the HTTP.jl docs as we speak, so I’ll try to include something like this if it works for you.

---

<div class="post-metadata">

**Author:** ![JohnDuffy](https://avatars.discourse-cdn.com/v4/letter/j/f14d63/32.png) [@JohnDuffy](https://discourse.julialang.org/u/JohnDuffy)\
**Post date:** [June 17, 2022, 8:10am UTC](https://discourse.julialang.org/t/http-streaming-chunked-json/82797/3 "2022-06-17T08:10:54Z")

</div>

Thank you Jacob

I’ll test it and get back to you shortly.

John

---

<div class="post-metadata">

**Author:** ![quinnj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/quinnj/32/11_2.png) [@quinnj](https://discourse.julialang.org/u/quinnj)\
**Post date:** [June 17, 2022, 3:59pm UTC](https://discourse.julialang.org/t/http-streaming-chunked-json/82797/4 "2022-06-17T15:59:13Z")

</div>

I looked into things a bit more and I think it may not work quite right because the `HTTP.Stream` isn’t getting updated with the # of bytes being read, so it may not know when it’s “done” properly. That said, you may want to refactor the loop to check `eof(conn)` to wait until the underlying connection closes. I’ve opened an [issue](https://github.com/JuliaWeb/HTTP.jl/issues/852) to properly support this on the `Stream` object itself.

---

<div class="post-metadata">

**Author:** ![JohnDuffy](https://avatars.discourse-cdn.com/v4/letter/j/f14d63/32.png) [@JohnDuffy](https://discourse.julialang.org/u/JohnDuffy)\
**Post date:** [June 17, 2022, 7:35pm UTC](https://discourse.julialang.org/t/http-streaming-chunked-json/82797/5 "2022-06-17T19:35:08Z")

</div>

Thank you Jacob

The following appears to work.

```julia
HTTP.open("GET", url, headers) do io
  for line in eachline(io)
    println(line)
  end
end

```

Although, I do get a warning about reading single bytes being inefficient.

---

<div class="post-metadata">

**Author:** ![quinnj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/quinnj/32/11_2.png) [@quinnj](https://discourse.julialang.org/u/quinnj)\
**Post date:** [June 18, 2022, 2:38pm UTC](https://discourse.julialang.org/t/http-streaming-chunked-json/82797/6 "2022-06-18T14:38:47Z")

</div>

> [@quinnj](#):
>
> ```julia
> findnewline(bytes) = something(findfirst(==(UInt8('\n')), bytes), 0)
> 
> ```

Yeah, it’s not ideal and will be a little inefficient. I’ve implemented proper `readuntil` support on HTTP#master [here](https://github.com/JuliaWeb/HTTP.jl/blob/8a7ae8f86ee11240820eeedf978abf7b801059ef/src/Streams.jl#L307). Hoping to get a new release out with that functionality very soon.

---

<div class="post-metadata">

**Author:** ![quinnj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/quinnj/32/11_2.png) [@quinnj](https://discourse.julialang.org/u/quinnj)\
**Post date:** [June 21, 2022, 8:00pm UTC](https://discourse.julialang.org/t/http-streaming-chunked-json/82797/7 "2022-06-21T20:00:52Z")

</div>

In case others come across this thread, the new release has been made (1.0) for HTTP.jl that includes this functionality, so the right way to achieve the original request is to do:

```julia
findnewline(bytes) = something(findfirst(==(UInt8('\n')), bytes), 0)
HTTP.open("GET", url, headers) do io
    for bytes in readuntil(io, findnewline)
        x = JSON3.read(bytes)
        # do stuff with parsed JSON object `x`
    end
end

```

---

<div class="post-metadata">

**Author:** ![JohnDuffy](https://avatars.discourse-cdn.com/v4/letter/j/f14d63/32.png) [@JohnDuffy](https://discourse.julialang.org/u/JohnDuffy)\
**Post date:** [June 22, 2022, 7:54am UTC](https://discourse.julialang.org/t/http-streaming-chunked-json/82797/8 "2022-06-22T07:54:34Z")

</div>

Thank you Jacob.