# Easiest way to load a DataFrame from a compressed, newline delimited json file on the cloud?

**URL:** https://discourse.julialang.org/t/easiest-way-to-load-a-dataframe-from-a-compressed-newline-delimited-json-file-on-the-cloud/48678
**Category:** Data
**Tags:** dataframes
**Created:** [October 20, 2020, 11:53am UTC](https://discourse.julialang.org/t/easiest-way-to-load-a-dataframe-from-a-compressed-newline-delimited-json-file-on-the-cloud/48678 "2020-10-20T11:53:43Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![s-kap](https://avatars.discourse-cdn.com/v4/letter/s/ba8739/32.png) [@s-kap](https://discourse.julialang.org/u/s-kap)
#### Post date: [October 20, 2020, 11:53am UTC](https://discourse.julialang.org/t/easiest-way-to-load-a-dataframe-from-a-compressed-newline-delimited-json-file-on-the-cloud/48678/1 "2020-10-20T11:53:43Z")

</div>

I’m trying to convert a python notebook into Julia.

In python we have  
`pd.read_json("gs://bucket/file.json.gz", lines=True, compression="gzip")`

I don’t think there’s a way to do this.  
The approach I’ve taken is to first download the file and then

```julia
import Pkg; Pkg.add("DataFrames"); Pkg.add("JSON3"); Pkg.add("CodecZlib")
using DataFrames, CodecZlib, JSON3
path = "..../000000000000.json.gz"
df = open(path) do file
    DataFrame(JSON3.read.(eachline(GzipDecompressorStream(file))))
end

```

But I end up with some sort of key error, due to [this issue](https://github.com/JuliaData/Tables.jl/issues/207) that prevents you from populating a table if the first row has a column but a following row does not.

Do you have any other ideas for approaches?

---

<div class="post-metadata">

### Author: ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [October 20, 2020, 12:07pm UTC](https://discourse.julialang.org/t/easiest-way-to-load-a-dataframe-from-a-compressed-newline-delimited-json-file-on-the-cloud/48678/2 "2020-10-20T12:07:16Z")

</div>

You could just read the JSON and preprocess it before passing to DataFrame. If the file is huge, you could just implement a simple Tables.jl wrapper interface that adds that column to the rows iterator.

---

<div class="post-metadata">

### Author: ![quinnj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/quinnj/32/11_2.png) [@quinnj](https://discourse.julialang.org/u/quinnj)
#### Post date: [October 20, 2020, 1:08pm UTC](https://discourse.julialang.org/t/easiest-way-to-load-a-dataframe-from-a-compressed-newline-delimited-json-file-on-the-cloud/48678/3 "2020-10-20T13:08:38Z")

</div>

We’re actively thinking/working on a better overall Tables.jl solution here, but for now, DataFrames.jl has this functionality in [`push!`](https://juliadata.github.io/DataFrames.jl/stable/lib/functions/#Base.push!), so in your case, something like:

```julia
df = open(path) do file
    df = DataFrame()
    for line in eachline(GzipDecompressorStream(file))
        push!(df, JSON3.read(line); cols=:union)
    end
end

```