# Initial version of my first package: A JSON Lines reader

**URL:** <https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552>\
**Category:** Package Announcements\
**Tags:** package\
**Created:** [August 8, 2020, 7:33am UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552 "2020-08-08T07:33:03Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![ImreSamu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/imresamu/32/20677_2.png) [@ImreSamu](https://discourse.julialang.org/u/ImreSamu)\
**Post date:** [August 9, 2020, 11:18am UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/21 "2020-08-09T11:18:15Z")

</div>

> [@danielw2904](#):
>
> Please let me know if you have ideas for additional functionality (

IMHO:  
with a little change it can be extend to process [wikidata json dumps](https://www.wikidata.org/wiki/Wikidata:Database_download#JSON_dumps_(recommended))

- Wikidata Json Dump is [very huge](https://dumps.wikimedia.org/wikidatawiki/entities/) , compressed gz \> 80GB:
  - `latest-all.json.bz2 05-Aug-2020 11:19 58459544787`
  - `latest-all.json.gz 05-Aug-2020 05:40 87793345321`

- it is **“a single JSON array”**
- the structure is similar to JSONLines - with this extra parameters :
  - extra first line `[`
  - extra last line `]`
  - extra comma as a line separator: `",\n"` or `",\r\n"`

```julia
[
{"type":"item","id":"Q31","labels": .... },
{"type":"item","id":"Q8","labels": .... },
...
]

```

- test command check: `zcat latest-all.json.gz | head -n3`

because - it is large compressed file ( \> 82G ) the keys:

- compressed file reading
- thread support ( channels ? ) for filtering … parallel processing
- writing the filtered result to similar JSON Dump file
- speed / multi core support …

My use case:

- pre-processing wikidata JSON dump
- filtering geodata related items
- and write a smaller JSON dump OR load to PostGIS database.

IMHO: It is not critical - because I have a Golang script … but for the future it can be an interesting use case for Julia ( and can be a good benchmark ! )

---

<div class="post-metadata">

**Author:** ![Nosferican](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nosferican/32/9275_2.png) [@Nosferican](https://discourse.julialang.org/u/Nosferican)\
**Post date:** [August 9, 2020, 2:32pm UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/22 "2020-08-09T14:32:04Z")

</div>

See for example this [test dataset](https://raw.githubusercontent.com/pnnl/socialsim_package/master/examples/data/test_dataset.txt). I do have pretty big files but most are confidential.

---

<div class="post-metadata">

**Author:** ![tlienart](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tlienart/32/7640_2.png) [@tlienart](https://discourse.julialang.org/u/tlienart)\
**Post date:** [August 9, 2020, 3:08pm UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/23 "2020-08-09T15:08:07Z")

</div>

Maybe out of topic but has anyone tried to port simdjson to Julia?

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [August 9, 2020, 3:11pm UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/24 "2020-08-09T15:11:49Z")

</div>

I think that it might be better to have this work exactly to the spec for JSON Lines. This would be more in the spirit of a typical Julia package – it does exactly what the name implies.

IMHO If people need to read JSON files that are similar to, but not the same as, JSON Lines, then they can hack it together using your package and other JSON readers. Perhaps if you have unexported functions that might help in these regards you can document them.

---

<div class="post-metadata">

**Author:** ![danielw2904](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/danielw2904/32/10890_2.png) [@danielw2904](https://discourse.julialang.org/u/danielw2904)\
**Post date:** [August 9, 2020, 4:28pm UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/25 "2020-08-09T16:28:12Z")

</div>

Makes sense. I guess I can internally expose the character expected before \n so that it can be changed by others.

---

<div class="post-metadata">

**Author:** ![danielw2904](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/danielw2904/32/10890_2.png) [@danielw2904](https://discourse.julialang.org/u/danielw2904)\
**Post date:** [August 9, 2020, 6:18pm UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/26 "2020-08-09T18:18:06Z")

</div>

Thank you for taking the time to put that together. I’ll try it out.

---

<div class="post-metadata">

**Author:** ![danielw2904](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/danielw2904/32/10890_2.png) [@danielw2904](https://discourse.julialang.org/u/danielw2904)\
**Post date:** [August 10, 2020, 2:23pm UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/27 "2020-08-10T14:23:51Z")

</div>

I tried running this code with a slight modification but it does not seem to work as a DataFrames input. Using the file you provided as test input:

```julia
using JSON3: JSON3, StructType, Mutable
using DataFrames
mutable struct Node
    nodeID::String
    nodeUserID::String
    parentID::String
    nodeTime::String
    informationID::String
    Node() = new()
end
StructType(::Type{Node}) = Mutable()
lns = readlines("test.jsonl")
data = [JSON3.read(ln, Node) for ln in lns]
DataFrame(data)

```

```julia
julia> data = [JSON3.read(ln, Node) for ln in lns]
5001-element Array{Node,1}:
 Node(#undef, #undef, #undef, #undef, #undef)
 Node("phkmqmzpbv", "ufmgvgoure", "sgdibnesgi", "2019-03-05 13:10:51", "tyaekgxmsr")
 Node("ugtkjodxer", "fhffxqbsoa", #undef, "2019-03-04 14:09:58", "bolwagblhx")
 Node("alwwtfyunw", "qccqocykfm", "orjhhcvomh", "2019-03-05 01:37:58", "cagyezgppo")
 ⋮
 Node("zuwjjgexbl", "opmkvyipxm", "sxavzrxldl", "2019-03-04 19:09:19", "tswwdiktno")
 Node("ubaomyspwd", "jdoescksnv", #undef, "2019-03-05 15:18:43", "dbermznthm")
 Node("optbrwcfli", "trrheeevlx", "ooxfkaspca", "2019-03-04 21:51:06", "vhdzurjxro")
 Node("mlcxmfplei", "ojshhztncj", "ygotmuetnj", "2019-03-04 18:14:03", "rukafhqowm")

julia> DataFrame(data)
ERROR: UndefRefError: access to undefined reference

```

Any idea what I am doing wrong? (the modification is `Status = new()` =\> `Node = new()` otherwise I get a `Node is not callable` error)

EDIT: The problem are the `#undef` values. I can create e.g. a `Tables.columntable` out of rows 2 and 4  
EDIT2: The following redefinition does work as a Tables.jl input as it avoids undef in construction

```julia
using StructTypes
mutable struct Node
    nodeID::Union{String, Missing}
    nodeUserID::Union{String, Missing}
    parentID::Union{String, Missing}
    nodeTime::Union{String, Missing}
    informationID::Union{String, Missing}
    Node() = new(missing, missing, missing, missing, missing)
end
StructTypes.StructType(::Type{Node}) = StructTypes.Mutable()

```

---

<div class="post-metadata">

**Author:** ![Nosferican](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nosferican/32/9275_2.png) [@Nosferican](https://discourse.julialang.org/u/Nosferican)\
**Post date:** [August 10, 2020, 3:11pm UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/28 "2020-08-10T15:11:13Z")

</div>

One thing to remember is that JSON is à la NoSQL in terms of not every record has the same schema. For example, in the example JSONL I shared the first line is a header which has a different schema from lines 2:end. I don’t remember if that particular file consistently uses `null` for missing vs omitted.

You can also check this other [example](https://github.com/Nosferican/AEEIncidentes/actions/runs/202645357) (will be available for 90 days, just need to download and inflate, 78 lines where each is a JSON Vector).

---

<div class="post-metadata">

**Author:** ![danielw2904](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/danielw2904/32/10890_2.png) [@danielw2904](https://discourse.julialang.org/u/danielw2904)\
**Post date:** [August 10, 2020, 4:34pm UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/29 "2020-08-10T16:34:15Z")

</div>

I guess in case the lines do not have the same schema the `Tables.jl` interface cannot be implemented but it should be possible now to load such a file. The problem with structtypes is that `#undef` is returned for missing values.

---

<div class="post-metadata">

**Author:** ![danielw2904](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/danielw2904/32/10890_2.png) [@danielw2904](https://discourse.julialang.org/u/danielw2904)\
**Post date:** [August 10, 2020, 4:40pm UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/30 "2020-08-10T16:40:00Z")

</div>

So a bunch of updates:

- Using `JSON3.jl` for parsing each row
- Returning the vector of `JSON3.Object`s since that already works with the `Tables.jl` interface if the schema allows it. This also removed the `DataFrames` dependency.
- New keyword argument: `structtype` allows users to pass a `StructTypes.jl` struct to the `JSON3.read` function for each row (could be that the result still works for `Tables.jl` but not necessarily due to `undef` being returned of a value is not available in a row =\> any suggestions?)

Edit:  
Credit for the Tables.jl insight goes to @piever!

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [August 10, 2020, 4:41pm UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/31 "2020-08-10T16:41:54Z")

</div>

Julia natively supports `missing`, so why not fill in those spots with it?

---

<div class="post-metadata">

**Author:** ![danielw2904](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/danielw2904/32/10890_2.png) [@danielw2904](https://discourse.julialang.org/u/danielw2904)\
**Post date:** [August 10, 2020, 5:00pm UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/32 "2020-08-10T17:00:42Z")

</div>

How can I replace all `undef`s? I could not find any documentation on this. It is already possible for users to pass a `struct` that is initialized to missing as a workaround.

---

<div class="post-metadata">

**Author:** ![dilumaluthge](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dilumaluthge/32/29283_2.png) [@dilumaluthge](https://discourse.julialang.org/u/dilumaluthge)\
**Post date:** [August 10, 2020, 5:33pm UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/33 "2020-08-10T17:33:22Z")

</div>

I guess you can do use the `isassigned` function to check if a field is undef or not.

EDIT: I’m not sure if that works for structs. It works for arrays.

---

<div class="post-metadata">

**Author:** ![danielw2904](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/danielw2904/32/10890_2.png) [@danielw2904](https://discourse.julialang.org/u/danielw2904)\
**Post date:** [August 10, 2020, 7:46pm UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/34 "2020-08-10T19:46:18Z")

</div>

I’ll try it out

---

<div class="post-metadata">

**Author:** ![danielw2904](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/danielw2904/32/10890_2.png) [@danielw2904](https://discourse.julialang.org/u/danielw2904)\
**Post date:** [August 11, 2020, 6:25am UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/35 "2020-08-11T06:25:31Z")

</div>

Unfortunately it seems like `isassigned` only works for arrays. I get errors both for the assigned values and the unassigned ones with

```julia
[isassigned(getproperty(data[2], name)) for name in propertynames(data[2])]

```

---

<div class="post-metadata">

**Author:** ![thofma](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/thofma/32/1691_2.png) [@thofma](https://discourse.julialang.org/u/thofma)\
**Post date:** [August 11, 2020, 8:30am UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/36 "2020-08-11T08:30:45Z")

</div>

Try `isdefined`. More precisely, try `isdefined(data[2], name)`.

---

<div class="post-metadata">

**Author:** ![danielw2904](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/danielw2904/32/10890_2.png) [@danielw2904](https://discourse.julialang.org/u/danielw2904)\
**Post date:** [August 11, 2020, 9:46am UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/37 "2020-08-11T09:46:07Z")

</div>

Thank you! This works but I cannot replace with missing if the type is not `Union{TYPE, Missing}` even though the struct is mutable.

---

<div class="post-metadata">

**Author:** ![danielw2904](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/danielw2904/32/10890_2.png) [@danielw2904](https://discourse.julialang.org/u/danielw2904)\
**Post date:** [August 12, 2020, 11:18pm UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/38 "2020-08-12T23:18:39Z")

</div>

Quick update:  
A basic writer function is implemented. As always please let me know if you need more features or something is not working right. I have not yet implemented tests for the writer but it seems to work based on quick tests I did on my machine.

Next I want to implement a chunk reader that lets the user iterate over the file only keeping one chunk in memory at a time. I think this would be useful for eg filtering the data.

---

<div class="post-metadata">

**Author:** ![danielw2904](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/danielw2904/32/10890_2.png) [@danielw2904](https://discourse.julialang.org/u/danielw2904)\
**Post date:** [August 13, 2020, 12:21pm UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/39 "2020-08-13T12:21:04Z")

</div>

I have renamed the package to [JSONLines.jl and registered it](https://github.com/JuliaRegistries/General/pull/19428). I guess it will take 3 days for it to be available.  
🥳 🥳🥳

---

<div class="post-metadata">

**Author:** ![widged](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/widged/32/18739_2.png) [@widged](https://discourse.julialang.org/u/widged)\
**Post date:** [November 15, 2020, 12:09pm UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/40 "2020-11-15T12:09:33Z")

</div>

I have been using the jsonl (json lines) format for a few years, thanks to the excellent [https://github.com/louischatriot/nedb](https://github.com/louischatriot/nedb) (a javascript json lines datastore). I would love to have something equivalent (to nedb) in Julia… happy to help make it happen (but be warned that my familiarity with Julia is still limited)

[Previous page](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552.md?page=1)

[Next page](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552.md?page=3)