# Initial version of my first package: A JSON Lines reader

**URL:** <https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552>\
**Category:** Package Announcements\
**Tags:** package\
**Created:** [August 8, 2020, 7:33am UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552 "2020-08-08T07:33:03Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![danielw2904](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/danielw2904/32/10890_2.png) [@danielw2904](https://discourse.julialang.org/u/danielw2904)\
**Post date:** [August 8, 2020, 7:33am UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/1 "2020-08-08T07:33:03Z")

</div>

Hi All!

I have just “finished” implementing an inital version of a [JSON Lines](http://jsonlines.org/) reader ([JSONLines.jl](https://github.com/danielw2904/JSONLines.jl)) and would like to ask for some feedback. It implements a specific style of JSON Lines file used in other implementations (e.g. [Clickhouse](https://clickhouse.tech/docs/en/interfaces/formats/#jsoneachrow)) which has a JSON object in each line of the file. The actual JSON parsing is done using the [LazyJSON.jl](https://github.com/JuliaCloud/LazyJSON.jl) library. The most interesting feature in my opinion is that an arbitrary number of lines can be skipped and only a subset of lines can be loaded, resulting in memory allocation only for the loaded lines. This is achieved using `mmap` and line parsing done with little allocation (skipping all lines results in 34 allocations with 3.28 KiB).  
Any feedback is welcome! Please let me know if you have ideas for additional functionality (the obvious next step is a writer function). My next step will be to implement a Table.jl compatible output instead of returning a DataFrame. Eventually I would like to register the package. Any tips in that direction are also welcome. Thanks!  
P.s. Please let me know if this is the wrong category for the post.

EDIT: Updates [see here](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/30)

EDIT2: New package for registration is [JSONLines.jl](https://github.com/danielw2904/JSONLines.jl)

---

<div class="post-metadata">

**Author:** ![anon67531922](https://avatars.discourse-cdn.com/v4/letter/a/48db29/32.png) [@anon67531922](https://discourse.julialang.org/u/anon67531922)\
**Post date:** [August 8, 2020, 7:49am UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/2 "2020-08-08T07:49:38Z")

</div>

Very cool 👍😊

I had never heard of JSON Lines, but it makes perfect sense 👍

Some interesting names using it: [JSON Lines Examples](http://jsonlines.org/on_the_web/)

---

<div class="post-metadata">

**Author:** ![dilumaluthge](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dilumaluthge/32/29283_2.png) [@dilumaluthge](https://discourse.julialang.org/u/dilumaluthge)\
**Post date:** [August 8, 2020, 8:33am UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/3 "2020-08-08T08:33:32Z")

</div>

Looks very interesting!

Would you be open to slightly renaming the package? I was thinking that a name like `JSONLines.jl` might be more descriptive and recognizable. What do you think?

---

<div class="post-metadata">

**Author:** ![danielw2904](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/danielw2904/32/10890_2.png) [@danielw2904](https://discourse.julialang.org/u/danielw2904)\
**Post date:** [August 8, 2020, 8:33am UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/4 "2020-08-08T08:33:52Z")

</div>

Thank you! I really like the format especially for large files.

---

<div class="post-metadata">

**Author:** ![dilumaluthge](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dilumaluthge/32/29283_2.png) [@dilumaluthge](https://discourse.julialang.org/u/dilumaluthge)\
**Post date:** [August 8, 2020, 8:34am UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/5 "2020-08-08T08:34:13Z")

</div>

> [@danielw2904](#):
>
> Please let me know if you have ideas for additional functionality (the obvious next step is a writer function).

It would be great to have write functionality!

---

<div class="post-metadata">

**Author:** ![danielw2904](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/danielw2904/32/10890_2.png) [@danielw2904](https://discourse.julialang.org/u/danielw2904)\
**Post date:** [August 8, 2020, 8:35am UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/6 "2020-08-08T08:35:16Z")

</div>

You are completely right. I’ll rename it. 😃

---

<div class="post-metadata">

**Author:** ![Nosferican](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nosferican/32/9275_2.png) [@Nosferican](https://discourse.julialang.org/u/Nosferican)\
**Post date:** [August 8, 2020, 8:59am UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/7 "2020-08-08T08:59:27Z")

</div>

JSONL is the file extension though. I use that format pretty much always… You shouldn’t work with a JSON Vector of 13M JSON objects on memory so JSONL is probably what I work with the most. However, I usually just read the lines I want and then parse them through JSON3.

---

<div class="post-metadata">

**Author:** ![dilumaluthge](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dilumaluthge/32/29283_2.png) [@dilumaluthge](https://discourse.julialang.org/u/dilumaluthge)\
**Post date:** [August 8, 2020, 9:03am UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/8 "2020-08-08T09:03:16Z")

</div>

Sure, but:

1. JSON Lines is the name of the format in the documentation ([http://jsonlines.org/](http://jsonlines.org/))
2. JSONLines.jl is very easy to distinguish visually. JSONL.jl looks very similar to JSON.jl.
3. JSONLines.jl is a much more descriptive package name. As a general rule, we try to encourage package authors to choose more descriptive package names, instead of package names that are purely initialisms.

---

<div class="post-metadata">

**Author:** ![Nosferican](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nosferican/32/9275_2.png) [@Nosferican](https://discourse.julialang.org/u/Nosferican)\
**Post date:** [August 8, 2020, 10:12am UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/9 "2020-08-08T10:12:53Z")

</div>

I am not disagreeing with the assessment but since it is a pretty widely used file extension I don’t think it needs it just like JSON.jl isn’t JavaScriptObjectNotation.jl which is a more descriptive name rather than purely initialisms. We already have a few packages (JSON, JSON2, JSON3) which visually distinguishable enough. Another consideration is consistency which unless there is a strong reason for not following it should abide.

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [August 8, 2020, 1:58pm UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/10 "2020-08-08T13:58:42Z")

</div>

I was not familiar with the JSON Lines file format, so for me it was easy to miss that extra “L” in JSONL.jl.

---

<div class="post-metadata">

**Author:** ![anon37204545](https://avatars.discourse-cdn.com/v4/letter/a/439d5e/32.png) [@anon37204545](https://discourse.julialang.org/u/anon37204545)\
**Post date:** [August 8, 2020, 2:25pm UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/11 "2020-08-08T14:25:29Z")

</div>

This seems more appropriate for “Package announcements” category than for “First steps”, so I moved it there. That’s where people discuss new packages.

I’d also go with JSONLines.jl instead of JSONL.jl. Since the official name is also JSON Lines, there is no reason not to add the extra clarity. 😉

---

<div class="post-metadata">

**Author:** ![danielw2904](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/danielw2904/32/10890_2.png) [@danielw2904](https://discourse.julialang.org/u/danielw2904)\
**Post date:** [August 8, 2020, 2:27pm UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/12 "2020-08-08T14:27:29Z")

</div>

Thank you! It seems that “JSONLines.jl” is is clear to everyone while “JSONL.jl” is too close to “JSON.jl”.

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [August 8, 2020, 4:19pm UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/13 "2020-08-08T16:19:21Z")

</div>

There’s also newline delimited JSON, which uses a completely different extension (.ndjson). In light of that, JSONLines sounds general enough to accommodate both.

---

<div class="post-metadata">

**Author:** ![anon67531922](https://avatars.discourse-cdn.com/v4/letter/a/48db29/32.png) [@anon67531922](https://discourse.julialang.org/u/anon67531922)\
**Post date:** [August 8, 2020, 7:25pm UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/14 "2020-08-08T19:25:02Z")

</div>

Is each row of a JSONL file supposed to have the same structure?

Or can you have different object shapes on different rows?

If it is the same object in each row, it would be great to implement a Tables.jl interface 🙂

---

<div class="post-metadata">

**Author:** ![anon67531922](https://avatars.discourse-cdn.com/v4/letter/a/48db29/32.png) [@anon67531922](https://discourse.julialang.org/u/anon67531922)\
**Post date:** [August 8, 2020, 7:48pm UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/15 "2020-08-08T19:48:22Z")

</div>

On naming, ultimately your package = your decision 🙂

Suggestion…

I see you return a DataFrame. I don’t use DataFrames so that seems like an unnecessary imposition.

You might consider introducing something like

```julia
struct JSONL <: AbstractVector{Object}
    objects::Vector{Object}
end

```

and implement a Tables.jl interface so if we want to drop it into a DataFrame, we can just do

```julia
DataFrame(jsonl::JSONL)

```

If we want a StructArray (which is what I use), we’d just do

```julia
StructArray(jsonl)

```

Another suggestion…

Consider using JSON3 and allow the user to specify the struct type of each row so you can immediately construct a vector of structs. I think this would also be faster.

Just some ideas thinking aloud. I see so much potential for this 🙂

---

<div class="post-metadata">

**Author:** ![danielw2904](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/danielw2904/32/10890_2.png) [@danielw2904](https://discourse.julialang.org/u/danielw2904)\
**Post date:** [August 8, 2020, 7:55pm UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/16 "2020-08-08T19:55:10Z")

</div>

I’m working on the `Tables.jl` interface right now but am struggling a bit with it. Will ask a separate question once I am fully stuck. JSON3 looks good will try that next. Basically I would read each line and let users pass an optional struct for the types correct?  
Thank you very much for the suggestions!

---

<div class="post-metadata">

**Author:** ![danielw2904](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/danielw2904/32/10890_2.png) [@danielw2904](https://discourse.julialang.org/u/danielw2904)\
**Post date:** [August 8, 2020, 10:56pm UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/17 "2020-08-08T22:56:25Z")

</div>

See [topic here](https://discourse.julialang.org/t/struggling-to-implement-tables-jl-interface-again/44587). Unfortunately, I cannot get `StructArray` to work with it.

---

<div class="post-metadata">

**Author:** ![Nosferican](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nosferican/32/9275_2.png) [@Nosferican](https://discourse.julialang.org/u/Nosferican)\
**Post date:** [August 9, 2020, 1:07am UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/18 "2020-08-09T01:07:33Z")

</div>

Each line doesn’t have to have the same structure (no schema à la NoSQL).  
I usually do

```julia
using JSON3: JSON3, StructType, Mutable
mutable struct Node
    nodeID::String
    nodeUserID::String
    parentID::String
    nodeTime::String
    informationID::String
    Status() = new()
end
StructType(::Type{Node}) = Mutable()
lns = readlines("file.jsonl")
data = DataFrame(JSON3.read(ln, Node) for ln in lns)

```

for writing

```julia
io = open(touch("file.jsonl"), write = true)
for node in nodes
    JSON3.write(io, node)
    write(io, '\n')
end
close(io)

```

---

<div class="post-metadata">

**Author:** ![danielw2904](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/danielw2904/32/10890_2.png) [@danielw2904](https://discourse.julialang.org/u/danielw2904)\
**Post date:** [August 9, 2020, 9:38am UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/19 "2020-08-09T09:38:13Z")

</div>

Thank you for the suggestion. My plan would be that users can supply `Node`. I guess it is not possible in this case to implement `Tables.jl`?

---

<div class="post-metadata">

**Author:** ![danielw2904](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/danielw2904/32/10890_2.png) [@danielw2904](https://discourse.julialang.org/u/danielw2904)\
**Post date:** [August 9, 2020, 10:15am UTC](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552/20 "2020-08-09T10:15:24Z")

</div>

Could you give a small example of what `file.jsonl` could look like so I can play around with it and know exactly what you need?

[Next page](https://discourse.julialang.org/t/initial-version-of-my-first-package-a-json-lines-reader/44552.md?page=2)