# JSON Performance Tests

**URL:** <https://discourse.julialang.org/t/json-performance-tests/17133>\
**Category:** Data\
**Created:** [November 4, 2018, 7:52am UTC](https://discourse.julialang.org/t/json-performance-tests/17133 "2018-11-04T07:52:26Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![samoconnor](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/samoconnor/32/1802_2.png) [@samoconnor](https://discourse.julialang.org/u/samoconnor)\
**Post date:** [November 4, 2018, 7:52am UTC](https://discourse.julialang.org/t/json-performance-tests/17133/1 "2018-11-04T07:52:26Z")

</div>

I noticed that @kristoffer.carlsson has been working on parsing performance in JSON.jl: [A few performance improvements by KristofferC · Pull Request #263 · JuliaIO/JSON.jl · GitHub](https://github.com/JuliaIO/JSON.jl/pull/263/files) … and it got me thinking.

When I first created [LazyJSON.jl](https://discourse.julialang.org/t/announce-a-different-way-to-read-json-data-lazyjson-jl/9046) I targeted Julia 0.7-dev at a time when there were a lot of string changes going on. At the time performance comparisons with JSON.jl and JSON2.jl would vary wildly between Julia 0.7-dev nightly builds as various deprecation penalties came and went. Now that Julia 1.0 is out and JSON.jl and JSON2.jl have been updated for Julia 1.0, it seems like a good time to compare performance again.

These tests compare LazyJSON.jl to @kristoffer.carlsson’s [`kc/opt` branch](https://github.com/JuliaIO/JSON.jl/pull/263) and JSON2.jl v0.2.3.

Test code is here: [LazyJSON.jl/test/benchmark.jl at master · JuliaCloud/LazyJSON.jl · GitHub](https://github.com/samoconnor/LazyJSON.jl/blob/master/test/benchmark.jl)

LazyJSON.jl seems to take about the same time as JSON.jl to do flat non-lazy parsing to Julia Dict/Array etc (sometimes a bit faster, sometimes a bit slower).  
In lazy mode LazyJSON.jl is orders of magnitude faster when only part of the input data is used.

@fengyang.wang what are your thoughts on either adding lazy parsing as an option in JSON.jl or replacing the current parser with a lazy parser? (One thing that would have to be done is to add a validation option to do strict JSON syntax checking for use cases where that is required. As it stands the Lazy parser will ignore some JSON syntax errors due to lazyness).

@quinnj Can you comment on the JSON2 results? I know that JSON2 is optimised for marshalling/unmarshalling, so perhaps my tests that result in JSON2 returning `NamedTuples` are a degenerate case. In test6 I tried to test JSON2’s direct-to-struct parsing in a way that seems to me to be “how JSON2 is intended to be used”, but it still seems a bit slow. Perhaps you can suggest a test case that would best demonstrate JSON2’s strengths.

@Nosferican I believe that you have been using LazyJSON.jl in `NCEI.jl`. Do you have any feedback from your use of the package?

## test1

Reads [`ec2-2016-11-15.normal.json`](https://github.com/samoconnor/LazyJSON.jl/blob/master/test/ec2-2016-11-15.normal.json) and extracts a single value:  
`operations.AcceptReservedInstancesExchangeQuote.input.shape`.  
This value is close to the start of the input data.

Variants:

- Lazy: LazyJSON.jl `AbstractDict` interface.
- Lazy (B): LazyJSON.jl `getproperty` interface.
- Lazy (C): LazyJSON.jl `lazy=false` (parse whole input to Dicts etc like JSON.jl does)
- JSON: JSON.jl `parse` interface.
- JSON2: JSON2.jl `read -> NamedTuple` interface.

```julia-auto
results = 5×6 DataFrame
│ Row │ Test │ Variant │ μs │ bytes │ poolalloc │ bigalloc │
├─────┼───────┼──────────┼────────┼───────────┼───────────┼──────────┤
│ 1 │ test1 │ Lazy │ 54 │ 5184 │ 269 │ 0 │
│ 2 │ test1 │ Lazy (B) │ 51 │ 5408 │ 277 │ 0 │
│ 3 │ test1 │ Lazy (C) │ 105628 │ 51409504 │ 980424 │ 300 │
│ 4 │ test1 │ JSON │ 103870 │ 50429936 │ 491747 │ 510 │
│ 5 │ test1 │ JSON2 │ 609448 │ 147471280 │ 4162257 │ 890 │

```

Note: LazyJSON.jl is similar to JSON.jl in speed and memory use in non-lazy mode.

## test2

Read `ec2-2016-11-15.normal.json` and extracts an array value:  
`shapes.scope.enum`  
This value is close to the end of the input data.

Variants:

- Lazy: LazyJSON.jl `AbstractDict` interface.
- Lazy (B): LazyJSON.jl `getproperty` interface.
- Lazy (C): LazyJSON.jl `lazy=false` (parse whole input to Dicts etc)
- JSON: JSON.jl `parse` interface.
- JSON2: JSON2.jl `read -> NamedTuple` interface.

```julia-auto
results = 5×6 DataFrame
│ Row │ Test │ Variant │ μs │ bytes │ poolalloc │ bigalloc │
├─────┼───────┼──────────┼────────┼───────────┼───────────┼──────────┤
│ 1 │ test2 │ Lazy │ 11035 │ 3296 │ 162 │ 0 │
│ 2 │ test2 │ Lazy (B) │ 11028 │ 3440 │ 168 │ 0 │
│ 3 │ test2 │ Lazy (C) │ 115045 │ 51409600 │ 980426 │ 300 │
│ 4 │ test2 │ JSON │ 91334 │ 50429936 │ 491747 │ 510 │
│ 5 │ test2 │ JSON2 │ 605269 │ 147471280 │ 4162257 │ 890 │

```

Note: It takes LazyJSON.jl a bit longer to access values near the end of  
the input.

## test3

Modifes `ec2-2016-11-15.normal.json` by replacing a value near the  
start of the file and two values near the end.

Variants:

- Lazy: LazyJSON.jl `getproperty` interface finds values and  
`LazyJSON.splice` modifies the JSON data in-place.
- JSON: JSON.jl `parse` to `Dict`, modify, then write new JSON text.
- JSON2: _Parses to immutable `NamedTuples`. Modificaiton not supported._

```julia-auto
results = 2×6 DataFrame
│ Row │ Test │ Variant │ μs │ bytes │ poolalloc │ bigalloc │
├─────┼───────┼─────────┼────────┼───────────┼───────────┼──────────┤
│ 1 │ test3 │ Lazy │ 235024 │ 880768 │ 33622 │ 0 │
│ 2 │ test3 │ JSON │ 671735 │ 126950528 │ 1407838 │ 1021 │

```

## test4

Reads a 1.2MB GeoJSON file an extracts a country name near the middle  
of the file.

Variants:

- Lazy: `LazyJSON.parse(j)["features"][15]["properties"]["formal_en"]`
- Lazy (B): `LazyJSON.parse(j; getproperty=true).features[15].properties.formal_en`
- Lazy (C): `LazyJSON.parse(j; lazy=false)["features"][15]["properties"]["formal_en"]`
- JSON: `JSON.parse(j)["features"][15]["properties"]["formal_en"]`
- JSON2: J`JSON2.read(j).features[15].properties.formal_en`

```julia-auto
results = 5×6 DataFrame
│ Row │ Test │ Variant │ μs │ bytes │ poolalloc │ bigalloc │
├─────┼───────┼──────────┼───────┼──────────┼───────────┼──────────┤
│ 1 │ test4 │ Lazy │ 310 │ 2288 │ 115 │ 0 │
│ 2 │ test4 │ Lazy (B) │ 312 │ 2432 │ 121 │ 0 │
│ 3 │ test4 │ Lazy (C) │ 40696 │ 13134624 │ 462247 │ 48 │
│ 4 │ test4 │ JSON │ 41609 │ 6336752 │ 135146 │ 100 │
│ 5 │ test4 │ JSON2 │ 84167 │ 22868160 │ 477011 │ 48 │

```

Note: LazyJSON.jl in non-lazy mode is a bit faster than JSON.jl for this  
input.

## test5

Reads a 1.2MB GeoJSON file and checks that the outline polygon for  
a single country is within an expected lat/lon range.

```julia-auto
r = r["features"][15]["geometry"]["coordinates"][6][1]
@assert r[1][1] == 134.41651451900023
for (x, y) in r
   @assert 134.2 < x < 134.5
   @assert 7.21 < y < 7.32
end

```

```julia-auto
results = 3×6 DataFrame
│ Row │ Test │ Variant │ μs │ bytes │ poolalloc │ bigalloc │
├─────┼───────┼─────────┼───────┼──────────┼───────────┼──────────┤
│ 1 │ test5 │ Lazy │ 399 │ 22992 │ 967 │ 0 │
│ 2 │ test5 │ JSON │ 40635 │ 6340592 │ 135296 │ 100 │
│ 3 │ test5 │ JSON2 │ 81213 │ 22872000 │ 477161 │ 48 │

```

## test6

Defines `struct Operation`, `struct IOType` and `struct HTTP` with  
fields that match the API operations data in `ec2-2016-11-15.normal.json`.  
It then does JSON2-style direct-to-struct parsing to read the JSON data  
into a Julia object `Dict{String,Operation}`  
(LazyJSON provides `@generated` `Base.convert` methods for this).

Variants:

- Lazy: LazyJSON.jl `AbstractDict` interface.  
`convert(Dict{String,Operation}, LazyJSON.parse(j))`
- JSON2: JSON2.jl `read -> NamedTuple` interface.  
`JSON2.read(j, Dict{String,Operation})`

```julia-auto
results = 2×6 DataFrame
│ Row │ Test │ Variant │ μs │ bytes │ poolalloc │ bigalloc │
├─────┼───────┼─────────┼───────┼─────────┼───────────┼──────────┤
│ 1 │ test6 │ Lazy │ 6866 │ 1125600 │ 39538 │ 16 │
│ 2 │ test6 │ JSON2 │ 13096 │ 3427888 │ 135789 │ 60 │

```

Note:  
For all of the above tests, the content of `ec2-2016-11-15.normal.json` has been  
duplicated 10 times into a top level JSON array “[, , , …]” this  
results in an overall input data size of ~10MB.

---

<div class="post-metadata">

**Author:** ![fengyang.wang](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fengyang.wang/32/104_2.png) [@fengyang.wang](https://discourse.julialang.org/u/fengyang.wang)\
**Post date:** [November 4, 2018, 4:45pm UTC](https://discourse.julialang.org/t/json-performance-tests/17133/2 "2018-11-04T16:45:41Z")

</div>

Thanks for running these tests. It is not surprising to me that the namedtuples incur a high cost, as JSON is often used for relatively “unstructured” data, so specializations need to be compiled for a large number of types. It is also not surprising that the lazy parser does much better on tests that do not require reading the entire input.

My feeling is that it is most sensible to have both a greedy and a lazy parser, as both have their use cases. The confusion for the end user could potentially be alleviated by abstracting the JSON (which could be renamed SimpleJSON) and LazyJSON packages, and providing a uniform interface with appropriate keyword arguments `lazy=false`, `strict=false`, etc., somewhat similar to how the DifferentialEquations.jl ecosystem is structured. I believe that this would be better than having distinct JSON and LazyJSON packages both provide duplicated functionality.

On a related note, there are some outstanding changes proposed by @ScottPJones that could potentially improve the performance of JSON.jl, but we would need some updating and review on those old pull requests.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [November 4, 2018, 5:13pm UTC](https://discourse.julialang.org/t/json-performance-tests/17133/3 "2018-11-04T17:13:31Z")

</div>

It would also be interesting to measure some other state of the art JSON parsers to give us an idea where we are right now compared to the rest of the world.

---

<div class="post-metadata">

**Author:** ![samoconnor](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/samoconnor/32/1802_2.png) [@samoconnor](https://discourse.julialang.org/u/samoconnor)\
**Post date:** [November 5, 2018, 2:26am UTC](https://discourse.julialang.org/t/json-performance-tests/17133/4 "2018-11-05T02:26:31Z")

</div>

> other state of the art JSON parsers

These would probably be a good starting point…

> **[GitHub - Tencent/rapidjson: A fast JSON parser/generator for C++ with both...](https://github.com/Tencent/rapidjson)**
>
> A fast JSON parser/generator for C++ with both SAX/DOM style API - GitHub - Tencent/rapidjson: A fast JSON parser/generator for C++ with both SAX/DOM style API

> **[GitHub - miloyip/nativejson-benchmark: C/C++ JSON parser/generator benchmark](https://github.com/miloyip/nativejson-benchmark)**
>
> C/C++ JSON parser/generator benchmark. Contribute to miloyip/nativejson-benchmark development by creating an account on GitHub.

---

<div class="post-metadata">

**Author:** ![samoconnor](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/samoconnor/32/1802_2.png) [@samoconnor](https://discourse.julialang.org/u/samoconnor)\
**Post date:** [November 5, 2018, 4:03am UTC](https://discourse.julialang.org/t/json-performance-tests/17133/5 "2018-11-05T04:03:06Z")

</div>

I’ve also just tried running some of the LazyJSON.jl correctness tests agains JSON.jl and JSON2.jl.  
For the most part, JSON.jl only fails tests that the NST JSONTestSuite classes as optional:

[https://github.com/JuliaIO/JSON.jl/issues/267](https://github.com/JuliaIO/JSON.jl/issues/267)

[https://github.com/JuliaIO/JSON.jl/issues/266](https://github.com/JuliaIO/JSON.jl/issues/266)

[https://github.com/quinnj/JSON2.jl/issues/19](https://github.com/quinnj/JSON2.jl/issues/19)

[https://github.com/quinnj/JSON2.jl/issues/18](https://github.com/quinnj/JSON2.jl/issues/18)

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [November 5, 2018, 9:26am UTC](https://discourse.julialang.org/t/json-performance-tests/17133/6 "2018-11-05T09:26:04Z")

</div>

Just to chime in: I find lazy reading of JSON data very useful, but like @fengyang.wang mentioned, it’s kind of hard to juggle slightly different APIs (that of JSON, JSON2, and LazyJSON) for slightly different tasks (explicitly/lazily reading/writing simple/complex types). So I just default to JSON and make do.

---

<div class="post-metadata">

**Author:** ![samoconnor](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/samoconnor/32/1802_2.png) [@samoconnor](https://discourse.julialang.org/u/samoconnor)\
**Post date:** [November 6, 2018, 2:40am UTC](https://discourse.julialang.org/t/json-performance-tests/17133/7 "2018-11-06T02:40:02Z")

</div>

@yakir12, @fengyang.wang,

I agree that the multiple packages are confusing. I’ve made some notes about a proposed unified API here:

> <https://github.com/JuliaIO/JSON.jl/issues/268>
>
> This issue follows \[Discourse comments about unifying the APIs of JSON.jl, LazyJ…SON.jl and JSON2.jl\](https://discourse.julialang.org/t/json-performance-tests/17133/6).
> 
> 
> \### 1. Define julia types for JSON Values
> 
> The current API uses \`Base.String\` to represent encoded JSON and \`Base.Dict\` etc to represent decoded JSON. The functions \`JSON.parse\` and \`JSON.json\` are used to convert between the two representations. This API restricts the implementation to be non-lazy. It also precludes the possibility of implementing short-cut methods for JSON derived values. 
> 
> \[JavaScript Object Notation\](https://www.json.org) defines 6 value types.
> Defining Julia types to represent these JSON value types will enable us to hide the implementation details (lazy vs eager parsing, encoded string representation vs decoded AST representation, etc).
> 
> Treating JSON values as first class types (rather than as something that must be converted to a \`Base\` type) allows dispatch on these types and transparent implementation of short-cut methods as needed for efficiency.
> 
> \`\`\`julia
> const JSON.Value = Union{
> JSON.Object,
> JSON.Array,
> JSON.Number,
> JSON.String,
> JSON.Bool,
> JSON.Null
> }
> \`\`\`
> 
> \### 2. Construct JSON Value objects from strings
> 
> The implementation might immediately parse encoding strings into Julia collection types, or it might parse to intermediate AST types, or it might just lazily wrap the encoded string. That implementation detail would be hidden from the user (unless there are compelling use cases where user-supplied implementation hints are a big performance win, e.g. a \`lazy=false\` option). It seems likely that a combination of sensible defaults and heuristics can achieve good performance in most cases without any need for the user to fiddle with options. 
> 
> \`\`\`
> """
> JSON.Value(::AbstractString)::JSON.Value
> 
> Create a JSON object from a JSON formatted string.
> """
> \`\`\`
> 
> \`\`\`julia
> julia\> x = JSON.Value("""{
> "object": {"field": "value"},
> "array": \[1,2,3\],
> "number": 43,
> "bool": true,
> "null": null
> }""")
> 
> julia\> x.object.field
> "value"
> 
> julia\> x.array\[1\]
> 1
> \`\`\`
> 
> \### 2. Construct JSON Value objects from julia objects.
> 
> The implementation might immediately encode the julia objects to a JSON string, or it might
> just wrap them and do nothing, or it might convert them to an intermediate representation.
> That detail is hidden from the user.
> 
> \`\`\`
> """
> JSON.Value(o)::JSON.Value
> 
> Create a JSON object from a Julia object.
> """
> \`\`\`
> 
> \`\`\`julia
> julia\> x = JSON.Value(Dict(
> "object" =\> Dict("field" =\> "value"),
> "array" =\> \[1,2,3\]
> "number" =\> 43,
> "bool" =\> true,
> "null" =\> nothing
> ))
> 
> julia\> x.object.field
> "value"
> 
> julia\> x.array\[1\]
> 1
> \`\`\`
> 
> \### 3. Use \`Base.string\` to produce JSON encoded strings.
> 
> Rather than using \`JSON.json\` to produce encoded strings, just use \`Base.string\`.
> Depending on the \`JSON.Value\` implementation, \`string\` might just return a preexisting encoded string, or it might have to produce an encoded string from an internal representation.
> 
> \`\`\`
> Base.string(o::JSON.Value)::AbstractString
> 
> JSON formatted string representation of a JSON object.
> \`\`\`
> 
> \`\`\`julia
> julia\> x = JSON.Value(Dict(
> "object" =\> Dict("field" =\> "value"),
> "array" =\> \[1,2,3\]
> "number" =\> 43,
> "bool" =\> true,
> "null" =\> nothing
> ))
> julia\> string(x)
> "{\\"object\\":{\\"field\\":\\"value\\"},\\"array\\":\[1,2,3\],\\"number\\":43,\\"bool\\":true,\\"null\\":null}"
> \`\`\`
> 
> \### 4. Use \`Base.convert\` to do direct-to-struct parsing.
> 
> e.g. like the direct-to-string parsing feature first implemented in JSON2: 
> \`\`\`julia
> julia\> struct MyType
> field
> end
> julia\> convert(MyType, JSON.Value("""{"field": "value"}""")
> MyType("value")
> \`\`\`
> 
> The \`convert\` methods would be \[\`@generated\`\](https://github.com/samoconnor/LazyJSON.jl/blob/master/src/AbstractDict.jl#L76-L89).
> 
> 
> \### 5. Use \`Base.convert\` in cases when specific \`Base\` types are needed.
> 
> Most of the time \`JSON.Value\` types that implement \`AbstractDict\`, \`AbstractArray\`, \`Base.Real\`, \`AbstractString\` etc are all that the user will need.
> 
> In cases where the user wants a specific type, they can use \`convert\`:
> 
> \`\`\`julia
> julia\> convert(Vector{Float64}, JSON.Value("\[0.25, 0.5, 1, 2, 4, 8\]"))
> 6-element Array{Float64,1}:
> 0.25
> 0.5
> 1.0
> 2.0
> 4.0
> 8.0
> \`\`\`
> 
> \### 6. Backwards compatibility
> 
> The existing API could be maintained as follows:
> 
> \`\`\`julia
> JSON.parse(x; kw...) = JSON.Value(x; kw...)
> JSON.json(x) = string(JSON.Value(x))
> \`\`\`
> 
> We could implement \`parse\` so that it produces lazy value objects by default and produces non-lazy values only when the \`dicttype=\` or \`inttype=\` options are supplied. Or we could disable laziness entirely for the \`JSON.parse\` interface and say "if you want the new lazy thing, use \`JSON.Value\`".
> 
> If returning an \`AbstractDict\` from \`parse\` instead of a \`Dict\` causes breakage (or performance regression) in existing code, then we should start out by returning \`Dict\`.
> 
> \### 7. Implementation
> 
> We can cherry-pick implementation detail from the various existing JSON codebases. i.e. Use the fast float decoder from over here, but use the more robust UTF-16 decoder from there.
> 
> It might turn out that using the lazy parser is just as fast as the non-lazy one for doing a full non-lazy parse. In that case we may only need one parser. Or, if there are cases where the existing non-lazy parser has big wins, we can keep both. The user should not be able to tell the difference.

> My feeling is that it is most sensible to have both a greedy and a lazy parser, as both have their use cases.

I am hopeful that with sufficient tweaking the lazy parser can perform as well as a greedy parser in most use cases. However, that can only be proven or disproven by testing. The first step is to decide on an API that can support both implementations. We might end up with a `lazy=true/false` flag, or we might end up with an implementation that combines laziness with caching to handle degenerate access patterns.

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [November 6, 2018, 6:09am UTC](https://discourse.julialang.org/t/json-performance-tests/17133/8 "2018-11-06T06:09:10Z")

</div>

I think

> [@samoconnor](#):
>
> `lazy=true/false` flag

is totally fine, at least to begin with. We could have a rule of thumb that if you need to access only 10% of the available keys then you should use `lazy=true` or some such.