Trouble with JSON3 Generated Types

General Usage
1

I’m trying to generate types from an input JSON string using JSON3.jl. It generally works very well, but I’m hitting a weird snag that I can’t seem to debug.

First, here is a MWE of my code:

using JSON3

function parse_json(json_string)
    JSON3.@generatetypes json_string
    parsed = JSON3.read(json_string, JSONTypes.Root)
    return parsed
end

const body = """{"file_id": 302,
"schema": "In-Network",
"file_s3_key": "www.centene.com/2022-06-29_centene-management-company-llc_fidelis-es_in-network.json",
"data": {"negotiation_arrangement": "ffs", "name": "INJ CEFTRIAXONE SODM PER 250", "billing_code_type": "HCPCS", "billing_code_type_version": "2022", "billing_code": "J0696", "description": "INJECTION, CEFTRIAXONE SODIUM, PER 250 MG", "negotiated_rates": [{"negotiated_prices": [{"negotiated_type": "negotiated", "negotiated_rate": "0.36", "expiration_date": "9999-12-31", "service_code": ["11"], "billing_class": "professional"}], "provider_groups": [{"npi": [1770736076], "tin": {"type": "ein", "value": "852123937"}}, {"npi": [1639129026], "tin": {"type": "ein", "value": "823602522"}}, {"npi": [1891942645], "tin": {"type": "ein", "value": "208305340"}}, {"npi": [1104837806], "tin": {"type": "ein", "value": "842533193"}}, {"npi": [1649543653], "tin": {"type": "ein", "value": "842243679"}}, {"npi": [1609223429], "tin": {"type": "ein", "value": "852697229"}}, {"npi": [1235394412], "tin": {"type": "ein", "value": "214699633"}}, {"npi": [1093940843], "tin": {"type": "ein", "value": "834127306"}}, {"npi": [1285024034], "tin": {"type": "ein", "value": "842426133"}}, {"npi": [1932202074, 1043579527], "tin": {"type": "ein", "value": "820836819"}}, {"npi": [1962438598], "tin": {"type": "ein", "value": "812627417"}}, {"npi": [1083658918], "tin": {"type": "ein", "value": "851126175"}}, {"npi": [1194212522], "tin": {"type": "ein", "value": "854140142"}}, {"npi": [1831269398], "tin": {"type": "ein", "value": "843089453"}}, {"npi": [1265694020, 1568783652], "tin": {"type": "ein", "value": "841750751"}}, {"npi": [1699784611], "tin": {"type": "ein", "value": "75484906"}}, {"npi": [1275560187], "tin": {"type": "ein", "value": "871191244"}}, {"npi": [1164785390], "tin": {"type": "ein", "value": "850520136"}}, {"npi": [1649221417], "tin": {"type": "ein", "value": "133653838"}}, {"npi": [1871530873], "tin": {"type": "ein", "value": "201442983"}}, {"npi": [1003896929], "tin": {"type": "ein", "value": "871038946"}}, {"npi": [1033306204], "tin": {"type": "ein", "value": "821581949"}}, {"npi": [1285065557], "tin": {"type": "ein", "value": "851409030"}}, {"npi": [1639326424], "tin": {"type": "ein", "value": "814496947"}}, {"npi": [1578820379], "tin": {"type": "ein", "value": "832007516"}}, {"npi": [1902005622], "tin": {"type": "ein", "value": "463331735"}}, {"npi": [1730468380, 1992051635], "tin": {"type": "ein", "value": "823208491"}}, {"npi": [1790813624, 1700377934, 1891920682], "tin": {"type": "ein", "value": "852802458"}}, {"npi": [1396858205], "tin": {"type": "ein", "value": "112610239"}}, {"npi": [1053514190, 1790727584], "tin": {"type": "ein", "value": "113378755"}}, {"npi": [1396792867], "tin": {"type": "ein", "value": "474687663"}}, {"npi": [1801034277], "tin": {"type": "ein", "value": "452543425"}}, {"npi": [1285047092], "tin": {"type": "ein", "value": "852559579"}}, {"npi": [1427123090, 1770967754, 1801017470, 1568750057, 1861550782, 1861590648, 1760890552, 1285794925, 1013361864, 1982775649, 1851755631, 1285707091, 1114011285, 1740698612, 1336472380, 1033277827, 1518058957, 1386175560, 1841297223, 1093887465, 1720159163, 1376608208, 1699843532, 1528126315], "tin": {"type": "ein", "value": "61578286"}}, {"npi": [1336111624], "tin": {"type": "ein", "value": "75801280"}}, {"npi": [1902190432], "tin": {"type": "ein", "value": "833912827"}}, {"npi": [1942375258], "tin": {"type": "ein", "value": "852020054"}}, {"npi": [1447412481], "tin": {"type": "ein", "value": "461679141"}}]}, {"negotiated_prices": [{"negotiated_type": "negotiated", "negotiated_rate": "0.43", "expiration_date": "9999-12-31", "service_code": ["11"], "billing_class": "professional"}], "provider_groups": [{"npi": [1679642565], "tin": {"type": "ein", "value": "260614979"}}]}]}}
"""

parse_json(body)

If I run this from the top, it throws MethodError: no method matching Main.JSONTypes.NegotiatedPrice(::String, ::String, ::String, ::Vector{String}, ::String) at the line parsed = JSON3.read(json_string, JSONTypes.Root).

But if I rerun the definition of parse_json, by that I mean run the lines from function ... end, and then run parse_json(body) again, it works perfectly.

I think it might have something to do with the generate macro? But I don’t know how to make sense of it.

2

As far as I understand (@mcmcgrath13 can probably chime in here), the JSON3.@generatetypes macro is meant to be called from the top-level with example json to generate a module/file that can be used/included. Then, you can use the generated code to parse things. So yes, @generatetypes needs to be run previously to calling JSON3.read with the generated types.

3

This usage of type generation is not exactly the intended use case, but should work. What’s interesting is if we do what the macro is doing manually, it does work

function parse_json(json_string)
    eval(JSON3.generatetypes(json_string, :JSONTypes))
    parsed = JSON3.read(json_string, JSONTypes.Root)
    return parsed
end

I wonder if this is an issue with world ages or scoping?

1 Like
4

Marked as solution without testing, rookie mistake. I’m still getting the same error even using the eval syntax, so I’m wondering if you could potentially lay out the intended use case a little more.

If I have an example json, I should create a separate file called MakeJsonTypes.jl and in it have:

using JSON3
const json_example = """ {.... json here }"""
JSON3.@generatetypes json_example

And then in the script where I want to use the generated types to read JSON, I should include("MakeJsonTypes.jl"). Is that right?

Edit: that does seem to work. Also, just a note on the behavior in case someone else happens upon this: It seems that once a function is running, it doesn’t check for updates to JSONTypes.Root until complete. Even if I add a long sleep to the function, it never finds out that the type information has been updated until we return to the REPL’s global scope.

Edit2: It is a world age problem! This solved it!

function parse_json_w_types(json_str)
    json_types = JSON3.@generatetypes json_str
    parsed = Base.invokelatest(JSON3.read, json_str, json_types.Root)
    return parsed
end
1 Like
5

Glad you solved it!

In terms of intended use case, I’d say the best use case for generating types is when you need to process a bunch of JSON and have some samples that you want to determine a type from and can save off then use for the rest of the input. Generating the type requires reading the JSON, so is not cheap/free, so if we generate a type for a JSON then read into that type, we’ve read the JSON twice. The type generated also might not be perfect for your use case (or the reality of the full dataset), so it might be more robust to write the types to file using JSON3.writetypes and then inspect/edit that file and use it where needed.

6

What I’m trying to accomplish is taking a very, very large JSON and flattening it into a dataframe with more complex logic than jsontables supports (if I get something robust enough, I’d love to contribute, but I’m not there yet).

I want to go through the whole string first to check for missingness, types, etc. so that I have that info while building the dataframe. The generated types do all that analysis already, so I’m sure there is a more efficient way, but it’s helping me brute force the problem at the moment.

I’m definitely open to suggestions if you have any!