A way to create large dictionary? "ERROR: LoadError: syntax: expression too large"

New to Julia
,
1

Hi there,

What is the best way to create a large dictionary?
This expression works for smaller Dicts:
pDemand = Dict(
(:DEM_ELC_DH, :ELC, :AL, :2018, :d001_h00) => 20.4280246,
(:DEM_ELC_DH, :ELC, :AR, :2018, :d001_h00) => 9.4801576,
(:DEM_ELC_DH, :ELC, :AZ, :2018, :d001_h00) => 12.5974967,
(:DEM_ELC_DH, :ELC, :CA, :2018, :d001_h00) => 22.195113,
(:DEM_ELC_DH, :ELC, :CO, :2018, :d001_h00) => 7.1842716,
…
)
But if there are 400000+ rows, the error appears:
“ERROR: LoadError: syntax: expression too large”
(which is a bit surprising, I haven’t experienced any limits on expressions length in other languages - like GAMS or Python/Pyomo)

What would be a better way to create a Dictionary in Julia? We use the format as data input & mapping in JuMP.

Here is a big Dict to experiment with: long_dict.jl

Thanks!

2

I think the problem is that no one assumed that people would be writing megabyte large expressions in Julia. Would it make sense to store this data as a CSV or some other type of file and then load that into a dictionary?

1 Like
3

I believe the issue is that you are making the whole dictionary as a comprehension. Did you try something like,

pDemand = Dict( (:DEM_ELC_DH, :ELC, :AL, :2018, :d001_h00) => 20.4280246 )
pDemand[(:DEM_ELC_DH, :ELC, :AR, :2018, :d001_h00)] = 9.4801576
pDemand[(:DEM_ELC_DH, :ELC, :AZ, :2018, :d001_h00)] = 12.5974967
…
2 Likes
4

This seems to work – will test it for #37256. Thanks!

5

Agree! Though if this is an unnecessary restriction, I would vote to remove it.

6

The strategy suggested by ccoffrin works well for a particular parameter/dictionary. But now there is the same error:

ERROR: LoadError: LoadError: ReadOnlyMemoryError()

while loading a long file with the script, following Julia crush. Is it an issue with long .jl files in general? Any ways to allocate more memory for the operation?

Here is the data.jl file itself.

7

I didn’t have a chance to look at your .jl file in detail, but if you save the data as text (e.g. CSV) and then load it one line at a time it should work. You would need to convert the strings to symbols during the loading process.

8

I’m guessing you would also run into issues if you generated a C++ file with 400,000 lines of code to initialize a dictionary and tried to compile it. In general you should read data programatically from a data source (e.g., CSV) instead of generating julia source code.

1 Like
9

Yes, trying CSV now…
But it is not just about importing data. Rather about sourcing large files.
(BTW, In some math-prog languages, like like GLPK/MathProg, there is no CSV option. And large scripts with data is normal practice in GAMS. Though they are not general programming languages…)