# A way to create large dictionary? "ERROR: LoadError: syntax: expression too large"

**URL:** <https://discourse.julialang.org/t/a-way-to-create-large-dictionary-error-loaderror-syntax-expression-too-large/37247>\
**Category:** New to Julia\
**Tags:** question, jump\
**Created:** [April 8, 2020, 9:55pm UTC](https://discourse.julialang.org/t/a-way-to-create-large-dictionary-error-loaderror-syntax-expression-too-large/37247 "2020-04-08T21:55:33Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![olugovoy](https://avatars.discourse-cdn.com/v4/letter/o/a698b9/32.png) [@olugovoy](https://discourse.julialang.org/u/olugovoy)\
**Post date:** [April 8, 2020, 9:55pm UTC](https://discourse.julialang.org/t/a-way-to-create-large-dictionary-error-loaderror-syntax-expression-too-large/37247/1 "2020-04-08T21:55:34Z")

</div>

Hi there,

What is the best way to create a large dictionary?  
This expression works for smaller Dicts:  
pDemand = Dict(  
(:DEM\_ELC\_DH, :ELC, :AL, :2018, :d001\_h00) =\> 20.4280246,  
(:DEM\_ELC\_DH, :ELC, :AR, :2018, :d001\_h00) =\> 9.4801576,  
(:DEM\_ELC\_DH, :ELC, :AZ, :2018, :d001\_h00) =\> 12.5974967,  
(:DEM\_ELC\_DH, :ELC, :CA, :2018, :d001\_h00) =\> 22.195113,  
(:DEM\_ELC\_DH, :ELC, :CO, :2018, :d001\_h00) =\> 7.1842716,  
…  
)  
But if there are 400000+ rows, the error appears:  
“ERROR: LoadError: syntax: expression too large”  
(which is a bit surprising, I haven’t experienced any limits on expressions length in other languages - like GAMS or Python/Pyomo)

What would be a better way to create a Dictionary in Julia? We use the format as data input & mapping in JuMP.

Here is a big Dict to experiment with: [long\_dict.jl](https://www.dropbox.com/transfer/AAAAAFYGd-Zcj7JE3pp2icj5P2MSQLqZB62ny0yds4MfBRUrnNbdtJo)

Thanks!

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [April 9, 2020, 12:06am UTC](https://discourse.julialang.org/t/a-way-to-create-large-dictionary-error-loaderror-syntax-expression-too-large/37247/2 "2020-04-09T00:06:36Z")

</div>

I think the problem is that no one assumed that people would be writing megabyte large expressions in Julia. Would it make sense to store this data as a CSV or some other type of file and then load that into a dictionary?

---

<div class="post-metadata">

**Author:** ![ccoffrin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ccoffrin/32/400_2.png) [@ccoffrin](https://discourse.julialang.org/u/ccoffrin)\
**Post date:** [April 9, 2020, 1:03am UTC](https://discourse.julialang.org/t/a-way-to-create-large-dictionary-error-loaderror-syntax-expression-too-large/37247/3 "2020-04-09T01:03:16Z")

</div>

I believe the issue is that you are making the whole dictionary as a comprehension. Did you try something like,

```julia
pDemand = Dict( (:DEM_ELC_DH, :ELC, :AL, :2018, :d001_h00) => 20.4280246 )
pDemand[(:DEM_ELC_DH, :ELC, :AR, :2018, :d001_h00)] = 9.4801576
pDemand[(:DEM_ELC_DH, :ELC, :AZ, :2018, :d001_h00)] = 12.5974967
…

```

---

<div class="post-metadata">

**Author:** ![olugovoy](https://avatars.discourse-cdn.com/v4/letter/o/a698b9/32.png) [@olugovoy](https://discourse.julialang.org/u/olugovoy)\
**Post date:** [April 9, 2020, 1:37am UTC](https://discourse.julialang.org/t/a-way-to-create-large-dictionary-error-loaderror-syntax-expression-too-large/37247/4 "2020-04-09T01:37:10Z")

</div>

This seems to work – will test it for #37256. Thanks!

---

<div class="post-metadata">

**Author:** ![olugovoy](https://avatars.discourse-cdn.com/v4/letter/o/a698b9/32.png) [@olugovoy](https://discourse.julialang.org/u/olugovoy)\
**Post date:** [April 9, 2020, 1:40am UTC](https://discourse.julialang.org/t/a-way-to-create-large-dictionary-error-loaderror-syntax-expression-too-large/37247/5 "2020-04-09T01:40:40Z")

</div>

Agree! Though if this is an unnecessary restriction, I would vote to remove it.

---

<div class="post-metadata">

**Author:** ![olugovoy](https://avatars.discourse-cdn.com/v4/letter/o/a698b9/32.png) [@olugovoy](https://discourse.julialang.org/u/olugovoy)\
**Post date:** [April 9, 2020, 6:06pm UTC](https://discourse.julialang.org/t/a-way-to-create-large-dictionary-error-loaderror-syntax-expression-too-large/37247/6 "2020-04-09T18:06:24Z")

</div>

The strategy suggested by [ccoffrin](https://discourse.julialang.org/t/a-way-to-create-large-dictionary-error-loaderror-syntax-expression-too-large/37247/3) works well for a particular parameter/dictionary. But now there is the same error:

ERROR: LoadError: LoadError: ReadOnlyMemoryError()

while loading a long file with the script, following Julia crush. Is it an issue with long .jl files in general? Any ways to allocate more memory for the operation?

Here is the [data.jl](https://www.dropbox.com/transfer/AAAAAETubZuLADdiu0G3TBUxO0Ai-1DRuiC410qC4bW9Ibzj-Ab9VrI) file itself.

---

<div class="post-metadata">

**Author:** ![ccoffrin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ccoffrin/32/400_2.png) [@ccoffrin](https://discourse.julialang.org/u/ccoffrin)\
**Post date:** [April 9, 2020, 6:15pm UTC](https://discourse.julialang.org/t/a-way-to-create-large-dictionary-error-loaderror-syntax-expression-too-large/37247/7 "2020-04-09T18:15:57Z")

</div>

I didn’t have a chance to look at your `.jl` file in detail, but if you save the data as text (e.g. CSV) and then load it one line at a time it should work. You would need to convert the strings to symbols during the loading process.

---

<div class="post-metadata">

**Author:** ![miles.lubin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/miles.lubin/32/279_2.png) [@miles.lubin](https://discourse.julialang.org/u/miles.lubin)\
**Post date:** [April 9, 2020, 6:19pm UTC](https://discourse.julialang.org/t/a-way-to-create-large-dictionary-error-loaderror-syntax-expression-too-large/37247/8 "2020-04-09T18:19:27Z")

</div>

> [@olugovoy](#):
>
> which is a bit surprising, I haven’t experienced any limits on expressions length in other languages

I’m guessing you would also run into issues if you generated a C++ file with 400,000 lines of code to initialize a dictionary and tried to compile it. In general you should read data programatically from a data source (e.g., CSV) instead of generating julia source code.

---

<div class="post-metadata">

**Author:** ![olugovoy](https://avatars.discourse-cdn.com/v4/letter/o/a698b9/32.png) [@olugovoy](https://discourse.julialang.org/u/olugovoy)\
**Post date:** [April 9, 2020, 6:45pm UTC](https://discourse.julialang.org/t/a-way-to-create-large-dictionary-error-loaderror-syntax-expression-too-large/37247/9 "2020-04-09T18:45:23Z")

</div>

Yes, trying CSV now…  
But it is not just about importing data. Rather about sourcing large files.  
(BTW, In some math-prog languages, like like GLPK/MathProg, there is no CSV option. And large scripts with data is normal practice in GAMS. Though they are not general programming languages…)