# Efficient merging of large dictionaries

**URL:** <https://discourse.julialang.org/t/efficient-merging-of-large-dictionaries/40687>\
**Category:** Performance\
**Created:** [June 3, 2020, 5:39pm UTC](https://discourse.julialang.org/t/efficient-merging-of-large-dictionaries/40687 "2020-06-03T17:39:08Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![racinmat](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/racinmat/32/11715_2.png) [@racinmat](https://discourse.julialang.org/u/racinmat)\
**Post date:** [June 3, 2020, 5:39pm UTC](https://discourse.julialang.org/t/efficient-merging-of-large-dictionaries/40687/1 "2020-06-03T17:39:08Z")

</div>

Hi, I have few (tens, max. hundreds) Dict{String, Int} dictionaries which I want to merge together using merge(+, dicts…).  
The dictionaries are fairly large (up to 100k keys), and have lots of common keys.  
It’s working as expected, but it’s very slow. Most of the time is spent in the actual merging in ht\_keyindex and ht\_keyindex2!. Is there anything I could do to make it faster, or did I hit the limit? I see that the merge itself is simple [julia/abstractdict.jl at v1.4.2 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/blob/v1.4.2/base/abstractdict.jl#L216), so I’m wondering if there is anything that could be done for this usecase

---

<div class="post-metadata">

**Author:** ![pixel27](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pixel27/32/8902_2.png) [@pixel27](https://discourse.julialang.org/u/pixel27)\
**Post date:** [June 3, 2020, 7:55pm UTC](https://discourse.julialang.org/t/efficient-merging-of-large-dictionaries/40687/2 "2020-06-03T19:55:12Z")

</div>

Not sure if it would be faster per se but you could try extracting all the key/value pairs from the various Dict objects into a Vector of Tuple{String, Int}, sort them, then create the Dict by iterating over lists at the same time. It will take a lot of memory since you will need all the Dict objects in memory at once. You could also use threading to perform the conversion/sort of the Vectors in parallel.

If it’s faster or not probably depends on the number of cores you have.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [June 4, 2020, 9:32am UTC](https://discourse.julialang.org/t/efficient-merging-of-large-dictionaries/40687/3 "2020-06-04T09:32:12Z")

</div>

I would consider something like

```julia
function merge_all(combine, dicts)
    d1 = first(dicts)
    result = Dict{keytype(d1),valtype(d1)}()
    for d in dicts
        merge!(combine, result, d)
    end
    result
end

```

(untested)

---

<div class="post-metadata">

**Author:** ![racinmat](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/racinmat/32/11715_2.png) [@racinmat](https://discourse.julialang.org/u/racinmat)\
**Post date:** [June 4, 2020, 9:33am UTC](https://discourse.julialang.org/t/efficient-merging-of-large-dictionaries/40687/4 "2020-06-04T09:33:31Z")

</div>

I think this is basically the same thing as official implementation, right?

But I just discovered [GitHub - andyferris/Dictionaries.jl: An alternative interface for dictionaries in Julia, for improved productivity and performance](https://github.com/andyferris/Dictionaries.jl) so I’ll give them a try.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [June 4, 2020, 10:39am UTC](https://discourse.julialang.org/t/efficient-merging-of-large-dictionaries/40687/5 "2020-06-04T10:39:03Z")

</div>

Yes, for ` merge!`.