# Extreme memory usage seemingly caused by using the functions \`map\` and \`Iterators.flatten\`

**URL:** <https://discourse.julialang.org/t/extreme-memory-usage-seemingly-caused-by-using-the-functions-map-and-iterators-flatten/17141>\
**Category:** Performance\
**Tags:** memory, memory-allocation\
**Created:** [November 4, 2018, 5:12pm UTC](https://discourse.julialang.org/t/extreme-memory-usage-seemingly-caused-by-using-the-functions-map-and-iterators-flatten/17141 "2018-11-04T17:12:11Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![SZJX](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/szjx/32/4036_2.png) [@SZJX](https://discourse.julialang.org/u/SZJX)\
**Post date:** [November 4, 2018, 5:12pm UTC](https://discourse.julialang.org/t/extreme-memory-usage-seemingly-caused-by-using-the-functions-map-and-iterators-flatten/17141/1 "2018-11-04T17:12:11Z")

</div>

I had a program that was running extremely slowly so I ran memory benchmark on it. Apparently one particular line of code accumulated a whopping 27GB of memory allocation…

```julia
        (lgamma(theta(pyp)) - lgamma(theta(pyp) + pyp.crp.totalcustomers) +
                lgamma(theta(pyp) / d(pyp) + pyp.crp.ntablegroups) -
                lgamma(theta(pyp) / d(pyp)) +
                pyp.crp.ntablegroups * (log(d(pyp)) - lgamma(1 - d(pyp))) +
                sum(map(c -> lgamma(c - d(pyp)), Iterators.flatten(values(pyp.crp.tablegroups))))
                )

```

I suspected it’s the part

```julia
                sum(map(c -> lgamma(c - d(pyp)), Iterators.flatten(values(pyp.crp.tablegroups))))

```

that caused the problem. So I rewrote it as a simple loop:

```julia
        temp::Float64 = 0.0
        for tablegroup in values(pyp.crp.tablegroups)
            for table_customer_count in tablegroup
                temp += lgamma(table_customer_count - d(pyp))
            end
        end

        (lgamma(theta(pyp)) - lgamma(theta(pyp) + pyp.crp.totalcustomers) +
                lgamma(theta(pyp) / d(pyp) + pyp.crp.ntablegroups) -
                lgamma(theta(pyp) / d(pyp)) +
                pyp.crp.ntablegroups * (log(d(pyp)) - lgamma(1 - d(pyp))) +
                temp
                )

```

and in a test run on simple input data, the memory usage by that part was reduced 10-fold, from nearly 1GB (far exceeding any other part of the program) to only about 100MB.

One huge advantage of Julia is how easy it is to use FP patterns in it. However I didn’t expect such a huge performance hit. Maybe I didn’t write the code in an optimal way? Could this memory issue be improved in some other way, e.g. with more type annotations?

Or maybe there are some other problems with my program which were somehow dealt with by this change of code?

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [November 4, 2018, 5:15pm UTC](https://discourse.julialang.org/t/extreme-memory-usage-seemingly-caused-by-using-the-functions-map-and-iterators-flatten/17141/2 "2018-11-04T17:15:48Z")

</div>

`flatten` has recently been improved ([https://github.com/JuliaLang/julia/pull/29786](https://github.com/JuliaLang/julia/pull/29786)) and this change will be in Julia v1.0.2 which will be released in a few days. Hopefully this will help with this case. If you can build from source you can try out the `release-1.0` branch in the meantime.

---

<div class="post-metadata">

**Author:** ![SZJX](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/szjx/32/4036_2.png) [@SZJX](https://discourse.julialang.org/u/SZJX)\
**Post date:** [November 4, 2018, 5:40pm UTC](https://discourse.julialang.org/t/extreme-memory-usage-seemingly-caused-by-using-the-functions-map-and-iterators-flatten/17141/3 "2018-11-04T17:40:24Z")

</div>

Thanks. I was also wondering whether this is some performance issue that has been fixed in v1.0.2. I’ll give it a try later when it’s released then. I had to use the binary version because of [https://github.com/JuliaLinearAlgebra/Arpack.jl/issues/5](https://github.com/JuliaLinearAlgebra/Arpack.jl/issues/5)