# Is there way to avoid \`materialize\` and \`copy\`?

**URL:** <https://discourse.julialang.org/t/is-there-way-to-avoid-materialize-and-copy/17467>\
**Category:** Performance\
**Created:** [November 13, 2018, 9:31am UTC](https://discourse.julialang.org/t/is-there-way-to-avoid-materialize-and-copy/17467 "2018-11-13T09:31:44Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![misha\_mikhasenko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/misha_mikhasenko/32/5060_2.png) [@misha\_mikhasenko](https://discourse.julialang.org/u/misha_mikhasenko)\
**Post date:** [November 13, 2018, 9:31am UTC](https://discourse.julialang.org/t/is-there-way-to-avoid-materialize-and-copy/17467/1 "2018-11-13T09:31:44Z")

</div>

I have a function `createLLHandGRAD(data, kw...)` which returns several callbacks.  
One of them is the `LLH` function

```julia
function LLH(pars)
    # get a vector of parameters
    @inbounds res = sum(log(COHSQ(EXTND(data[e,:]).*pars)) for e in 1:Nd)
    bilin = Nd * (pars ⋅ (BM * pars)) # BM is a matrix
    - res + bilin # just a number
end

```

By profiling, I realized that in addition to calculation time(first towel) and `_getindex`(last towel)  
40% of the time is spent to `materialize` and `copy`.

 ![profiling](https://global.discourse-cdn.com/julialang/original/3X/e/b/eb6091218224422b5dbd4c75f36ef6c08273deb4.png)

Is it something one can expect? Are there any performance tips to me?

Thank you in advance.

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [November 13, 2018, 11:37am UTC](https://discourse.julialang.org/t/is-there-way-to-avoid-materialize-and-copy/17467/2 "2018-11-13T11:37:42Z")

</div>

> [@misha\_mikhasenko](#):
>
> data[e,:]

I would guess `data[e,:]` is costly. Not only are you allocating the slice `data[e,:]`, you are also accessing the data in a suboptimal pattern. Access according to `data[:,e]` would be more efficient since julia uses column major storage. Would it be possible to transpose the `data` array?

To avoid allocation once the access pattern is changed, try `@view(data[:,e])`

Edit: The problem with the access pattern is probably only significant if `data` is larger than your smallest cache size. The allocation is probably always an issue. It’s hard to say anything more without knowing what `EXTND` and `COHSQ` are doing.

---

<div class="post-metadata">

**Author:** ![misha\_mikhasenko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/misha_mikhasenko/32/5060_2.png) [@misha\_mikhasenko](https://discourse.julialang.org/u/misha_mikhasenko)\
**Post date:** [November 13, 2018, 12:03pm UTC](https://discourse.julialang.org/t/is-there-way-to-avoid-materialize-and-copy/17467/3 "2018-11-13T12:03:01Z")

</div>

Thank you for the reply. Cool, it believe it will help.  
The `data` is a huge array ~100x1e6.  
I will report as soon as I test it.

---

<div class="post-metadata">

**Author:** ![DNF](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dnf/32/10191_2.png) [@DNF](https://discourse.julialang.org/u/DNF)\
**Post date:** [November 13, 2018, 12:25pm UTC](https://discourse.julialang.org/t/is-there-way-to-avoid-materialize-and-copy/17467/4 "2018-11-13T12:25:11Z")

</div>

This is hard to fix without more information about the variables involved.

`data`, `Nd`, and `BM` are not input arguments. Are they globals? That could be bad. In general, it’s advantageous to pass everything as input arguments to your function.

You are creating a lot of temporary arrays. Try to use fused broadcasting, where appropriate:

```julia
sum(log.(COHSQ.(EXTND.(data[e,:]).*pars)) for e in 1:Nd)

```

and maybe `view`s instead of slices.

It is possible to get good performance with vectorized expressions, but it is also easy to fail. Try to rewrite the `res` expression using loops, and you might get dramatic speed-ups.

`pars ⋅ (BM * pars)` might be inefficient. I would guess there’s a BLAS call that efficiently calculates `x'*A*x` type expressions, though I’m not to well versed in BLAS gymnastics.

**Edit:** Not sure about the BLAS thing now, but, at least on my computer, it’s faster to do

```julia
pars' * BM * pars

```

---

<div class="post-metadata">

**Author:** ![DNF](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dnf/32/10191_2.png) [@DNF](https://discourse.julialang.org/u/DNF)\
**Post date:** [November 13, 2018, 12:31pm UTC](https://discourse.julialang.org/t/is-there-way-to-avoid-materialize-and-copy/17467/5 "2018-11-13T12:31:50Z")

</div>

> [@misha\_mikhasenko](#):
>
> `- res + bilin`

Oh, and I would strongly advise you to use explicit `return` statements:

```julia
return -res + bilin

```

I read your code several times, thinking that the last line was a continuation of the expression on the second-to-last line. Implicit `return` statements can be very confusing.

---

<div class="post-metadata">

**Author:** ![misha\_mikhasenko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/misha_mikhasenko/32/5060_2.png) [@misha\_mikhasenko](https://discourse.julialang.org/u/misha_mikhasenko)\
**Post date:** [November 13, 2018, 5:13pm UTC](https://discourse.julialang.org/t/is-there-way-to-avoid-materialize-and-copy/17467/6 "2018-11-13T17:13:38Z")

</div>

Thank you for replies and sorry for providing not enough details.

- `EXTND(v)` creates a new vector by picking components of the `v`
- `COHSQ(v)` calculates a bilinear form of the `v`

To the comment

> [@DNF](#):
>
> You are creating a lot of temporary arrays. Try to use fused broadcasting, where appropriate:
> 
> ```julia
> sum(log.(COHSQ.(EXTND.(data[e,:]).*pars)) for e in 1:Nd)
> 
> ```

As far as I understand the fused broadcasting is element-wise operations. I do not see how to translate my operations `EXTND` and `COHSQ` to ones which could be broadcasted.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [November 13, 2018, 5:16pm UTC](https://discourse.julialang.org/t/is-there-way-to-avoid-materialize-and-copy/17467/7 "2018-11-13T17:16:59Z")

</div>

It is really very hard to say anything at all without knowing the specifics of `COHSQ` and `EXTND`.

In general, you will get much more accurate help if it is actually possible to run your code.

---

<div class="post-metadata">

**Author:** ![DNF](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dnf/32/10191_2.png) [@DNF](https://discourse.julialang.org/u/DNF)\
**Post date:** [November 13, 2018, 5:27pm UTC](https://discourse.julialang.org/t/is-there-way-to-avoid-materialize-and-copy/17467/8 "2018-11-13T17:27:27Z")

</div>

Well, I just had to take a guess.

---

<div class="post-metadata">

**Author:** ![misha\_mikhasenko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/misha_mikhasenko/32/5060_2.png) [@misha\_mikhasenko](https://discourse.julialang.org/u/misha_mikhasenko)\
**Post date:** [November 13, 2018, 7:50pm UTC](https://discourse.julialang.org/t/is-there-way-to-avoid-materialize-and-copy/17467/9 "2018-11-13T19:50:34Z")

</div>

Thank you for the advice. The code is not that simple, I was afraid that nobody would bother to look.  
Anyway, the code is in github, indeed.

- The main function [`createLLHandGRAD`](https://github.com/mmikhasenko/pwa_from_scratch/blob/dev/src/FittingPWALikelihood.jl#L11) wraps the data and returns a few functions to calculate likelihood and derivatives.
- [`LLH` function](https://github.com/mmikhasenko/pwa_from_scratch/blob/dev/src/FittingPWALikelihood.jl#L32) as discussed above.
- The [EXTND](https://github.com/mmikhasenko/pwa_from_scratch/blob/dev/src/PWAHelper.jl#L66) and [COHSQ](https://github.com/mmikhasenko/pwa_from_scratch/blob/dev/src/PWAHelper.jl#L77).

I will finish my tests and try to summarize with a simple example.

---

<div class="post-metadata">

**Author:** ![misha\_mikhasenko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/misha_mikhasenko/32/5060_2.png) [@misha\_mikhasenko](https://discourse.julialang.org/u/misha_mikhasenko)\
**Post date:** [November 13, 2018, 9:14pm UTC](https://discourse.julialang.org/t/is-there-way-to-avoid-materialize-and-copy/17467/10 "2018-11-13T21:14:37Z")

</div>

The suggestions of the memory access and `@view` helped.  
I have got from 300ms to 260ms for the likelihood function  
and from 1.12s to 0.959s for the gradient, which is something already.

Copying and materializing is still a quite some part of the total time.

 ![update](https://global.discourse-cdn.com/julialang/original/3X/b/f/bfbae04dafc21bc3ce564dc8634f9f88b135cbad.png)

For some reason, Juno profiler does not tell me where exactly the time is spent in my code showing directly `broadcast.jl`. I should probably investigate other profiling methods.

---

<div class="post-metadata">

**Author:** ![mbauman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbauman/32/31082_2.png) [@mbauman](https://discourse.julialang.org/u/mbauman)\
**Post date:** [November 13, 2018, 11:18pm UTC](https://discourse.julialang.org/t/is-there-way-to-avoid-materialize-and-copy/17467/11 "2018-11-13T23:18:34Z")

</div>

I think you’re getting distracted by the names here. They’re not just re-arranging memory — they’re actually doing the execution of the broadcast. They really are a part of your calculation. The part there that’s not is `similar` — that’s just allocating the space for the (likely intermediate) result. I’m a little surprised it’s so big compared to the execution of the broadcast, but it’s looking like your broadcast is just a single `.*` and I’m guessing that there’s also lots of GC time collecting temporaries that ends up getting attributed to `similar`. But hey, I suppose I shouldn’t be surprised since I still don’t know what code you’re running.

How to fix it? Do the broadcast in-place with a pre-allocated scratch buffer (with `.=`). Or work to make things fuse farther. Or use more laziness to do the multiplication “on the fly” as you need it instead of allocating a temporary.

Have you seen:

[https://docs.julialang.org/en/v1/manual/performance-tips/](https://docs.julialang.org/en/v1/manual/performance-tips/)

---

<div class="post-metadata">

**Author:** ![juthohaegeman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juthohaegeman/32/8620_2.png) [@juthohaegeman](https://discourse.julialang.org/u/juthohaegeman)\
**Post date:** [November 14, 2018, 6:32am UTC](https://discourse.julialang.org/t/is-there-way-to-avoid-materialize-and-copy/17467/12 "2018-11-14T06:32:13Z")

</div>

Significant time spent allocating large temporaries is something I have also observed in other contexts.  
If you `@(b)time` your operation, how much time is spent in GC?  
There is also one or more issues about this: [GC improvements for #28986 by JeffBezanson · Pull Request #29290 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/pull/29290)

---

<div class="post-metadata">

**Author:** ![misha\_mikhasenko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/misha_mikhasenko/32/5060_2.png) [@misha\_mikhasenko](https://discourse.julialang.org/u/misha_mikhasenko)\
**Post date:** [November 14, 2018, 7:48am UTC](https://discourse.julialang.org/t/is-there-way-to-avoid-materialize-and-copy/17467/13 "2018-11-14T07:48:18Z")

</div>

Thank you for your replies. I think I got an idea in which direction to research.  
I should also come up with a simple example.

Benchmarking:

```julia
julia> @benchmark LLH($(pars0))
  memory estimate: 385.79 MiB
  allocs estimate: 1705852
  --------------
  minimum time: 212.383 ms (14.96% GC)
  median time: 225.188 ms (15.65% GC)
  mean time: 225.274 ms (15.46% GC)
  maximum time: 237.897 ms (14.86% GC)
  --------------
  samples: 23
  evals/sample: 1

```

```julia
julia> LLH(pars0)
  0.309661 seconds (1.71 M allocations: 385.791 MiB, 18.37% gc time)
elapsed time (ns): 309661086
gc time (ns): 56893873
bytes allocated: 404530672
pool allocs: 1584010
non-pool GC allocs:121846
GC pauses: 17

```

---

<div class="post-metadata">

**Author:** ![lwabeke](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lwabeke/32/4005_2.png) [@lwabeke](https://discourse.julialang.org/u/lwabeke)\
**Post date:** [November 14, 2018, 8:20am UTC](https://discourse.julialang.org/t/is-there-way-to-avoid-materialize-and-copy/17467/14 "2018-11-14T08:20:04Z")

</div>

As a start I would suggest creating a version of EXTND! that takes as input a temp variable where the output should be stored.

Changing cohsq to use the abs2 as a function parameter should also help to prevent it from creating another temporary to store the abs2 values:

```julia
function cohsq(X, pblocks)
    sum(abs2, (sum(@view(X[bl]))) for bl in pblocks)
end

```

Again in LLH, either use `sum(log.(` or `sum(log,` although if Nd isn’t big, it might not make much of a difference.

---

<div class="post-metadata">

**Author:** ![misha\_mikhasenko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/misha_mikhasenko/32/5060_2.png) [@misha\_mikhasenko](https://discourse.julialang.org/u/misha_mikhasenko)\
**Post date:** [November 14, 2018, 5:31pm UTC](https://discourse.julialang.org/t/is-there-way-to-avoid-materialize-and-copy/17467/15 "2018-11-14T17:31:54Z")

</div>

> [@lwabeke](#):
>
> As a start I would suggest creating a version of EXTND! that takes as input a temp variable where the output should be stored.

Wow! It gained a factor of 2 in speed and 5-6 times less garbage.  
Thank you, Master!

```julia
  memory estimate: 32.54 MiB
  allocs estimate: 1401242
  --------------
  minimum time: 135.111 ms (2.09% GC)
  median time: 147.920 ms (3.48% GC)
  mean time: 150.981 ms (3.39% GC)
  maximum time: 171.962 ms (6.02% GC)
  --------------
  samples: 34
  evals/sample: 1

```

New LLH summation:

```julia
    function LLH(pars)
        v = EXTND(@view(transposedPsiDT[:,1]))
        res = sum(log,
            let
                EXTND!(v,@view(transposedPsiDT[:,e]))
                v .*= pars
                COHSQ(v)
            end for e in 1:Nd)
        bilin = Nd * (pars' * BM * pars)
        return -res + bilin
    end

```

---

<div class="post-metadata">

**Author:** ![tkluck](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkluck/32/15769_2.png) [@tkluck](https://discourse.julialang.org/u/tkluck)\
**Post date:** [November 14, 2018, 10:07pm UTC](https://discourse.julialang.org/t/is-there-way-to-avoid-materialize-and-copy/17467/16 "2018-11-14T22:07:00Z")

</div>

> For some reason, Juno profiler does not tell me where exactly the time is spent in my code showing directly `broadcast.jl` . I should probably investigate other profiling methods.

You may be interested in trying [StatProfilerHTML](https://github.com/tkluck/StatProfilerHTML.jl), which gives [line-by-line explorable code](https://www.infty.nl/StatProfilerHTML.jl/example-output/). (Disclosure: I wrote the package.)

It’s available through `add StatProfilerHTML`. If you’re on Julia 1.0 and if you use macros, it’s a bit hampered by [a line number bug](https://github.com/JuliaLang/julia/issues/28618) in Julia itself, but [a fix](https://github.com/JuliaLang/julia/pull/30011) should soon be available in the next point release.

---

<div class="post-metadata">

**Author:** ![foobar\_lv2](https://avatars.discourse-cdn.com/v4/letter/f/ee59a6/32.png) [@foobar\_lv2](https://discourse.julialang.org/u/foobar_lv2)\
**Post date:** [November 14, 2018, 10:08pm UTC](https://discourse.julialang.org/t/is-there-way-to-avoid-materialize-and-copy/17467/17 "2018-11-14T22:08:28Z")

</div>

Now you still allocate and clear 1.4 million objects with a mean size of 24 bytes. So, these must be mostly tiny wrapper objects.

I assume `pars` is your shorthand for “various parameters I don’t want to post on discourse”? Or are `Nd`, `transposedPsiDT` and `BM` globals? In that case, consider passing them as parameters. Type-instability in `LLH` is relatively benign, since you only execute very few expensive operations, but this would clear up the profiles a lot.

If they are closures (`LLH` is inner function), then the same applies: Closures without boxes are tricky, so it is often better to just forget that inner functions are a feature that julia supports.

---

<div class="post-metadata">

**Author:** ![misha\_mikhasenko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/misha_mikhasenko/32/5060_2.png) [@misha\_mikhasenko](https://discourse.julialang.org/u/misha_mikhasenko)\
**Post date:** [November 15, 2018, 8:57am UTC](https://discourse.julialang.org/t/is-there-way-to-avoid-materialize-and-copy/17467/18 "2018-11-15T08:57:08Z")

</div>

Thank you for the comment.

> [@foobar\_lv2](#):
>
> Now you still allocate and clear 1.4 million objects with a mean size of 24 bytes. So, these must be mostly tiny wrapper objects.

With last improvements (got rid of `EXTND`), the LLH call is 80ms

```julia
    function LLH(pars)
        res = sum(log,
                sum(abs2,
                    @view(transposedExtendedPsiDT[bl,e]) ⋅ @view(pars[bl])
                for bl in pblocks)
                    for e in 1:Nd)
        bilin = Nd * (pars' * BM * pars)
        return -res + bilin
    end

```

however, still 1.5M allocations. I guess internal of `sum`…(?)

> [@foobar\_lv2](#):
>
> I assume `pars` is your shorthand for “various parameters I don’t want to post on discourse”?  
> Or are `Nd` , `transposedPsiDT` and `BM` globals?

`pars` is a vector of `Float64`. The `Nd` , `transposedPsiDT` and `BM` are local values of the function which returns closure `LLH`?

> [@foobar\_lv2](#):
>
> If they are closures ( `LLH` is inner function), then the same applies: Closures without boxes are tricky, so it is often better to just forget that inner functions are a feature that julia supports.

What do you mean with boxes?  
It is so convenient to create likelihood function for given probability density and data set. How else could I archive that?

---

<div class="post-metadata">

**Author:** ![pfitzseb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pfitzseb/32/45566_2.png) [@pfitzseb](https://discourse.julialang.org/u/pfitzseb)\
**Post date:** [November 15, 2018, 9:02am UTC](https://discourse.julialang.org/t/is-there-way-to-avoid-materialize-and-copy/17467/19 "2018-11-15T09:02:45Z")

</div>

> [@misha\_mikhasenko](#):
>
> For some reason, Juno profiler does not tell me where exactly the time is spent in my code showing directly `broadcast.jl` . I should probably investigate other profiling methods.

You can just click on one of the boxes in the Profiler Pane and that’ll take you to the source code, which will allow you to figure out what exactly is taking a long time:

 ![](https://global.discourse-cdn.com/julialang/original/3X/9/2/9280357520bda127ecd0053a9c7ccac9b95a8083.png)

---

<div class="post-metadata">

**Author:** ![misha\_mikhasenko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/misha_mikhasenko/32/5060_2.png) [@misha\_mikhasenko](https://discourse.julialang.org/u/misha_mikhasenko)\
**Post date:** [November 15, 2018, 9:10am UTC](https://discourse.julialang.org/t/is-there-way-to-avoid-materialize-and-copy/17467/20 "2018-11-15T09:10:10Z")

</div>

That is right. I use this nice tool every day.  
I just sometimes have an impression that it skips some intermediate blocks and point me directly to internal `Base` functions modules which I do not touch usually and it is also not straightforward to figure out what does wrong.

[Next page](https://discourse.julialang.org/t/is-there-way-to-avoid-materialize-and-copy/17467.md?page=2)