# ForwardDiff & mul!()

**URL:** <https://discourse.julialang.org/t/forwarddiff-mul/28144>\
**Category:** Numerics\
**Created:** [August 28, 2019, 11:47pm UTC](https://discourse.julialang.org/t/forwarddiff-mul/28144 "2019-08-28T23:47:29Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![PharmCat](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pharmcat/32/6953_2.png) [@PharmCat](https://discourse.julialang.org/u/PharmCat)\
**Post date:** [August 28, 2019, 11:47pm UTC](https://discourse.julialang.org/t/forwarddiff-mul/28144/1 "2019-08-28T23:47:29Z")

</div>

Is it possible to make ForwardDiff works with mul!() inside function?

Or how to optimize allocations when I try to get gradient/hessian of likelihood function?

---

<div class="post-metadata">

**Author:** ![longemen3000](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/longemen3000/32/7298_2.png) [@longemen3000](https://discourse.julialang.org/u/longemen3000)\
**Post date:** [August 28, 2019, 11:54pm UTC](https://discourse.julialang.org/t/forwarddiff-mul/28144/2 "2019-08-28T23:54:48Z")

</div>

i can find mul! in base, where it is?

---

<div class="post-metadata">

**Author:** ![PharmCat](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pharmcat/32/6953_2.png) [@PharmCat](https://discourse.julialang.org/u/PharmCat)\
**Post date:** [August 29, 2019, 1:27am UTC](https://discourse.julialang.org/t/forwarddiff-mul/28144/3 "2019-08-29T01:27:06Z")

</div>

this is in LinearAlgebra

LinearAlgebra.mul!

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [August 29, 2019, 6:01am UTC](https://discourse.julialang.org/t/forwarddiff-mul/28144/4 "2019-08-29T06:01:59Z")

</div>

> [@PharmCat](#):
>
> Or how to optimize allocations when I try to get gradient/hessian of likelihood function?

You can allocate the result with element type for the appropriate `Dual`, but I don’t think it is worth the hassle. Just use non-modifying versions of everything.

---

<div class="post-metadata">

**Author:** ![PharmCat](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pharmcat/32/6953_2.png) [@PharmCat](https://discourse.julialang.org/u/PharmCat)\
**Post date:** [August 29, 2019, 10:00am UTC](https://discourse.julialang.org/t/forwarddiff-mul/28144/5 "2019-08-29T10:00:24Z")

</div>

I function i have part:

```julia
function reml(yv, Zv, p, Xv, θvec, β)
...
    for i = 1:n
        θ2m += Xv[i]'*iV*Xv[i]
        r = yv[i]-Xv[i]*β
        θ3 += r'*iV*r
    end
 ...
end

```

where Xv - array of matreces, i try to make temp array before cycle and modyfy like this:

```julia
     temp = zeros(p,p)
     for i = 1:n
        mul!(temp, Xv[i]', iV)
        θ2m += temp*Xv[i]
        r = yv[i]-Xv[i]*β
        θ3 += r'*iV*r
    end

```

and in general case it reduse allocations, bur ForwardDiff get me error:

```julia
ERROR: MethodError: no method matching Float64(::ForwardDiff.Dual{ForwardDiff.Tag{typeof(remlf2),Float64},ForwardDiff.Dual{ForwardDiff.Tag{typeof(remlf2),Float64},Float64,5},5})
Closest candidates are:
  Float64(::Real, ::RoundingMode) where T<:AbstractFloat at rounding.jl:194
  Float64(::T<:Number) where T<:Number at boot.jl:741
  Float64(::Int8) at float.jl:60
  ...
Stacktrace:
 [1] convert(::Type{Float64}, ::ForwardDiff.Dual{ForwardDiff.Tag{typeof(remlf2),Float64},ForwardDiff.Dual{ForwardDiff.Tag{typeof(remlf2),Float64},Float64,5},5}) at .\number.jl:7
 [2] setindex! at .\array.jl:769 [inlined]
 [3] _generic_matmatmul!(::Array{Float64,2}, ::Char, ::Char, ::Array{Float64,2}, ::Array{ForwardDiff.Dual{ForwardDiff.Tag{typeof(remlf2),Float64},ForwardDiff.Dual{ForwardDiff.Tag{typeof(remlf2),Float64},Float64,5},5},2}) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\LinearAlgebra\src\matmul.jl:727
 [4] generic_matmatmul! at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\LinearAlgebra\src\matmul.jl:581 [inlined]
 [5] mul!(::Array{Float64,2}, ::Adjoint{Float64,Array{Float64,2}}, ::Array{ForwardDiff.Dual{ForwardDiff.Tag{typeof(remlf2),Float64},ForwardDiff.Dual{ForwardDiff.Tag{typeof(remlf2),Float64},Float64,5},5},2}) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\LinearAlgebra\src\matmul.jl:293
 [6] reml2(::Array{Array{Float64,1},1}, ::Array{Array{Float64,2},1}, ::Int64, ::Array{Array{Float64,2},1}, ::Array{ForwardDiff.Dual{ForwardDiff.Tag{typeof(remlf2),Float64},ForwardDiff.Dual{ForwardDiff.Tag{typeof(remlf2),Float64},Float64,5},5},1}, ::Array{Float64,1}) at G:\Cloud\Julia\Mixed\mixbe.jl:376
 [7] remlf2(::Array{ForwardDiff.Dual{ForwardDiff.Tag{typeof(remlf2),Float64},ForwardDiff.Dual{ForwardDiff.Tag{typeof(remlf2),Float64},Float64,5},5},1}) at G:\Cloud\Julia\Mixed\mixbe.jl:386
 [8] vector_mode_gradient(::typeof(remlf2), ::Array{ForwardDiff.Dual{ForwardDiff.Tag{typeof(remlf2),Float64},Float64,5},1}, ::ForwardDiff.GradientConfig{ForwardDiff.Tag{typeof(remlf2),Float64},ForwardDiff.Dual{ForwardDiff.Tag{typeof(remlf2),Float64},Float64,5},5,Array{ForwardDiff.Dual{ForwardDiff.Tag{typeof(remlf2),Float64},ForwardDiff.Dual{ForwardDiff.Tag{typeof(remlf2),Float64},Float64,5},5},1}}) at C:\...\.julia\packages\ForwardDiff\N0wMF\src\apiutils.jl:37
 [9] gradient at C:\Users\vsarn\.julia\packages\ForwardDiff\N0wMF\src\gradient.jl:17 [inlined]
 [10] #94 at C:\Users\vsarn\.julia\packages\ForwardDiff\N0wMF\src\hessian.jl:16 [inlined]
 [11] vector_mode_dual_eval(::getfield(ForwardDiff, Symbol("##94#95")){typeof(remlf2),ForwardDiff.HessianConfig{ForwardDiff.Tag{typeof(remlf2),Float64},Float64,5,Array{ForwardDiff.Dual{ForwardDiff.Tag{typeof(remlf2),Float64},ForwardDiff.Dual{ForwardDiff.Tag{typeof(remlf2),Float64},Float64,5},5},1},Array{ForwardDiff.Dual{ForwardDiff.Tag{typeof(remlf2),Float64},Float64,5},1}}}, ::Array{Float64,1}, ::ForwardDiff.JacobianConfig{ForwardDiff.Tag{typeof(remlf2),Float64},Float64,5,Array{ForwardDiff.Dual{ForwardDiff.Tag{typeof(remlf2),Float64},Float64,5},1}}) at C:\...\.julia\packages\ForwardDiff\N0wMF\src\apiutils.jl:37
 [12] vector_mode_jacobian(::getfield(ForwardDiff, Symbol("##94#95")){typeof(remlf2),ForwardDiff.HessianConfig{ForwardDiff.Tag{typeof(remlf2),Float64},Float64,5,Array{ForwardDiff.Dual{ForwardDiff.Tag{typeof(remlf2),Float64},ForwardDiff.Dual{ForwardDiff.Tag{typeof(remlf2),Float64},Float64,5},5},1},Array{ForwardDiff.Dual{ForwardDiff.Tag{typeof(remlf2),Float64},Float64,5},1}}}, ::Array{Float64,1}, ::ForwardDiff.JacobianConfig{ForwardDiff.Tag{typeof(remlf2),Float64},Float64,5,Array{ForwardDiff.Dual{ForwardDiff.Tag{typeof(remlf2),Float64},Float64,5},1}}) at C:\...\.julia\packages\ForwardDiff\N0wMF\src\jacobian.jl:140
 [13] jacobian(::Function, ::Array{Float64,1}, ::ForwardDiff.JacobianConfig{ForwardDiff.Tag{typeof(remlf2),Float64},Float64,5,Array{ForwardDiff.Dual{ForwardDiff.Tag{typeof(remlf2),Float64},Float64,5},1}}, ::Val{false}) at C:\...\.julia\packages\ForwardDiff\N0wMF\src\jacobian.jl:17
 [14] hessian(::Function, ::Array{Float64,1}, ::ForwardDiff.HessianConfig{ForwardDiff.Tag{typeof(remlf2),Float64},Float64,5,Array{ForwardDiff.Dual{ForwardDiff.Tag{typeof(remlf2),Float64},ForwardDiff.Dual{ForwardDiff.Tag{typeof(remlf2),Float64},Float64,5},5},1},Array{ForwardDiff.Dual{ForwardDiff.Tag{typeof(remlf2),Float64},Float64,5},1}}, ::Val{true}) at C:\...\.julia\packages\ForwardDiff\N0wMF\src\hessian.jl:17
 [15] hessian(::Function, ::Array{Float64,1}, ::ForwardDiff.HessianConfig{ForwardDiff.Tag{typeof(remlf2),Float64},Float64,5,Array{ForwardDiff.Dual{ForwardDiff.Tag{typeof(remlf2),Float64},ForwardDiff.Dual{ForwardDiff.Tag{typeof(remlf2),Float64},Float64,5},5},1},Array{ForwardDiff.Dual{ForwardDiff.Tag{typeof(remlf2),Float64},Float64,5},1}}) at C:\...\.julia\packages\ForwardDiff\N0wMF\src\hessian.jl:15 (repeats 2 times)
 [16] top-level scope at none:0

```

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [August 29, 2019, 10:16am UTC](https://discourse.julialang.org/t/forwarddiff-mul/28144/6 "2019-08-29T10:16:33Z")

</div>

ForwardDiff works by operating on dual numbers:

[http://www.juliadiff.org/ForwardDiff.jl/latest/dev/how\_it\_works.html](http://www.juliadiff.org/ForwardDiff.jl/latest/dev/how_it_works.html)

As I said, you should either consider using non-modifying versions of these functions, or (if you are feeling adventurous) pre-allocate buffers of the correct type. It is better to do this for an inner loop only, where the outer part will know the type (`Dual` has a type tag).

To help with the latter, it would be great to have a full MWE.

> [@Please read: make it easier to help you](https://discourse.julialang.org/t/psa-make-it-easier-to-help-you/14757):
>
> Welcome to the Julia Discourse! We are enthusiastic about helping Julia programmers, both beginner and experienced. This public service announcement (PSA) outlines best practices when asking for help. Following these points makes it easier for us to help you and more likely you’ll get a prompt, useful answer. Keywords are highlighted to make it easier to refer to specific points. Choose a descriptive title that captures the key part of your question, eg “plots with multiple axes” instead of …

---

<div class="post-metadata">

**Author:** ![Elrod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/elrod/32/22461_2.png) [@Elrod](https://discourse.julialang.org/u/Elrod)\
**Post date:** [August 29, 2019, 10:38am UTC](https://discourse.julialang.org/t/forwarddiff-mul/28144/7 "2019-08-29T10:38:10Z")

</div>

```julia
temp = zeros(promote_type(eltype(first(Xv)),eltype(iV)),p,p)

```