ForwardDiff cannot assign ForwardDiff.Dual to input array

General Usage
,
1

Hi,

I am trying to solve a following problem. I have an objective function that gets some initial parameters (init in the example) and metaparameter(s) (meta in example). The parameters are then updated w.r.t. metaparameters and data (not included in the example for simplicity) and a value of the objective is computed based on the updated parameters (and data). The task is to find optimal metaparameters with JuMP for which I need to compute the gradient.

The problem is that the init is a Float64 array and I am getting the cannot convert ForwardDiff.Dual to Float64 error.

function f(meta, init)
    params = copy(init)
    for i=2:5
        params[i] = meta * params[i-1]
    end
    return sum(params)
end

s = [1.,2.,3.,4.,5.]
ForwardDiff.gradient(x -> f(x, s), [2])

On slack, I was directed here https://docs.sciml.ai/stable/basics/faq/#I-get-Dual-number-errors-when-I-solve-my-ODE-with-Rosenbrock-or-SDIRK-methods-1 but I could not quite get it to work.

2

Hello and welcome to the Julia community!

You need to allow the type of the vector over which you are differentiating to be parametric in order for ForwardDiff to operate correctly. Your issue was that by doing params = copy(init) you are forcing the type of params to be an array of floats. Please see the example below.

function f_new(meta::Array{T}, init) where {T<:Real}
    params = Array{T}(undef, size(init)[1], 1)
    for i=1:5
        params[i] = init[i]
    end
    for i=2:5
        params[i] = meta[1] * params[i-1]
    end
    return sum(params)
end

s = [1.,2.,3.,4.,5.]
ForwardDiff.gradient(x -> f_new(x, s), [2.0])

julia> ForwardDiff.gradient(x -> f_new(x, s), [2.0])
1-element Array{Float64,1}:
 49.0
3

Thank you. This seems to work for my MWE attempt, but not for my actual code. I have to check where is the difference.

4

Try using eltype() to debug. This often how I catch errors when I’m using ForwardDiff in my code. It’s likely that one of the many parameters might be having its type forced into something like an array of floats.

Same code using eltype() below. Notice that special ForwardDiff.Dual type being created. This is why you want to avoid forcing the type of a parameter to be Float64 because Julia can’t convert ForwardDiff.Dual type to a Float64, hence the "“cannot convert ForwardDiff.Dual to Float64 error.”

If you force the type of your parameter to be an array of floats, your code will try to promote ForwardDiff.Dual type to a Float64 and will fail.

function f_new(meta::Array{T}, init::Array{F}) where {T<:Real, F<:Float64}
    params = Array{T}(undef, size(init)[1], 1)
    println(eltype(params))
    for i=1:5
        params[i] = init[i]
    end
    println(eltype(params))
    for i=2:5
        params[i] = meta[1] * params[i-1]
    end
    println(eltype(params))
    return sum(params)
end

s = [1.,2.,3.,4.,5.]
ForwardDiff.gradient(x -> f_new(x, s), [2.0])

julia> ForwardDiff.gradient(x -> f_new(x, s), [2.0])
ForwardDiff.Dual{ForwardDiff.Tag{getfield(Main, Symbol("##73#74")),Float64},Float64,1}
ForwardDiff.Dual{ForwardDiff.Tag{getfield(Main, Symbol("##73#74")),Float64},Float64,1}
ForwardDiff.Dual{ForwardDiff.Tag{getfield(Main, Symbol("##73#74")),Float64},Float64,1}
1-element Array{Float64,1}:
 49.0
5

That’s what I have been trying. The problem was that I allocated the array according to a type of init and not meta.

In my case, the init is Array{Tuple{4, Float64}} and meta are actually “hidden” in an object (model). So now I am using T = typeof(model.metaparameter1) to allocate Array{Tuple{4, T}}. It looks like it is working :] Hopefully this does not break anything.

6

Perfect! Yes, I think that might be a problem. Note that you can pass the parametric type T in the function signature itself :slight_smile: