Does svd() not support forwarddiff?

julia> using LinearAlgebra

julia> using ForwardDiff

julia> function f(x)
       a=[1 2;3 4]
       a=a.*x
       _,s,_=svd(a)
       return sum(s)
       end
f (generic function with 1 method)

julia> ForwardDiff.derivative(f,2)
ERROR: MethodError: no method matching svd!(::Matrix{ForwardDiff.Dual{ForwardDiff.Tag{typeof(f), Int64}, Float64, 1}}; full=false, alg=LinearAlgebra.DivideAndConquer())
Closest candidates are:
  svd!(::LinearAlgebra.AbstractTriangular; kwargs...) at ~/julia-1.7.2/share/julia/stdlib/v1.7/LinearAlgebra/src/triangular.jl:2565
  svd!(::StridedMatrix{T}; full, alg) where T<:Union{Float32, Float64, ComplexF32, ComplexF64} at ~/julia-1.7.2/share/julia/stdlib/v1.7/LinearAlgebra/src/svd.jl:97
  svd!(::StridedVector{T}; full, alg) where T<:Union{Float32, Float64, ComplexF32, ComplexF64} at ~/julia-1.7.2/share/julia/stdlib/v1.7/LinearAlgebra/src/svd.jl:106
  ...
Stacktrace:
 [1] svd(A::Matrix{ForwardDiff.Dual{ForwardDiff.Tag{typeof(f), Int64}, Int64, 1}}; full::Bool, alg::LinearAlgebra.DivideAndConquer)
   @ LinearAlgebra ~/julia-1.7.2/share/julia/stdlib/v1.7/LinearAlgebra/src/svd.jl:176
 [2] svd(A::Matrix{ForwardDiff.Dual{ForwardDiff.Tag{typeof(f), Int64}, Int64, 1}})
   @ LinearAlgebra ~/julia-1.7.2/share/julia/stdlib/v1.7/LinearAlgebra/src/svd.jl:176
 [3] f(x::ForwardDiff.Dual{ForwardDiff.Tag{typeof(f), Int64}, Int64, 1})
   @ Main ./REPL[3]:4
 [4] derivative(f::typeof(f), x::Int64)
   @ ForwardDiff ~/.julia/packages/ForwardDiff/wAaVJ/src/derivative.jl:14
 [5] top-level scope
   @ REPL[4]:1

I want to use Forwarddiff.jl to take the derivative of my function, but it seems that it does not support svd. How can I solve this problem? I don’t want to use Zygote.gradient(), which consumes more time and space memory than forward differential

Maybe this is related Compatibility with Base linear algebra functions · Issue #111 · JuliaDiff/ForwardDiff.jl · GitHub for why it doesn’t seem to be supported.

In terms of other solutions, maybe FiniteDiff.jl’s finite_difference_gradient!/finite_difference_derivative meets your needs.

The discussion @legola18 linked mentions GenericLinearAlgebra.jl, a project to implement linear algebra functions in pure Julia with support for generic arguments. The svd function from this package seems to work just fine with ForwardDiff:

using GenericLinearAlgebra: svd
using ForwardDiff

function f(x)
    a = [1 2; 3 4]
    a = a .* x
    _, s, _ = svd(a)
    return sum(s)
end

ForwardDiff.derivative(f,2)
# 5.830951894845303
1 Like

@mohamed82008 if we make ImplicitDifferentiation.jl compatible with ForwardDiff.jl, how much work would it be to do the same with DifferentiableFactorizations.jl?

It would be automatically compatible with FD if ImplicitDifferentiation is.

OK so all we need is to specify the calling behavior of ImplicitFunction on x::AbstractArray{<:Dual}?
That would make ForwardDiff.jl a dependency though

Yes