Does svd() not support forwarddiff?

General Usage
1 
julia> using LinearAlgebra

julia> using ForwardDiff

julia> function f(x)
       a=[1 2;3 4]
       a=a.*x
       _,s,_=svd(a)
       return sum(s)
       end
f (generic function with 1 method)

julia> ForwardDiff.derivative(f,2)
ERROR: MethodError: no method matching svd!(::Matrix{ForwardDiff.Dual{ForwardDiff.Tag{typeof(f), Int64}, Float64, 1}}; full=false, alg=LinearAlgebra.DivideAndConquer())
Closest candidates are:
  svd!(::LinearAlgebra.AbstractTriangular; kwargs...) at ~/julia-1.7.2/share/julia/stdlib/v1.7/LinearAlgebra/src/triangular.jl:2565
  svd!(::StridedMatrix{T}; full, alg) where T<:Union{Float32, Float64, ComplexF32, ComplexF64} at ~/julia-1.7.2/share/julia/stdlib/v1.7/LinearAlgebra/src/svd.jl:97
  svd!(::StridedVector{T}; full, alg) where T<:Union{Float32, Float64, ComplexF32, ComplexF64} at ~/julia-1.7.2/share/julia/stdlib/v1.7/LinearAlgebra/src/svd.jl:106
  ...
Stacktrace:
 [1] svd(A::Matrix{ForwardDiff.Dual{ForwardDiff.Tag{typeof(f), Int64}, Int64, 1}}; full::Bool, alg::LinearAlgebra.DivideAndConquer)
   @ LinearAlgebra ~/julia-1.7.2/share/julia/stdlib/v1.7/LinearAlgebra/src/svd.jl:176
 [2] svd(A::Matrix{ForwardDiff.Dual{ForwardDiff.Tag{typeof(f), Int64}, Int64, 1}})
   @ LinearAlgebra ~/julia-1.7.2/share/julia/stdlib/v1.7/LinearAlgebra/src/svd.jl:176
 [3] f(x::ForwardDiff.Dual{ForwardDiff.Tag{typeof(f), Int64}, Int64, 1})
   @ Main ./REPL[3]:4
 [4] derivative(f::typeof(f), x::Int64)
   @ ForwardDiff ~/.julia/packages/ForwardDiff/wAaVJ/src/derivative.jl:14
 [5] top-level scope
   @ REPL[4]:1

I want to use Forwarddiff.jl to take the derivative of my function, but it seems that it does not support svd. How can I solve this problem? I don’t want to use Zygote.gradient(), which consumes more time and space memory than forward differential

2

Maybe this is related Compatibility with Base linear algebra functions · Issue #111 · JuliaDiff/ForwardDiff.jl · GitHub for why it doesn’t seem to be supported.

In terms of other solutions, maybe FiniteDiff.jl’s finite_difference_gradient!/finite_difference_derivative meets your needs.

3

The discussion @DanielVandH linked mentions GenericLinearAlgebra.jl, a project to implement linear algebra functions in pure Julia with support for generic arguments. The svd function from this package seems to work just fine with ForwardDiff:

using GenericLinearAlgebra: svd
using ForwardDiff

function f(x)
    a = [1 2; 3 4]
    a = a .* x
    _, s, _ = svd(a)
    return sum(s)
end

ForwardDiff.derivative(f,2)
# 5.830951894845303
1 Like
4

@mohamed82008 if we make ImplicitDifferentiation.jl compatible with ForwardDiff.jl, how much work would it be to do the same with DifferentiableFactorizations.jl?

5

It would be automatically compatible with FD if ImplicitDifferentiation is.

6

OK so all we need is to specify the calling behavior of ImplicitFunction on x::AbstractArray{<:Dual}?
That would make ForwardDiff.jl a dependency though

7

Yes