]test DiffEqGPU errors with " unsupported call to the Julia runtime"

Specific Domains GPU
1

hi, i’m trying to experiment with ODE solving on GPUs but can’t seem to get the packages running properly. this is on a machine with CUDA 9.0 installed and 4 Tesla K80 GPUs.

i can install DiffEqGPU but the test fails as indicated above. i did not find minimum requirements for the GPU compute capabilities or the CUDA version so i am assuming this should be ok.

a snippet from the error output:

InvalidIRError: [...] Reason: unsupported call to the Julia runtime (call to jl_f_tuple)

did i do something wrong? sorry if this is off-topic for this forum!

2

this is on julia 1.2 and today’s versions of all required packages. i also get the following error with ]test CuArrays:

symbol lookup error: /opt/cuda-9.0/lib64/libcusolver.so: undefined symbol: omp_get_max_threads

3

Can we see the full backtrace? We need to see the entire error message to know what’s going on. This could possibly be fixed already on GPUArrays.jl master. Try doing

add GPUArrays#master

and see if it works.

4

sure, will try.

ok, so i did add GPUArrays#master which completed without errors, in a fresh REPL. i then did precompile and then test DiffEqGPU. this gave the following output:

(v1.2) pkg> test DiffEqGPU
   Testing DiffEqGPU
 Resolving package versions...
    Status `/tmp/jl_A7B6sE/Manifest.toml`
  [621f4979] AbstractFFTs v0.4.1
  [79e6a3ab] Adapt v1.0.0
  [ec485272] ArnoldiMethod v0.0.4
  [4fba245c] ArrayInterface v1.2.1
  [aae01518] BandedMatrices v0.10.1
  [9e28174c] BinDeps v0.8.10
  [b99e7846] BinaryProvider v0.5.6
  [8e7c35d0] BlockArrays v0.9.1
  [ffab5731] BlockBandedMatrices v0.4.6
  [fa961155] CEnum v0.2.0
  [00ebfdb7] CSTParser v0.6.2
  [3895d2a7] CUDAapi v1.1.0
  [c5f51814] CUDAdrv v3.1.0
  [be33ccc6] CUDAnative v2.3.1
  [49dc2e85] Calculus v0.5.0
  [7057c7e9] Cassette v0.2.6
  [bbf7d656] CommonSubexpressions v0.2.0
  [34da2185] Compat v2.1.0
  [8f4d0f93] Conda v1.3.0
  [a8cc5b0e] Crayons v4.0.0
  [3a865a2d] CuArrays v1.2.1
  [864edb3b] DataStructures v0.17.0
  [2b5f629d] DiffEqBase v6.2.3
  [01453d9d] DiffEqDiffTools v1.3.0
  [071ae1c0] DiffEqGPU v0.1.0
  [163ba53b] DiffResults v0.0.4
  [b552c78f] DiffRules v0.0.10
  [b4f34e82] Distances v0.8.2
  [ffbed154] DocStringExtensions v0.8.0
  [d4d017d3] ExponentialUtilities v1.5.1
  [7a1cc6ca] FFTW v1.0.0
  [1a297f60] FillArrays v0.6.4
  [f6369f11] ForwardDiff v0.10.3
  [069b7b12] FunctionWrappers v1.0.0
  [0c68f7d7] GPUArrays v1.0.3 #master (https://github.com/JuliaGPU/GPUArrays.jl.git)
  [ba82f77b] GPUifyLoops v0.2.8
  [01680d73] GenericSVD v0.2.1
  [d25df0c9] Inflate v0.1.1
  [42fd0dbc] IterativeSolvers v0.8.1
  [82899510] IteratorInterfaceExtensions v1.0.0
  [682c06a0] JSON v0.21.0
  [929cbde3] LLVM v1.3.0
  [5078a376] LazyArrays v0.10.0
  [093fc24a] LightGraphs v1.3.0
  [d3d80556] LineSearches v7.0.1
  [1914dd2f] MacroTools v0.5.1
  [a3b82374] MatrixFactorizations v0.1.0
  [46d2c3a1] MuladdMacro v0.2.1
  [d41bc354] NLSolversBase v7.4.1
  [2774e3e8] NLsolve v4.1.0
  [872c559c] NNlib v0.6.0
  [77ba4419] NaNMath v0.3.2
  [bac558e1] OrderedCollections v1.1.0
  [1dea7af3] OrdinaryDiffEq v5.17.0
  [d96e819e] Parameters v0.12.0
  [69de0a69] Parsers v0.3.7
  [3cdcf5f2] RecipesBase v0.7.0
  [731186ca] RecursiveArrayTools v1.0.2
  [f2c3362d] RecursiveFactorization v0.1.0
  [189a3867] Reexport v0.2.0
  [ae029012] Requires v0.5.2
  [f2b01f46] Roots v0.8.3
  [699a6c99] SimpleTraits v0.9.0
  [47a9eef4] SparseDiffTools v0.9.0
  [276daf66] SpecialFunctions v0.8.0
  [90137ffa] StaticArrays v0.11.0
  [3783bdb8] TableTraits v1.0.0
  [a759f4b9] TimerOutputs v0.5.0
  [0796e94c] Tokenize v0.5.6
  [a2a6695c] TreeViews v0.3.0
  [30578b45] URIParser v0.4.0
  [81def892] VersionParsing v1.1.3
  [19fa3120] VertexSafeGraphs v0.1.0
  [2a0f44e3] Base64  [`@stdlib/Base64`]
  [ade2ca70] Dates  [`@stdlib/Dates`]
  [8bb1440f] DelimitedFiles  [`@stdlib/DelimitedFiles`]
  [8ba89e20] Distributed  [`@stdlib/Distributed`]
  [b77e0a4c] InteractiveUtils  [`@stdlib/InteractiveUtils`]
  [76f85450] LibGit2  [`@stdlib/LibGit2`]
  [8f399da3] Libdl  [`@stdlib/Libdl`]
  [37e2e46d] LinearAlgebra  [`@stdlib/LinearAlgebra`]
  [56ddb016] Logging  [`@stdlib/Logging`]
  [d6f4376e] Markdown  [`@stdlib/Markdown`]
  [a63ad114] Mmap  [`@stdlib/Mmap`]
  [44cfe95a] Pkg  [`@stdlib/Pkg`]
  [de0858da] Printf  [`@stdlib/Printf`]
  [9abbd945] Profile  [`@stdlib/Profile`]
  [3fa0cd96] REPL  [`@stdlib/REPL`]
  [9a3f8284] Random  [`@stdlib/Random`]
  [ea8e919c] SHA  [`@stdlib/SHA`]
  [9e88b42a] Serialization  [`@stdlib/Serialization`]
  [1a1011a3] SharedArrays  [`@stdlib/SharedArrays`]
  [6462fe0b] Sockets  [`@stdlib/Sockets`]
  [2f01184e] SparseArrays  [`@stdlib/SparseArrays`]
  [10745b16] Statistics  [`@stdlib/Statistics`]
  [4607b0f0] SuiteSparse  [`@stdlib/SuiteSparse`]
  [8dfed614] Test  [`@stdlib/Test`]
  [cf7118a7] UUIDs  [`@stdlib/UUIDs`]
  [4ec0a83e] Unicode  [`@stdlib/Unicode`]
WARNING: Method definition overdub(Cassette.Context{N, M, T, P, B, H} where H<:Union{Cassette.DisableHooks, Nothing} where B<:Union{Nothing, Base.IdDict{Module, Base.Dict{Symbol, Cassette.BindingMeta}}} where P<:Cassette.AbstractPass where T<:Union{Nothing, Cassette.Tag{N, X, E} where E where X where N<:Cassette.AbstractContextName} where M where N<:Cassette.AbstractContextName, Any...) in module Cassette at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/Cassette/YCOeN/src/overdub.jl:524 overwritten in module GPUifyLoops at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/Cassette/YCOeN/src/overdub.jl:524.
  ** incremental compilation may be fatally broken for this module **

WARNING: Method definition recurse(Cassette.Context{N, M, T, P, B, H} where H<:Union{Cassette.DisableHooks, Nothing} where B<:Union{Nothing, Base.IdDict{Module, Base.Dict{Symbol, Cassette.BindingMeta}}} where P<:Cassette.AbstractPass where T<:Union{Nothing, Cassette.Tag{N, X, E} where E where X where N<:Cassette.AbstractContextName} where M where N<:Cassette.AbstractContextName, Any...) in module Cassette at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/Cassette/YCOeN/src/overdub.jl:537 overwritten in module GPUifyLoops at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/Cassette/YCOeN/src/overdub.jl:537.
  ** incremental compilation may be fatally broken for this module **

[ Info: Building the CUDAnative run-time library for your sm_37 device, this might take a while...
ERROR: LoadError: InvalidIRError: compiling gpu_kernel(Cassette.Context{nametype(Ctx),Nothing,Nothing,getfield(GPUifyLoops, Symbol("##PassType#371")),Nothing,Cassette.DisableHooks}, typeof(DiffEqGPU.gpu_kernel), ODEFunction{true,typeof(lorenz),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing}, CUDAnative.CuDeviceArray{Float32,2,CUDAnative.AS.Global}, CUDAnative.CuDeviceArray{Float32,2,CUDAnative.AS.Global}, CUDAnative.CuDeviceArray{DiffEqBase.NullParameters,2,CUDAnative.AS.Global}, Float32) resulted in invalid LLVM IR
Reason: unsupported call to the Julia runtime (call to jl_f_tuple)
Stacktrace:
 [1] overdub at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/Cassette/YCOeN/src/overdub.jl:524
 [2] multiple call sites at unknown:0
Reason: unsupported call to the Julia runtime (call to jl_f_getfield)
Stacktrace:
 [1] overdub at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/Cassette/YCOeN/src/overdub.jl:524
 [2] multiple call sites at unknown:0
Stacktrace:
 [1] check_ir(::CUDAnative.CompilerJob, ::LLVM.Module) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/CUDAnative/UWBIY/src/compiler/validation.jl:114
 [2] macro expansion at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/CUDAnative/UWBIY/src/compiler/driver.jl:188 [inlined]
 [3] macro expansion at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/TimerOutputs/7zSea/src/TimerOutput.jl:216 [inlined]
 [4] #codegen#130(::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::typeof(CUDAnative.codegen), ::Symbol, ::CUDAnative.CompilerJob) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/CUDAnative/UWBIY/src/compiler/driver.jl:186
 [5] #codegen at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/CUDAnative/UWBIY/src/compiler/driver.jl:0 [inlined]
 [6] #compile#129(::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::typeof(CUDAnative.compile), ::Symbol, ::CUDAnative.CompilerJob) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/CUDAnative/UWBIY/src/compiler/driver.jl:47
 [7] #compile at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/CUDAnative/UWBIY/src/compiler/common.jl:0 [inlined]
 [8] #compile#128 at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/CUDAnative/UWBIY/src/compiler/driver.jl:28 [inlined]
 [9] #compile at ./none:0 [inlined] (repeats 2 times)
 [10] macro expansion at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/CUDAnative/UWBIY/src/execution.jl:389 [inlined]
 [11] #cufunction#170(::String, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(CUDAnative.cufunction), ::typeof(Cassette.overdub), ::Type{Tuple{Cassette.Context{nametype(Ctx),Nothing,Nothing,getfield(GPUifyLoops, Symbol("##PassType#371")),Nothing,Cassette.DisableHooks},typeof(DiffEqGPU.gpu_kernel),ODEFunction{true,typeof(lorenz),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},CUDAnative.CuDeviceArray{Float32,2,CUDAnative.AS.Global},CUDAnative.CuDeviceArray{Float32,2,CUDAnative.AS.Global},CUDAnative.CuDeviceArray{DiffEqBase.NullParameters,2,CUDAnative.AS.Global},Float32}}) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/CUDAnative/UWBIY/src/execution.jl:357
 [12] (::getfield(CUDAnative, Symbol("#kw##cufunction")))(::NamedTuple{(:name,),Tuple{String}}, ::typeof(CUDAnative.cufunction), ::Function, ::Type) at ./none:0
 [13] macro expansion at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/GPUifyLoops/mjszO/src/GPUifyLoops.jl:125 [inlined]
 [14] macro expansion at ./gcutils.jl:87 [inlined]
 [15] #launch#50(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(GPUifyLoops.launch), ::GPUifyLoops.CUDA, ::typeof(DiffEqGPU.gpu_kernel), ::Function, ::Vararg{Any,N} where N) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/GPUifyLoops/mjszO/src/GPUifyLoops.jl:121
 [16] launch at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/GPUifyLoops/mjszO/src/GPUifyLoops.jl:119 [inlined]
 [17] macro expansion at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/GPUifyLoops/mjszO/src/GPUifyLoops.jl:54 [inlined]
 [18] #12 at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqGPU/QB1WC/src/DiffEqGPU.jl:61 [inlined]
 [19] ODEFunction at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqBase/8uyX3/src/diffeqfunction.jl:230 [inlined]
 [20] initialize!(::OrdinaryDiffEq.ODEIntegrator{Tsit5,true,CuArray{Float32,2},Float32,CuArray{DiffEqBase.NullParameters,2},Float32,Float32,Float32,Array{CuArray{Float32,2},1},ODESolution{Float32,3,Array{CuArray{Float32,2},1},Nothing,Nothing,Array{Float32,1},Array{Array{CuArray{Float32,2},1},1},ODEProblem{CuArray{Float32,2},Tuple{Float32,Float32},true,CuArray{DiffEqBase.NullParameters,2},ODEFunction{true,getfield(DiffEqGPU, Symbol("##12#22")){ODEFunction{true,typeof(lorenz),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},Tsit5,OrdinaryDiffEq.InterpolationData{ODEFunction{true,getfield(DiffEqGPU, Symbol("##12#22")){ODEFunction{true,typeof(lorenz),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Array{CuArray{Float32,2},1},Array{Float32,1},Array{Array{CuArray{Float32,2},1},1},OrdinaryDiffEq.Tsit5Cache{CuArray{Float32,2},CuArray{Float32,2},CuArray{Float32,2},OrdinaryDiffEq.Tsit5ConstantCache{Float32,Float32}}},DiffEqBase.DEStats},ODEFunction{true,getfield(DiffEqGPU, Symbol("##12#22")){ODEFunction{true,typeof(lorenz),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},OrdinaryDiffEq.Tsit5Cache{CuArray{Float32,2},CuArray{Float32,2},CuArray{Float32,2},OrdinaryDiffEq.Tsit5ConstantCache{Float32,Float32}},OrdinaryDiffEq.DEOptions{Float32,Float32,Float32,Float32,typeof(DiffEqBase.ODE_DEFAULT_NORM),typeof(LinearAlgebra.opnorm),CallbackSet{Tuple{},Tuple{}},typeof(DiffEqBase.ODE_DEFAULT_ISOUTOFDOMAIN),typeof(DiffEqBase.ODE_DEFAULT_PROG_MESSAGE),typeof(DiffEqBase.ODE_DEFAULT_UNSTABLE_CHECK),DataStructures.BinaryHeap{Float32,DataStructures.LessThan},DataStructures.BinaryHeap{Float32,DataStructures.LessThan},Nothing,Nothing,Int64,Array{Float32,1},Float32,Array{Float32,1}},CuArray{Float32,2},Float32,Nothing}, ::OrdinaryDiffEq.Tsit5Cache{CuArray{Float32,2},CuArray{Float32,2},CuArray{Float32,2},OrdinaryDiffEq.Tsit5ConstantCache{Float32,Float32}}) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/OrdinaryDiffEq/tQd6p/src/perform_step/low_order_rk_perform_step.jl:623
 [21] #__init#335(::Float32, ::Array{Float32,1}, ::Array{Float32,1}, ::Nothing, ::Bool, ::Bool, ::Bool, ::Bool, ::Nothing, ::Bool, ::Bool, ::Float32, ::Float32, ::Float32, ::Bool, ::Bool, ::Rational{Int64}, ::Nothing, ::Nothing, ::Rational{Int64}, ::Int64, ::Int64, ::Int64, ::Rational{Int64}, ::Bool, ::Int64, ::Nothing, ::Nothing, ::Int64, ::typeof(DiffEqBase.ODE_DEFAULT_NORM), ::typeof(LinearAlgebra.opnorm), ::typeof(DiffEqBase.ODE_DEFAULT_ISOUTOFDOMAIN), ::typeof(DiffEqBase.ODE_DEFAULT_UNSTABLE_CHECK), ::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::Int64, ::String, ::typeof(DiffEqBase.ODE_DEFAULT_PROG_MESSAGE), ::Nothing, ::Bool, ::Bool, ::Bool, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(DiffEqBase.__init), ::ODEProblem{CuArray{Float32,2},Tuple{Float32,Float32},true,CuArray{DiffEqBase.NullParameters,2},ODEFunction{true,getfield(DiffEqGPU, Symbol("##12#22")){ODEFunction{true,typeof(lorenz),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem}, ::Tsit5, ::Array{CuArray{Float32,2},1}, ::Array{Float32,1}, ::Array{Any,1}, ::Type{Val{true}}) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/OrdinaryDiffEq/tQd6p/src/solve.jl:352
 [22] (::getfield(DiffEqBase, Symbol("#kw##__init")))(::NamedTuple{(:saveat,),Tuple{Float32}}, ::typeof(DiffEqBase.__init), ::ODEProblem{CuArray{Float32,2},Tuple{Float32,Float32},true,CuArray{DiffEqBase.NullParameters,2},ODEFunction{true,getfield(DiffEqGPU, Symbol("##12#22")){ODEFunction{true,typeof(lorenz),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem}, ::Tsit5, ::Array{CuArray{Float32,2},1}, ::Array{Float32,1}, ::Array{Any,1}, ::Type{Val{true}}) at ./none:0 (repeats 4 times)
 [23] #__solve#334 at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/OrdinaryDiffEq/tQd6p/src/solve.jl:4 [inlined]
 [24] #__solve at ./none:0 [inlined]
 [25] #solve_call#425(::Base.Iterators.Pairs{Symbol,Float32,Tuple{Symbol},NamedTuple{(:saveat,),Tuple{Float32}}}, ::typeof(DiffEqBase.solve_call), ::ODEProblem{CuArray{Float32,2},Tuple{Float32,Float32},true,CuArray{DiffEqBase.NullParameters,2},ODEFunction{true,getfield(DiffEqGPU, Symbol("##12#22")){ODEFunction{true,typeof(lorenz),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem}, ::Tsit5) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqBase/8uyX3/src/solve.jl:40
 [26] #solve_call at ./none:0 [inlined]
 [27] #solve#426 at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqBase/8uyX3/src/solve.jl:57 [inlined]
 [28] (::getfield(DiffEqBase, Symbol("#kw##solve")))(::NamedTuple{(:saveat,),Tuple{Float32}}, ::typeof(solve), ::ODEProblem{CuArray{Float32,2},Tuple{Float32,Float32},true,CuArray{DiffEqBase.NullParameters,2},ODEFunction{true,getfield(DiffEqGPU, Symbol("##12#22")){ODEFunction{true,typeof(lorenz),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem}, ::Tsit5) at ./none:0
 [29] #batch_solve#5(::Base.Iterators.Pairs{Symbol,Float32,Tuple{Symbol},NamedTuple{(:saveat,),Tuple{Float32}}}, ::typeof(DiffEqGPU.batch_solve), ::EnsembleProblem{ODEProblem{Array{Float32,1},Tuple{Float32,Float32},true,DiffEqBase.NullParameters,ODEFunction{true,typeof(lorenz),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},getfield(DiffEqBase, Symbol("##330#336")),getfield(DiffEqBase, Symbol("##329#335")),getfield(DiffEqBase, Symbol("##331#337")),Array{Any,1}}, ::Tsit5, ::EnsembleGPUArray, ::UnitRange{Int64}) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqGPU/QB1WC/src/DiffEqGPU.jl:66
 [30] #batch_solve at ./none:0 [inlined]
 [31] (::getfield(DiffEqGPU, Symbol("##3#4")){Int64,Int64,Base.Iterators.Pairs{Symbol,Float32,Tuple{Symbol},NamedTuple{(:saveat,),Tuple{Float32}}},EnsembleProblem{ODEProblem{Array{Float32,1},Tuple{Float32,Float32},true,DiffEqBase.NullParameters,ODEFunction{true,typeof(lorenz),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},getfield(DiffEqBase, Symbol("##330#336")),getfield(DiffEqBase, Symbol("##329#335")),getfield(DiffEqBase, Symbol("##331#337")),Array{Any,1}},Tsit5,EnsembleGPUArray})(::Int64) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqGPU/QB1WC/src/DiffEqGPU.jl:37
 [32] iterate at ./generator.jl:47 [inlined]
 [33] _collect(::UnitRange{Int64}, ::Base.Generator{UnitRange{Int64},getfield(DiffEqGPU, Symbol("##3#4")){Int64,Int64,Base.Iterators.Pairs{Symbol,Float32,Tuple{Symbol},NamedTuple{(:saveat,),Tuple{Float32}}},EnsembleProblem{ODEProblem{Array{Float32,1},Tuple{Float32,Float32},true,DiffEqBase.NullParameters,ODEFunction{true,typeof(lorenz),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},getfield(DiffEqBase, Symbol("##330#336")),getfield(DiffEqBase, Symbol("##329#335")),getfield(DiffEqBase, Symbol("##331#337")),Array{Any,1}},Tsit5,EnsembleGPUArray}}, ::Base.EltypeUnknown, ::Base.HasShape{1}) at ./array.jl:619
 [34] collect_similar(::UnitRange{Int64}, ::Base.Generator{UnitRange{Int64},getfield(DiffEqGPU, Symbol("##3#4")){Int64,Int64,Base.Iterators.Pairs{Symbol,Float32,Tuple{Symbol},NamedTuple{(:saveat,),Tuple{Float32}}},EnsembleProblem{ODEProblem{Array{Float32,1},Tuple{Float32,Float32},true,DiffEqBase.NullParameters,ODEFunction{true,typeof(lorenz),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},getfield(DiffEqBase, Symbol("##330#336")),getfield(DiffEqBase, Symbol("##329#335")),getfield(DiffEqBase, Symbol("##331#337")),Array{Any,1}},Tsit5,EnsembleGPUArray}}) at ./array.jl:548
 [35] map(::Function, ::UnitRange{Int64}) at ./abstractarray.jl:2073
 [36] macro expansion at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqGPU/QB1WC/src/DiffEqGPU.jl:31 [inlined]
 [37] macro expansion at ./util.jl:213 [inlined]
 [38] #__solve#2(::Int64, ::Int64, ::Base.Iterators.Pairs{Symbol,Float32,Tuple{Symbol},NamedTuple{(:saveat,),Tuple{Float32}}}, ::typeof(DiffEqBase.__solve), ::EnsembleProblem{ODEProblem{Array{Float32,1},Tuple{Float32,Float32},true,DiffEqBase.NullParameters,ODEFunction{true,typeof(lorenz),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},getfield(DiffEqBase, Symbol("##330#336")),getfield(DiffEqBase, Symbol("##329#335")),getfield(DiffEqBase, Symbol("##331#337")),Array{Any,1}}, ::Tsit5, ::EnsembleGPUArray) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqGPU/QB1WC/src/DiffEqGPU.jl:30
 [39] #__solve at ./none:0 [inlined]
 [40] #solve#427 at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqBase/8uyX3/src/solve.jl:64 [inlined]
 [41] (::getfield(DiffEqBase, Symbol("#kw##solve")))(::NamedTuple{(:trajectories, :saveat),Tuple{Int64,Float32}}, ::typeof(solve), ::EnsembleProblem{ODEProblem{Array{Float32,1},Tuple{Float32,Float32},true,DiffEqBase.NullParameters,ODEFunction{true,typeof(lorenz),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},getfield(DiffEqBase, Symbol("##330#336")),getfield(DiffEqBase, Symbol("##329#335")),getfield(DiffEqBase, Symbol("##331#337")),Array{Any,1}}, ::Tsit5, ::EnsembleGPUArray) at ./none:0
 [42] top-level scope at util.jl:156
 [43] include at ./boot.jl:328 [inlined]
 [44] include_relative(::Module, ::String) at ./loading.jl:1094
 [45] include(::Module, ::String) at ./Base.jl:31
 [46] include(::String) at ./client.jl:431
 [47] top-level scope at none:5
in expression starting at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqGPU/QB1WC/test/runtests.jl:20
ERROR: Package DiffEqGPU errored during testing
5

fwiw, i also get a warning message when using DiffEqGPU which complains about incremental compilation not working:

┌ Info: Recompiling stale cache file /home/bq_nbecker/isi/julia_pkg/GPUTest/compiled/v1.2/DiffEqGPU/8rwkT.ji for DiffEqGPU [071ae1c0-96b5-11e9-1965-c90190d839ea]
└ @ Base loading.jl:1240
WARNING: Method definition overdub(Cassette.Context{N, M, T, P, B, H} where H<:Union{Cassette.DisableHooks, Nothing} where B<:Union{Nothing, Base.IdDict{Module, Base.Dict{Symbol, Cassette.BindingMeta}}} where P<:Cassette.AbstractPass where T<:Union{Nothing, Cassette.Tag{N, X, E} where E where X where N<:Cassette.AbstractContextName} where M where N<:Cassette.AbstractContextName, Any...) in module Cassette at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/Cassette/YCOeN/src/overdub.jl:524 overwritten in module GPUifyLoops at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/Cassette/YCOeN/src/overdub.jl:524.
  ** incremental compilation may be fatally broken for this module **
6

another bit: the gpu has cuda compute capability 3.7

7

I also tried to copy-and-paste the code in DiffEqGPU.jl/test/runtests.jl to a REPL. Except for the fact that i have to manually install OrdinaryDiffEq, and i get the incremental compilation warnings above, this works. i can successfully run the full file, and indeed the gpu ensembles are faster than EnsembleCPUArray. (although the timings are extremely variable: 2s-20s for the GPU ensembles from run to run, after several runs(

8

Glad to see it’s working. I am not sure what that error you’re getting is, it seems hard to track it down if we can’t get a way to reproduce it.

See what happens if you GC before the runs.

9

yes, thanks for looking into it. i think the gpu is working in principle (although the test failure persists). i tried running Base.GC.gc() before the @time ... tests and this somewhat influenced the timings. but the large increase in timing to over 30s appears only when i run the test from within a (remote) jupyter notebook. this is likely an unrelated issue which could have to do with the disk quota being maxed out on that computer.

i have one more related question: how much of the automatic GPU parallelism in DiffEqGPU is currently supposed to be working?

  1. can i define an arbitrary scalar ODE problem and then solve a parameter-ensemble of the problem on the GPU, or do restrictions apply (except for handling events which was mentioned in your blog post)

  2. if i define a small system of ODEs to be run as a parameter ensemble, what array type to i have to use for u, u0 and p? (i see the test uses Float32 arrays, not CuArrays or some such)

10

A little bit, but not much.

For now it needs to be non-stiff, but events and stiff ODEs are being fixed hopefully by the end of the week: Jac by ChrisRackauckas · Pull Request #9 · SciML/DiffEqGPU.jl · GitHub

Float32 arrays. It takes your single problem and generates a GPU-based problem from it, so it’s invisible on your end.

11

i have been testing further using example 3 from http://docs.juliadiffeq.org/latest/tutorials/ode_example.html . There the RHS of the ode is just A * u. I made A and u into Float32 arrays and made an ensemble problem by adding small amounts of noise on the u0 initial condition.

A  = Float32[
    1. 0  0 -5
     4 -2  4 -3
     -4  0  0  1
      5 -2  2  3]
u0 = Array{Float32}(rand(4,2))
tspan = (0.0,1.0)
f(u,p,t) = A*u
prob = ODEProblem(f,u0,tspan)
function prob_func(prob, i, repeat)
    prob.u0 .+= 0.01 * Array{Float32}(randn(4,2))
    prob
end
saveat = 0.01
trajs = 123
ensprobfull=EnsembleProblem(prob, prob_func=prob_func)
simfull = solve(ensprobfull,Tsit5(),EnsembleDistributed(), trajectories=trajs, saveat=saveat)
# now try on gpu
using DiffEqGPU
simgpu = solve(ensprobfull, Tsit5(), EnsembleGPUArray(), trajectories=trajs, saveat=saveat)

i think this cannot be terribly efficient since it’s not in-place modification of the du vector – however it does not run at all on the GPU for me, i get

InvalidIRError: compiling gpu_kernel(Cassette.Context{nametype(Ctx),Nothing,Nothing,getfield(GPUifyLoops, Symbol("##PassType#371")),Nothing,Cassette.DisableHooks}, typeof(DiffEqGPU.gpu_kernel), ODEFunction{false,typeof(f),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing}, CUDAnative.CuDeviceArray{Float32,2,CUDAnative.AS.Global}, CUDAnative.CuDeviceArray{Float32,2,CUDAnative.AS.Global}, CUDAnative.CuDeviceArray{DiffEqBase.NullParameters,2,CUDAnative.AS.Global}, Float64) resulted in invalid LLVM IR
Reason: unsupported dynamic function invocation (call to f)

without specifying a solver algorithm it does run – at the same speed as the CPU version.

is this expected? (it wasn’t for me, i’m just learning julia diff eq)

12

Those two facts are the same thing. The GPU code must be non-allocating: it’s a current limitation. The fact that your code is allocating the arrays means it cannot auto-GPU.

13

ah ok, i must have missed that limitation. thanks

14

i’ve now modified the problem to be in-place, which i assumed should work:

function fip(du,u,p,t) 
    du .= A*u
end
probip = ODEProblem(fip,u0,tspan)
ensprobip=EnsembleProblem(probip, prob_func=prob_func)
using DiffEqGPU
simgpu = solve(ensprobip, Tsit5(), EnsembleGPUArray(), trajectories=trajs, saveat=saveat)

the problem probip is reported as being in-place.

for the gpu ensemble problem, i still get the same error as above: [...]invalid LLVM IR Reason: unsupported dynamic function invocation (call to overdub). without Tsit5 it does run, as before, but slower than on CPU with EnsembleDistributed – does this mean that it quietly falls back to CPU computation? i’m confused.

when i @time sol=solve(probip, Tsit5(), saveat=0.01) this reports 116MiB of allocations. so, is my code allocating and therefore does not run on gpu? (if that’s the case, a more specific error message from DiffEqGPU would be helpful.)

i don’t see where allocation happens and how to avoid it?

15

This allocates. You’d need to do mul!(du,A,u)

16

ah that makes sense. it’s not always obvious how clever a language is trying to be… thanks again!

17

still no luck. sorry for keeping you busy, but there might be a chance that this is not user error?

the following is a self contained minimal version which i tried to make non-allocating. also, the prob_func is not just a no-nothing function to exclude anything there (does that also have to be non-allocating?)

using LinearAlgebra, DifferentialEquations, DiffEqGPU
A  = Float32[
    1. 0  0 -5; 4 -2  4 -3; -4  0  0  1; 5 -2  2  3]
u0 = Array{Float32}(rand(4,2))
tspan = (0.0f0,1.0f0)
function fip(du,u,p,t) 
   mul!(du, A, u)
end
probip = ODEProblem(fip,u0,tspan)
function prob_func(prob, i, repeat)
    prob
end
ensprobip=EnsembleProblem(probip, prob_func=prob_func)
trajs = 123
simgpu = solve(ensprobip, Tsit5(), EnsembleGPUArray(), trajectories=trajs)

when i run it i get a similar stack trace:

InvalidIRError: compiling gpu_kernel(Cassette.Context{nametype(Ctx),Nothing,Nothing,getfield(GPUifyLoops, Symbol("##PassType#371")),Nothing,Cassette.DisableHooks}, typeof(DiffEqGPU.gpu_kernel), ODEFunction{true,typeof(fip),UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing}, CUDAnative.CuDeviceArray{Float32,2,CUDAnative.AS.Global}, CUDAnative.CuDeviceArray{Float32,2,CUDAnative.AS.Global}, CUDAnative.CuDeviceArray{DiffEqBase.NullParameters,2,CUDAnative.AS.Global}, Float32) resulted in invalid LLVM IR
Reason: unsupported dynamic function invocation (call to overdub)
Stacktrace:
 [1] fip at In[1]:10
 [2] ODEFunction at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqBase/8uyX3/src/diffeqfunction.jl:230
 [3] gpu_kernel at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqGPU/QB1WC/src/DiffEqGPU.jl:6
 [4] overdub at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/Cassette/YCOeN/src/overdub.jl:0

Stacktrace:
 [1] check_ir(::CUDAnative.CompilerJob, ::LLVM.Module) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/CUDAnative/UWBIY/src/compiler/validation.jl:114
 [2] macro expansion at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/CUDAnative/UWBIY/src/compiler/driver.jl:188 [inlined]
 [3] macro expansion at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/TimerOutputs/7zSea/src/TimerOutput.jl:216 [inlined]
 [4] #codegen#130(::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::typeof(CUDAnative.codegen), ::Symbol, ::CUDAnative.CompilerJob) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/CUDAnative/UWBIY/src/compiler/driver.jl:186
 [5] #codegen at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/CUDAnative/UWBIY/src/compiler/driver.jl:0 [inlined]
 [6] #compile#129(::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::typeof(CUDAnative.compile), ::Symbol, ::CUDAnative.CompilerJob) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/CUDAnative/UWBIY/src/compiler/driver.jl:47
 [7] #compile at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/CUDAnative/UWBIY/src/compiler/common.jl:0 [inlined]
 [8] #compile#128 at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/CUDAnative/UWBIY/src/compiler/driver.jl:28 [inlined]
 [9] #compile at ./none:0 [inlined] (repeats 2 times)
 [10] macro expansion at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/CUDAnative/UWBIY/src/execution.jl:389 [inlined]
 [11] #cufunction#170(::String, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(CUDAnative.cufunction), ::typeof(Cassette.overdub), ::Type{Tuple{Cassette.Context{nametype(Ctx),Nothing,Nothing,getfield(GPUifyLoops, Symbol("##PassType#371")),Nothing,Cassette.DisableHooks},typeof(DiffEqGPU.gpu_kernel),ODEFunction{true,typeof(fip),UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},CUDAnative.CuDeviceArray{Float32,2,CUDAnative.AS.Global},CUDAnative.CuDeviceArray{Float32,2,CUDAnative.AS.Global},CUDAnative.CuDeviceArray{DiffEqBase.NullParameters,2,CUDAnative.AS.Global},Float32}}) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/CUDAnative/UWBIY/src/execution.jl:357
 [12] (::getfield(CUDAnative, Symbol("#kw##cufunction")))(::NamedTuple{(:name,),Tuple{String}}, ::typeof(CUDAnative.cufunction), ::Function, ::Type) at ./none:0
 [13] #launch#50(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(GPUifyLoops.launch), ::GPUifyLoops.CUDA, ::typeof(DiffEqGPU.gpu_kernel), ::Function, ::Vararg{Any,N} where N) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/GPUifyLoops/mjszO/src/GPUifyLoops.jl:125
 [14] launch at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/GPUifyLoops/mjszO/src/GPUifyLoops.jl:119 [inlined]
 [15] macro expansion at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/GPUifyLoops/mjszO/src/GPUifyLoops.jl:54 [inlined]
 [16] #12 at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqGPU/QB1WC/src/DiffEqGPU.jl:61 [inlined]
 [17] ODEFunction at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqBase/8uyX3/src/diffeqfunction.jl:230 [inlined]
 [18] initialize!(::OrdinaryDiffEq.ODEIntegrator{Tsit5,true,CuArrays.CuArray{Float32,2},Float32,CuArrays.CuArray{DiffEqBase.NullParameters,2},Float32,Float32,Float32,Array{CuArrays.CuArray{Float32,2},1},ODESolution{Float32,3,Array{CuArrays.CuArray{Float32,2},1},Nothing,Nothing,Array{Float32,1},Array{Array{CuArrays.CuArray{Float32,2},1},1},ODEProblem{CuArrays.CuArray{Float32,2},Tuple{Float32,Float32},true,CuArrays.CuArray{DiffEqBase.NullParameters,2},ODEFunction{true,getfield(DiffEqGPU, Symbol("##12#22")){ODEFunction{true,typeof(fip),UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing}},UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},Tsit5,OrdinaryDiffEq.InterpolationData{ODEFunction{true,getfield(DiffEqGPU, Symbol("##12#22")){ODEFunction{true,typeof(fip),UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing}},UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Array{CuArrays.CuArray{Float32,2},1},Array{Float32,1},Array{Array{CuArrays.CuArray{Float32,2},1},1},OrdinaryDiffEq.Tsit5Cache{CuArrays.CuArray{Float32,2},CuArrays.CuArray{Float32,2},CuArrays.CuArray{Float32,2},OrdinaryDiffEq.Tsit5ConstantCache{Float32,Float32}}},DiffEqBase.DEStats},ODEFunction{true,getfield(DiffEqGPU, Symbol("##12#22")){ODEFunction{true,typeof(fip),UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing}},UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},OrdinaryDiffEq.Tsit5Cache{CuArrays.CuArray{Float32,2},CuArrays.CuArray{Float32,2},CuArrays.CuArray{Float32,2},OrdinaryDiffEq.Tsit5ConstantCache{Float32,Float32}},OrdinaryDiffEq.DEOptions{Float32,Float32,Float32,Float32,typeof(DiffEqBase.ODE_DEFAULT_NORM),typeof(opnorm),CallbackSet{Tuple{},Tuple{}},typeof(DiffEqBase.ODE_DEFAULT_ISOUTOFDOMAIN),typeof(DiffEqBase.ODE_DEFAULT_PROG_MESSAGE),typeof(DiffEqBase.ODE_DEFAULT_UNSTABLE_CHECK),DataStructures.BinaryHeap{Float32,DataStructures.LessThan},DataStructures.BinaryHeap{Float32,DataStructures.LessThan},Nothing,Nothing,Int64,Array{Float32,1},Array{Float32,1},Array{Float32,1}},CuArrays.CuArray{Float32,2},Float32,Nothing}, ::OrdinaryDiffEq.Tsit5Cache{CuArrays.CuArray{Float32,2},CuArrays.CuArray{Float32,2},CuArrays.CuArray{Float32,2},OrdinaryDiffEq.Tsit5ConstantCache{Float32,Float32}}) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/OrdinaryDiffEq/tQd6p/src/perform_step/low_order_rk_perform_step.jl:623
 [19] #__init#335(::Array{Float32,1}, ::Array{Float32,1}, ::Array{Float32,1}, ::Nothing, ::Bool, ::Bool, ::Bool, ::Bool, ::Nothing, ::Bool, ::Bool, ::Float32, ::Float32, ::Float32, ::Bool, ::Bool, ::Rational{Int64}, ::Nothing, ::Nothing, ::Rational{Int64}, ::Int64, ::Int64, ::Int64, ::Rational{Int64}, ::Bool, ::Int64, ::Nothing, ::Nothing, ::Int64, ::typeof(DiffEqBase.ODE_DEFAULT_NORM), ::typeof(opnorm), ::typeof(DiffEqBase.ODE_DEFAULT_ISOUTOFDOMAIN), ::typeof(DiffEqBase.ODE_DEFAULT_UNSTABLE_CHECK), ::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::Int64, ::String, ::typeof(DiffEqBase.ODE_DEFAULT_PROG_MESSAGE), ::Nothing, ::Bool, ::Bool, ::Bool, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(DiffEqBase.__init), ::ODEProblem{CuArrays.CuArray{Float32,2},Tuple{Float32,Float32},true,CuArrays.CuArray{DiffEqBase.NullParameters,2},ODEFunction{true,getfield(DiffEqGPU, Symbol("##12#22")){ODEFunction{true,typeof(fip),UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing}},UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem}, ::Tsit5, ::Array{CuArrays.CuArray{Float32,2},1}, ::Array{Float32,1}, ::Array{Any,1}, ::Type{Val{true}}) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/OrdinaryDiffEq/tQd6p/src/solve.jl:352
 [20] __init(::ODEProblem{CuArrays.CuArray{Float32,2},Tuple{Float32,Float32},true,CuArrays.CuArray{DiffEqBase.NullParameters,2},ODEFunction{true,getfield(DiffEqGPU, Symbol("##12#22")){ODEFunction{true,typeof(fip),UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing}},UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem}, ::Tsit5, ::Array{CuArrays.CuArray{Float32,2},1}, ::Array{Float32,1}, ::Array{Any,1}, ::Type{Val{true}}) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/OrdinaryDiffEq/tQd6p/src/solve.jl:66 (repeats 4 times)
 [21] #__solve#334 at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/OrdinaryDiffEq/tQd6p/src/solve.jl:4 [inlined]
 [22] __solve at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/OrdinaryDiffEq/tQd6p/src/solve.jl:4 [inlined]
 [23] #solve_call#425(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(DiffEqBase.solve_call), ::ODEProblem{CuArrays.CuArray{Float32,2},Tuple{Float32,Float32},true,CuArrays.CuArray{DiffEqBase.NullParameters,2},ODEFunction{true,getfield(DiffEqGPU, Symbol("##12#22")){ODEFunction{true,typeof(fip),UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing}},UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem}, ::Tsit5) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqBase/8uyX3/src/solve.jl:40
 [24] solve_call at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqBase/8uyX3/src/solve.jl:37 [inlined]
 [25] #solve#426 at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqBase/8uyX3/src/solve.jl:57 [inlined]
 [26] solve(::ODEProblem{CuArrays.CuArray{Float32,2},Tuple{Float32,Float32},true,CuArrays.CuArray{DiffEqBase.NullParameters,2},ODEFunction{true,getfield(DiffEqGPU, Symbol("##12#22")){ODEFunction{true,typeof(fip),UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing}},UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem}, ::Tsit5) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqBase/8uyX3/src/solve.jl:45
 [27] #batch_solve#5(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(DiffEqGPU.batch_solve), ::EnsembleProblem{ODEProblem{Array{Float32,2},Tuple{Float32,Float32},true,DiffEqBase.NullParameters,ODEFunction{true,typeof(fip),UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},typeof(prob_func),getfield(DiffEqBase, Symbol("##332#338")),getfield(DiffEqBase, Symbol("##334#340")),Array{Any,1}}, ::Tsit5, ::EnsembleGPUArray, ::UnitRange{Int64}) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqGPU/QB1WC/src/DiffEqGPU.jl:66
 [28] batch_solve at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqGPU/QB1WC/src/DiffEqGPU.jl:45 [inlined]
 [29] (::getfield(DiffEqGPU, Symbol("##3#4")){Int64,Int64,Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},EnsembleProblem{ODEProblem{Array{Float32,2},Tuple{Float32,Float32},true,DiffEqBase.NullParameters,ODEFunction{true,typeof(fip),UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},typeof(prob_func),getfield(DiffEqBase, Symbol("##332#338")),getfield(DiffEqBase, Symbol("##334#340")),Array{Any,1}},Tsit5,EnsembleGPUArray})(::Int64) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqGPU/QB1WC/src/DiffEqGPU.jl:37
 [30] iterate at ./generator.jl:47 [inlined]
 [31] _collect(::UnitRange{Int64}, ::Base.Generator{UnitRange{Int64},getfield(DiffEqGPU, Symbol("##3#4")){Int64,Int64,Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},EnsembleProblem{ODEProblem{Array{Float32,2},Tuple{Float32,Float32},true,DiffEqBase.NullParameters,ODEFunction{true,typeof(fip),UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},typeof(prob_func),getfield(DiffEqBase, Symbol("##332#338")),getfield(DiffEqBase, Symbol("##334#340")),Array{Any,1}},Tsit5,EnsembleGPUArray}}, ::Base.EltypeUnknown, ::Base.HasShape{1}) at ./array.jl:619
 [32] collect_similar(::UnitRange{Int64}, ::Base.Generator{UnitRange{Int64},getfield(DiffEqGPU, Symbol("##3#4")){Int64,Int64,Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},EnsembleProblem{ODEProblem{Array{Float32,2},Tuple{Float32,Float32},true,DiffEqBase.NullParameters,ODEFunction{true,typeof(fip),UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},typeof(prob_func),getfield(DiffEqBase, Symbol("##332#338")),getfield(DiffEqBase, Symbol("##334#340")),Array{Any,1}},Tsit5,EnsembleGPUArray}}) at ./array.jl:548
 [33] map(::Function, ::UnitRange{Int64}) at ./abstractarray.jl:2073
 [34] macro expansion at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqGPU/QB1WC/src/DiffEqGPU.jl:31 [inlined]
 [35] macro expansion at ./util.jl:213 [inlined]
 [36] #__solve#2(::Int64, ::Int64, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(DiffEqBase.__solve), ::EnsembleProblem{ODEProblem{Array{Float32,2},Tuple{Float32,Float32},true,DiffEqBase.NullParameters,ODEFunction{true,typeof(fip),UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},typeof(prob_func),getfield(DiffEqBase, Symbol("##332#338")),getfield(DiffEqBase, Symbol("##334#340")),Array{Any,1}}, ::Tsit5, ::EnsembleGPUArray) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqGPU/QB1WC/src/DiffEqGPU.jl:30
 [37] #__solve at ./none:0 [inlined]
 [38] #solve#427 at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqBase/8uyX3/src/solve.jl:64 [inlined]
 [39] (::getfield(DiffEqBase, Symbol("#kw##solve")))(::NamedTuple{(:trajectories,),Tuple{Int64}}, ::typeof(solve), ::EnsembleProblem{ODEProblem{Array{Float32,2},Tuple{Float32,Float32},true,DiffEqBase.NullParameters,ODEFunction{true,typeof(fip),UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},typeof(prob_func),getfield(DiffEqBase, Symbol("##332#338")),getfield(DiffEqBase, Symbol("##334#340")),Array{Any,1}}, ::Tsit5, ::EnsembleGPUArray) at ./none:0
 [40] top-level scope at In[3]:15
18

another thought: this example is unusual in that u is not a vector but a 4x2 matrix. maybe that confuses the compilation? it sure confuses the plot recipes…

no, it’s not that. if i change u0 to be a length-4 vector, the same error results

19

Open an issue for that. I am not sure why this one doesn’t compile. Reason: unsupported dynamic function invocation (call to overdub) is odd since I don’t see why this needs another overdub.

20

ok.

1 Like