# \]test DiffEqGPU errors with " unsupported call to the Julia runtime"

**URL:** <https://discourse.julialang.org/t/test-diffeqgpu-errors-with-unsupported-call-to-the-julia-runtime/28893>\
**Category:** GPU\
**Tags:** first-steps\
**Created:** [September 18, 2019, 1:49pm UTC](https://discourse.julialang.org/t/test-diffeqgpu-errors-with-unsupported-call-to-the-julia-runtime/28893 "2019-09-18T13:49:30Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![nilsbecker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilsbecker/32/12157_2.png) [@nilsbecker](https://discourse.julialang.org/u/nilsbecker)\
**Post date:** [September 18, 2019, 1:49pm UTC](https://discourse.julialang.org/t/test-diffeqgpu-errors-with-unsupported-call-to-the-julia-runtime/28893/1 "2019-09-18T13:49:31Z")

</div>

hi, i’m trying to experiment with ODE solving on GPUs but can’t seem to get the packages running properly. this is on a machine with CUDA 9.0 installed and 4 Tesla K80 GPUs.

i can install DiffEqGPU but the test fails as indicated above. i did not find minimum requirements for the GPU compute capabilities or the CUDA version so i am assuming this should be ok.

a snippet from the error output:

`InvalidIRError: [...] Reason: unsupported call to the Julia runtime (call to jl_f_tuple)`

did i do something wrong? sorry if this is off-topic for this forum!

---

<div class="post-metadata">

**Author:** ![nilsbecker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilsbecker/32/12157_2.png) [@nilsbecker](https://discourse.julialang.org/u/nilsbecker)\
**Post date:** [September 18, 2019, 2:03pm UTC](https://discourse.julialang.org/t/test-diffeqgpu-errors-with-unsupported-call-to-the-julia-runtime/28893/2 "2019-09-18T14:03:47Z")

</div>

this is on julia 1.2 and today’s versions of all required packages. i also get the following error with `]test CuArrays`:

` symbol lookup error: /opt/cuda-9.0/lib64/libcusolver.so: undefined symbol: omp_get_max_threads`

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [September 18, 2019, 3:37pm UTC](https://discourse.julialang.org/t/test-diffeqgpu-errors-with-unsupported-call-to-the-julia-runtime/28893/3 "2019-09-18T15:37:40Z")

</div>

Can we see the full backtrace? We need to see the entire error message to know what’s going on. This could possibly be fixed already on GPUArrays.jl master. Try doing

```julia
add GPUArrays#master

```

and see if it works.

---

<div class="post-metadata">

**Author:** ![nilsbecker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilsbecker/32/12157_2.png) [@nilsbecker](https://discourse.julialang.org/u/nilsbecker)\
**Post date:** [September 18, 2019, 3:41pm UTC](https://discourse.julialang.org/t/test-diffeqgpu-errors-with-unsupported-call-to-the-julia-runtime/28893/4 "2019-09-18T15:41:32Z")

</div>

sure, will try.

ok, so i did `add GPUArrays#master` which completed without errors, in a fresh REPL. i then did `precompile` and then `test DiffEqGPU`. this gave the following output:

```julia
(v1.2) pkg> test DiffEqGPU
   Testing DiffEqGPU
 Resolving package versions...
    Status `/tmp/jl_A7B6sE/Manifest.toml`
  [621f4979] AbstractFFTs v0.4.1
  [79e6a3ab] Adapt v1.0.0
  [ec485272] ArnoldiMethod v0.0.4
  [4fba245c] ArrayInterface v1.2.1
  [aae01518] BandedMatrices v0.10.1
  [9e28174c] BinDeps v0.8.10
  [b99e7846] BinaryProvider v0.5.6
  [8e7c35d0] BlockArrays v0.9.1
  [ffab5731] BlockBandedMatrices v0.4.6
  [fa961155] CEnum v0.2.0
  [00ebfdb7] CSTParser v0.6.2
  [3895d2a7] CUDAapi v1.1.0
  [c5f51814] CUDAdrv v3.1.0
  [be33ccc6] CUDAnative v2.3.1
  [49dc2e85] Calculus v0.5.0
  [7057c7e9] Cassette v0.2.6
  [bbf7d656] CommonSubexpressions v0.2.0
  [34da2185] Compat v2.1.0
  [8f4d0f93] Conda v1.3.0
  [a8cc5b0e] Crayons v4.0.0
  [3a865a2d] CuArrays v1.2.1
  [864edb3b] DataStructures v0.17.0
  [2b5f629d] DiffEqBase v6.2.3
  [01453d9d] DiffEqDiffTools v1.3.0
  [071ae1c0] DiffEqGPU v0.1.0
  [163ba53b] DiffResults v0.0.4
  [b552c78f] DiffRules v0.0.10
  [b4f34e82] Distances v0.8.2
  [ffbed154] DocStringExtensions v0.8.0
  [d4d017d3] ExponentialUtilities v1.5.1
  [7a1cc6ca] FFTW v1.0.0
  [1a297f60] FillArrays v0.6.4
  [f6369f11] ForwardDiff v0.10.3
  [069b7b12] FunctionWrappers v1.0.0
  [0c68f7d7] GPUArrays v1.0.3 #master (https://github.com/JuliaGPU/GPUArrays.jl.git)
  [ba82f77b] GPUifyLoops v0.2.8
  [01680d73] GenericSVD v0.2.1
  [d25df0c9] Inflate v0.1.1
  [42fd0dbc] IterativeSolvers v0.8.1
  [82899510] IteratorInterfaceExtensions v1.0.0
  [682c06a0] JSON v0.21.0
  [929cbde3] LLVM v1.3.0
  [5078a376] LazyArrays v0.10.0
  [093fc24a] LightGraphs v1.3.0
  [d3d80556] LineSearches v7.0.1
  [1914dd2f] MacroTools v0.5.1
  [a3b82374] MatrixFactorizations v0.1.0
  [46d2c3a1] MuladdMacro v0.2.1
  [d41bc354] NLSolversBase v7.4.1
  [2774e3e8] NLsolve v4.1.0
  [872c559c] NNlib v0.6.0
  [77ba4419] NaNMath v0.3.2
  [bac558e1] OrderedCollections v1.1.0
  [1dea7af3] OrdinaryDiffEq v5.17.0
  [d96e819e] Parameters v0.12.0
  [69de0a69] Parsers v0.3.7
  [3cdcf5f2] RecipesBase v0.7.0
  [731186ca] RecursiveArrayTools v1.0.2
  [f2c3362d] RecursiveFactorization v0.1.0
  [189a3867] Reexport v0.2.0
  [ae029012] Requires v0.5.2
  [f2b01f46] Roots v0.8.3
  [699a6c99] SimpleTraits v0.9.0
  [47a9eef4] SparseDiffTools v0.9.0
  [276daf66] SpecialFunctions v0.8.0
  [90137ffa] StaticArrays v0.11.0
  [3783bdb8] TableTraits v1.0.0
  [a759f4b9] TimerOutputs v0.5.0
  [0796e94c] Tokenize v0.5.6
  [a2a6695c] TreeViews v0.3.0
  [30578b45] URIParser v0.4.0
  [81def892] VersionParsing v1.1.3
  [19fa3120] VertexSafeGraphs v0.1.0
  [2a0f44e3] Base64 [`@stdlib/Base64`]
  [ade2ca70] Dates [`@stdlib/Dates`]
  [8bb1440f] DelimitedFiles [`@stdlib/DelimitedFiles`]
  [8ba89e20] Distributed [`@stdlib/Distributed`]
  [b77e0a4c] InteractiveUtils [`@stdlib/InteractiveUtils`]
  [76f85450] LibGit2 [`@stdlib/LibGit2`]
  [8f399da3] Libdl [`@stdlib/Libdl`]
  [37e2e46d] LinearAlgebra [`@stdlib/LinearAlgebra`]
  [56ddb016] Logging [`@stdlib/Logging`]
  [d6f4376e] Markdown [`@stdlib/Markdown`]
  [a63ad114] Mmap [`@stdlib/Mmap`]
  [44cfe95a] Pkg [`@stdlib/Pkg`]
  [de0858da] Printf [`@stdlib/Printf`]
  [9abbd945] Profile [`@stdlib/Profile`]
  [3fa0cd96] REPL [`@stdlib/REPL`]
  [9a3f8284] Random [`@stdlib/Random`]
  [ea8e919c] SHA [`@stdlib/SHA`]
  [9e88b42a] Serialization [`@stdlib/Serialization`]
  [1a1011a3] SharedArrays [`@stdlib/SharedArrays`]
  [6462fe0b] Sockets [`@stdlib/Sockets`]
  [2f01184e] SparseArrays [`@stdlib/SparseArrays`]
  [10745b16] Statistics [`@stdlib/Statistics`]
  [4607b0f0] SuiteSparse [`@stdlib/SuiteSparse`]
  [8dfed614] Test [`@stdlib/Test`]
  [cf7118a7] UUIDs [`@stdlib/UUIDs`]
  [4ec0a83e] Unicode [`@stdlib/Unicode`]
WARNING: Method definition overdub(Cassette.Context{N, M, T, P, B, H} where H<:Union{Cassette.DisableHooks, Nothing} where B<:Union{Nothing, Base.IdDict{Module, Base.Dict{Symbol, Cassette.BindingMeta}}} where P<:Cassette.AbstractPass where T<:Union{Nothing, Cassette.Tag{N, X, E} where E where X where N<:Cassette.AbstractContextName} where M where N<:Cassette.AbstractContextName, Any...) in module Cassette at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/Cassette/YCOeN/src/overdub.jl:524 overwritten in module GPUifyLoops at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/Cassette/YCOeN/src/overdub.jl:524.
  **incremental compilation may be fatally broken for this module**

WARNING: Method definition recurse(Cassette.Context{N, M, T, P, B, H} where H<:Union{Cassette.DisableHooks, Nothing} where B<:Union{Nothing, Base.IdDict{Module, Base.Dict{Symbol, Cassette.BindingMeta}}} where P<:Cassette.AbstractPass where T<:Union{Nothing, Cassette.Tag{N, X, E} where E where X where N<:Cassette.AbstractContextName} where M where N<:Cassette.AbstractContextName, Any...) in module Cassette at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/Cassette/YCOeN/src/overdub.jl:537 overwritten in module GPUifyLoops at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/Cassette/YCOeN/src/overdub.jl:537.
  **incremental compilation may be fatally broken for this module**

[ Info: Building the CUDAnative run-time library for your sm_37 device, this might take a while...
ERROR: LoadError: InvalidIRError: compiling gpu_kernel(Cassette.Context{nametype(Ctx),Nothing,Nothing,getfield(GPUifyLoops, Symbol("##PassType#371")),Nothing,Cassette.DisableHooks}, typeof(DiffEqGPU.gpu_kernel), ODEFunction{true,typeof(lorenz),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing}, CUDAnative.CuDeviceArray{Float32,2,CUDAnative.AS.Global}, CUDAnative.CuDeviceArray{Float32,2,CUDAnative.AS.Global}, CUDAnative.CuDeviceArray{DiffEqBase.NullParameters,2,CUDAnative.AS.Global}, Float32) resulted in invalid LLVM IR
Reason: unsupported call to the Julia runtime (call to jl_f_tuple)
Stacktrace:
 [1] overdub at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/Cassette/YCOeN/src/overdub.jl:524
 [2] multiple call sites at unknown:0
Reason: unsupported call to the Julia runtime (call to jl_f_getfield)
Stacktrace:
 [1] overdub at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/Cassette/YCOeN/src/overdub.jl:524
 [2] multiple call sites at unknown:0
Stacktrace:
 [1] check_ir(::CUDAnative.CompilerJob, ::LLVM.Module) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/CUDAnative/UWBIY/src/compiler/validation.jl:114
 [2] macro expansion at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/CUDAnative/UWBIY/src/compiler/driver.jl:188 [inlined]
 [3] macro expansion at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/TimerOutputs/7zSea/src/TimerOutput.jl:216 [inlined]
 [4] #codegen#130(::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::typeof(CUDAnative.codegen), ::Symbol, ::CUDAnative.CompilerJob) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/CUDAnative/UWBIY/src/compiler/driver.jl:186
 [5] #codegen at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/CUDAnative/UWBIY/src/compiler/driver.jl:0 [inlined]
 [6] #compile#129(::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::typeof(CUDAnative.compile), ::Symbol, ::CUDAnative.CompilerJob) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/CUDAnative/UWBIY/src/compiler/driver.jl:47
 [7] #compile at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/CUDAnative/UWBIY/src/compiler/common.jl:0 [inlined]
 [8] #compile#128 at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/CUDAnative/UWBIY/src/compiler/driver.jl:28 [inlined]
 [9] #compile at ./none:0 [inlined] (repeats 2 times)
 [10] macro expansion at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/CUDAnative/UWBIY/src/execution.jl:389 [inlined]
 [11] #cufunction#170(::String, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(CUDAnative.cufunction), ::typeof(Cassette.overdub), ::Type{Tuple{Cassette.Context{nametype(Ctx),Nothing,Nothing,getfield(GPUifyLoops, Symbol("##PassType#371")),Nothing,Cassette.DisableHooks},typeof(DiffEqGPU.gpu_kernel),ODEFunction{true,typeof(lorenz),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},CUDAnative.CuDeviceArray{Float32,2,CUDAnative.AS.Global},CUDAnative.CuDeviceArray{Float32,2,CUDAnative.AS.Global},CUDAnative.CuDeviceArray{DiffEqBase.NullParameters,2,CUDAnative.AS.Global},Float32}}) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/CUDAnative/UWBIY/src/execution.jl:357
 [12] (::getfield(CUDAnative, Symbol("#kw##cufunction")))(::NamedTuple{(:name,),Tuple{String}}, ::typeof(CUDAnative.cufunction), ::Function, ::Type) at ./none:0
 [13] macro expansion at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/GPUifyLoops/mjszO/src/GPUifyLoops.jl:125 [inlined]
 [14] macro expansion at ./gcutils.jl:87 [inlined]
 [15] #launch#50(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(GPUifyLoops.launch), ::GPUifyLoops.CUDA, ::typeof(DiffEqGPU.gpu_kernel), ::Function, ::Vararg{Any,N} where N) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/GPUifyLoops/mjszO/src/GPUifyLoops.jl:121
 [16] launch at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/GPUifyLoops/mjszO/src/GPUifyLoops.jl:119 [inlined]
 [17] macro expansion at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/GPUifyLoops/mjszO/src/GPUifyLoops.jl:54 [inlined]
 [18] #12 at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqGPU/QB1WC/src/DiffEqGPU.jl:61 [inlined]
 [19] ODEFunction at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqBase/8uyX3/src/diffeqfunction.jl:230 [inlined]
 [20] initialize!(::OrdinaryDiffEq.ODEIntegrator{Tsit5,true,CuArray{Float32,2},Float32,CuArray{DiffEqBase.NullParameters,2},Float32,Float32,Float32,Array{CuArray{Float32,2},1},ODESolution{Float32,3,Array{CuArray{Float32,2},1},Nothing,Nothing,Array{Float32,1},Array{Array{CuArray{Float32,2},1},1},ODEProblem{CuArray{Float32,2},Tuple{Float32,Float32},true,CuArray{DiffEqBase.NullParameters,2},ODEFunction{true,getfield(DiffEqGPU, Symbol("##12#22")){ODEFunction{true,typeof(lorenz),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},Tsit5,OrdinaryDiffEq.InterpolationData{ODEFunction{true,getfield(DiffEqGPU, Symbol("##12#22")){ODEFunction{true,typeof(lorenz),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Array{CuArray{Float32,2},1},Array{Float32,1},Array{Array{CuArray{Float32,2},1},1},OrdinaryDiffEq.Tsit5Cache{CuArray{Float32,2},CuArray{Float32,2},CuArray{Float32,2},OrdinaryDiffEq.Tsit5ConstantCache{Float32,Float32}}},DiffEqBase.DEStats},ODEFunction{true,getfield(DiffEqGPU, Symbol("##12#22")){ODEFunction{true,typeof(lorenz),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},OrdinaryDiffEq.Tsit5Cache{CuArray{Float32,2},CuArray{Float32,2},CuArray{Float32,2},OrdinaryDiffEq.Tsit5ConstantCache{Float32,Float32}},OrdinaryDiffEq.DEOptions{Float32,Float32,Float32,Float32,typeof(DiffEqBase.ODE_DEFAULT_NORM),typeof(LinearAlgebra.opnorm),CallbackSet{Tuple{},Tuple{}},typeof(DiffEqBase.ODE_DEFAULT_ISOUTOFDOMAIN),typeof(DiffEqBase.ODE_DEFAULT_PROG_MESSAGE),typeof(DiffEqBase.ODE_DEFAULT_UNSTABLE_CHECK),DataStructures.BinaryHeap{Float32,DataStructures.LessThan},DataStructures.BinaryHeap{Float32,DataStructures.LessThan},Nothing,Nothing,Int64,Array{Float32,1},Float32,Array{Float32,1}},CuArray{Float32,2},Float32,Nothing}, ::OrdinaryDiffEq.Tsit5Cache{CuArray{Float32,2},CuArray{Float32,2},CuArray{Float32,2},OrdinaryDiffEq.Tsit5ConstantCache{Float32,Float32}}) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/OrdinaryDiffEq/tQd6p/src/perform_step/low_order_rk_perform_step.jl:623
 [21] #__init#335(::Float32, ::Array{Float32,1}, ::Array{Float32,1}, ::Nothing, ::Bool, ::Bool, ::Bool, ::Bool, ::Nothing, ::Bool, ::Bool, ::Float32, ::Float32, ::Float32, ::Bool, ::Bool, ::Rational{Int64}, ::Nothing, ::Nothing, ::Rational{Int64}, ::Int64, ::Int64, ::Int64, ::Rational{Int64}, ::Bool, ::Int64, ::Nothing, ::Nothing, ::Int64, ::typeof(DiffEqBase.ODE_DEFAULT_NORM), ::typeof(LinearAlgebra.opnorm), ::typeof(DiffEqBase.ODE_DEFAULT_ISOUTOFDOMAIN), ::typeof(DiffEqBase.ODE_DEFAULT_UNSTABLE_CHECK), ::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::Int64, ::String, ::typeof(DiffEqBase.ODE_DEFAULT_PROG_MESSAGE), ::Nothing, ::Bool, ::Bool, ::Bool, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(DiffEqBase.__init), ::ODEProblem{CuArray{Float32,2},Tuple{Float32,Float32},true,CuArray{DiffEqBase.NullParameters,2},ODEFunction{true,getfield(DiffEqGPU, Symbol("##12#22")){ODEFunction{true,typeof(lorenz),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem}, ::Tsit5, ::Array{CuArray{Float32,2},1}, ::Array{Float32,1}, ::Array{Any,1}, ::Type{Val{true}}) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/OrdinaryDiffEq/tQd6p/src/solve.jl:352
 [22] (::getfield(DiffEqBase, Symbol("#kw##__init")))(::NamedTuple{(:saveat,),Tuple{Float32}}, ::typeof(DiffEqBase.__init), ::ODEProblem{CuArray{Float32,2},Tuple{Float32,Float32},true,CuArray{DiffEqBase.NullParameters,2},ODEFunction{true,getfield(DiffEqGPU, Symbol("##12#22")){ODEFunction{true,typeof(lorenz),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem}, ::Tsit5, ::Array{CuArray{Float32,2},1}, ::Array{Float32,1}, ::Array{Any,1}, ::Type{Val{true}}) at ./none:0 (repeats 4 times)
 [23] #__solve#334 at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/OrdinaryDiffEq/tQd6p/src/solve.jl:4 [inlined]
 [24] #__solve at ./none:0 [inlined]
 [25] #solve_call#425(::Base.Iterators.Pairs{Symbol,Float32,Tuple{Symbol},NamedTuple{(:saveat,),Tuple{Float32}}}, ::typeof(DiffEqBase.solve_call), ::ODEProblem{CuArray{Float32,2},Tuple{Float32,Float32},true,CuArray{DiffEqBase.NullParameters,2},ODEFunction{true,getfield(DiffEqGPU, Symbol("##12#22")){ODEFunction{true,typeof(lorenz),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem}, ::Tsit5) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqBase/8uyX3/src/solve.jl:40
 [26] #solve_call at ./none:0 [inlined]
 [27] #solve#426 at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqBase/8uyX3/src/solve.jl:57 [inlined]
 [28] (::getfield(DiffEqBase, Symbol("#kw##solve")))(::NamedTuple{(:saveat,),Tuple{Float32}}, ::typeof(solve), ::ODEProblem{CuArray{Float32,2},Tuple{Float32,Float32},true,CuArray{DiffEqBase.NullParameters,2},ODEFunction{true,getfield(DiffEqGPU, Symbol("##12#22")){ODEFunction{true,typeof(lorenz),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem}, ::Tsit5) at ./none:0
 [29] #batch_solve#5(::Base.Iterators.Pairs{Symbol,Float32,Tuple{Symbol},NamedTuple{(:saveat,),Tuple{Float32}}}, ::typeof(DiffEqGPU.batch_solve), ::EnsembleProblem{ODEProblem{Array{Float32,1},Tuple{Float32,Float32},true,DiffEqBase.NullParameters,ODEFunction{true,typeof(lorenz),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},getfield(DiffEqBase, Symbol("##330#336")),getfield(DiffEqBase, Symbol("##329#335")),getfield(DiffEqBase, Symbol("##331#337")),Array{Any,1}}, ::Tsit5, ::EnsembleGPUArray, ::UnitRange{Int64}) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqGPU/QB1WC/src/DiffEqGPU.jl:66
 [30] #batch_solve at ./none:0 [inlined]
 [31] (::getfield(DiffEqGPU, Symbol("##3#4")){Int64,Int64,Base.Iterators.Pairs{Symbol,Float32,Tuple{Symbol},NamedTuple{(:saveat,),Tuple{Float32}}},EnsembleProblem{ODEProblem{Array{Float32,1},Tuple{Float32,Float32},true,DiffEqBase.NullParameters,ODEFunction{true,typeof(lorenz),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},getfield(DiffEqBase, Symbol("##330#336")),getfield(DiffEqBase, Symbol("##329#335")),getfield(DiffEqBase, Symbol("##331#337")),Array{Any,1}},Tsit5,EnsembleGPUArray})(::Int64) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqGPU/QB1WC/src/DiffEqGPU.jl:37
 [32] iterate at ./generator.jl:47 [inlined]
 [33] _collect(::UnitRange{Int64}, ::Base.Generator{UnitRange{Int64},getfield(DiffEqGPU, Symbol("##3#4")){Int64,Int64,Base.Iterators.Pairs{Symbol,Float32,Tuple{Symbol},NamedTuple{(:saveat,),Tuple{Float32}}},EnsembleProblem{ODEProblem{Array{Float32,1},Tuple{Float32,Float32},true,DiffEqBase.NullParameters,ODEFunction{true,typeof(lorenz),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},getfield(DiffEqBase, Symbol("##330#336")),getfield(DiffEqBase, Symbol("##329#335")),getfield(DiffEqBase, Symbol("##331#337")),Array{Any,1}},Tsit5,EnsembleGPUArray}}, ::Base.EltypeUnknown, ::Base.HasShape{1}) at ./array.jl:619
 [34] collect_similar(::UnitRange{Int64}, ::Base.Generator{UnitRange{Int64},getfield(DiffEqGPU, Symbol("##3#4")){Int64,Int64,Base.Iterators.Pairs{Symbol,Float32,Tuple{Symbol},NamedTuple{(:saveat,),Tuple{Float32}}},EnsembleProblem{ODEProblem{Array{Float32,1},Tuple{Float32,Float32},true,DiffEqBase.NullParameters,ODEFunction{true,typeof(lorenz),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},getfield(DiffEqBase, Symbol("##330#336")),getfield(DiffEqBase, Symbol("##329#335")),getfield(DiffEqBase, Symbol("##331#337")),Array{Any,1}},Tsit5,EnsembleGPUArray}}) at ./array.jl:548
 [35] map(::Function, ::UnitRange{Int64}) at ./abstractarray.jl:2073
 [36] macro expansion at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqGPU/QB1WC/src/DiffEqGPU.jl:31 [inlined]
 [37] macro expansion at ./util.jl:213 [inlined]
 [38] #__solve#2(::Int64, ::Int64, ::Base.Iterators.Pairs{Symbol,Float32,Tuple{Symbol},NamedTuple{(:saveat,),Tuple{Float32}}}, ::typeof(DiffEqBase.__solve), ::EnsembleProblem{ODEProblem{Array{Float32,1},Tuple{Float32,Float32},true,DiffEqBase.NullParameters,ODEFunction{true,typeof(lorenz),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},getfield(DiffEqBase, Symbol("##330#336")),getfield(DiffEqBase, Symbol("##329#335")),getfield(DiffEqBase, Symbol("##331#337")),Array{Any,1}}, ::Tsit5, ::EnsembleGPUArray) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqGPU/QB1WC/src/DiffEqGPU.jl:30
 [39] #__solve at ./none:0 [inlined]
 [40] #solve#427 at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqBase/8uyX3/src/solve.jl:64 [inlined]
 [41] (::getfield(DiffEqBase, Symbol("#kw##solve")))(::NamedTuple{(:trajectories, :saveat),Tuple{Int64,Float32}}, ::typeof(solve), ::EnsembleProblem{ODEProblem{Array{Float32,1},Tuple{Float32,Float32},true,DiffEqBase.NullParameters,ODEFunction{true,typeof(lorenz),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},getfield(DiffEqBase, Symbol("##330#336")),getfield(DiffEqBase, Symbol("##329#335")),getfield(DiffEqBase, Symbol("##331#337")),Array{Any,1}}, ::Tsit5, ::EnsembleGPUArray) at ./none:0
 [42] top-level scope at util.jl:156
 [43] include at ./boot.jl:328 [inlined]
 [44] include_relative(::Module, ::String) at ./loading.jl:1094
 [45] include(::Module, ::String) at ./Base.jl:31
 [46] include(::String) at ./client.jl:431
 [47] top-level scope at none:5
in expression starting at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqGPU/QB1WC/test/runtests.jl:20
ERROR: Package DiffEqGPU errored during testing

```

---

<div class="post-metadata">

**Author:** ![nilsbecker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilsbecker/32/12157_2.png) [@nilsbecker](https://discourse.julialang.org/u/nilsbecker)\
**Post date:** [September 18, 2019, 3:54pm UTC](https://discourse.julialang.org/t/test-diffeqgpu-errors-with-unsupported-call-to-the-julia-runtime/28893/5 "2019-09-18T15:54:29Z")

</div>

fwiw, i also get a warning message when `using DiffEqGPU` which complains about incremental compilation not working:

```julia
┌ Info: Recompiling stale cache file /home/bq_nbecker/isi/julia_pkg/GPUTest/compiled/v1.2/DiffEqGPU/8rwkT.ji for DiffEqGPU [071ae1c0-96b5-11e9-1965-c90190d839ea]
└ @ Base loading.jl:1240
WARNING: Method definition overdub(Cassette.Context{N, M, T, P, B, H} where H<:Union{Cassette.DisableHooks, Nothing} where B<:Union{Nothing, Base.IdDict{Module, Base.Dict{Symbol, Cassette.BindingMeta}}} where P<:Cassette.AbstractPass where T<:Union{Nothing, Cassette.Tag{N, X, E} where E where X where N<:Cassette.AbstractContextName} where M where N<:Cassette.AbstractContextName, Any...) in module Cassette at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/Cassette/YCOeN/src/overdub.jl:524 overwritten in module GPUifyLoops at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/Cassette/YCOeN/src/overdub.jl:524.
  **incremental compilation may be fatally broken for this module**

```

---

<div class="post-metadata">

**Author:** ![nilsbecker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilsbecker/32/12157_2.png) [@nilsbecker](https://discourse.julialang.org/u/nilsbecker)\
**Post date:** [September 18, 2019, 4:04pm UTC](https://discourse.julialang.org/t/test-diffeqgpu-errors-with-unsupported-call-to-the-julia-runtime/28893/6 "2019-09-18T16:04:46Z")

</div>

another bit: the gpu has cuda compute capability 3.7

---

<div class="post-metadata">

**Author:** ![nilsbecker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilsbecker/32/12157_2.png) [@nilsbecker](https://discourse.julialang.org/u/nilsbecker)\
**Post date:** [September 18, 2019, 4:11pm UTC](https://discourse.julialang.org/t/test-diffeqgpu-errors-with-unsupported-call-to-the-julia-runtime/28893/7 "2019-09-18T16:11:57Z")

</div>

I also tried to copy-and-paste the code in ` DiffEqGPU.jl/test/runtests.jl` to a REPL. Except for the fact that i have to manually install `OrdinaryDiffEq`, and i get the incremental compilation warnings above, this works. i can successfully run the full file, and indeed the gpu ensembles are faster than `EnsembleCPUArray`. (although the timings are extremely variable: 2s-20s for the GPU ensembles from run to run, after several runs(

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [September 18, 2019, 5:58pm UTC](https://discourse.julialang.org/t/test-diffeqgpu-errors-with-unsupported-call-to-the-julia-runtime/28893/8 "2019-09-18T17:58:03Z")

</div>

Glad to see it’s working. I am not sure what that error you’re getting is, it seems hard to track it down if we can’t get a way to reproduce it.

> [@nilsbecker](#):
>
> although the timings are extremely variable: 2s-20s for the GPU ensembles from run to run, after several runs

See what happens if you GC before the runs.

---

<div class="post-metadata">

**Author:** ![nilsbecker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilsbecker/32/12157_2.png) [@nilsbecker](https://discourse.julialang.org/u/nilsbecker)\
**Post date:** [September 19, 2019, 8:11am UTC](https://discourse.julialang.org/t/test-diffeqgpu-errors-with-unsupported-call-to-the-julia-runtime/28893/9 "2019-09-19T08:11:49Z")

</div>

yes, thanks for looking into it. i think the gpu is working in principle (although the test failure persists). i tried running `Base.GC.gc()` before the `@time ...` tests and this somewhat influenced the timings. but the large increase in timing to over 30s appears only when i run the test from within a (remote) jupyter notebook. this is likely an unrelated issue which could have to do with the disk quota being maxed out on that computer.

i have one more related question: how much of the automatic GPU parallelism in DiffEqGPU is currently supposed to be working?

1. can i define an arbitrary scalar ODE problem and then solve a parameter-ensemble of the problem on the GPU, or do restrictions apply (except for handling events which was mentioned in your blog post)

2. if i define a small system of ODEs to be run as a parameter ensemble, what array type to i have to use for u, u0 and p? (i see the test uses `Float32` arrays, not CuArrays or some such)

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [September 19, 2019, 1:57pm UTC](https://discourse.julialang.org/t/test-diffeqgpu-errors-with-unsupported-call-to-the-julia-runtime/28893/10 "2019-09-19T13:57:54Z")

</div>

> [@nilsbecker](#):
>
> i have one more related question: how much of the automatic GPU parallelism in DiffEqGPU is currently supposed to be working?

A little bit, but not much.

> [@nilsbecker](#):
>
> - can i define an arbitrary scalar ODE problem and then solve a parameter-ensemble of the problem on the GPU, or do restrictions apply (except for handling events which was mentioned in your blog post)

For now it needs to be non-stiff, but events and stiff ODEs are being fixed hopefully by the end of the week: [Jac by ChrisRackauckas · Pull Request #9 · SciML/DiffEqGPU.jl · GitHub](https://github.com/JuliaDiffEq/DiffEqGPU.jl/pull/9)

> [@nilsbecker](#):
>
> 1. if i define a small system of ODEs to be run as a parameter ensemble, what array type to i have to use for u, u0 and p? (i see the test uses `Float32` arrays, not CuArrays or some such)

Float32 arrays. It takes your single problem and generates a GPU-based problem from it, so it’s invisible on your end.

---

<div class="post-metadata">

**Author:** ![nilsbecker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilsbecker/32/12157_2.png) [@nilsbecker](https://discourse.julialang.org/u/nilsbecker)\
**Post date:** [September 19, 2019, 4:28pm UTC](https://discourse.julialang.org/t/test-diffeqgpu-errors-with-unsupported-call-to-the-julia-runtime/28893/11 "2019-09-19T16:28:44Z")

</div>

i have been testing further using example 3 from [http://docs.juliadiffeq.org/latest/tutorials/ode\_example.html](http://docs.juliadiffeq.org/latest/tutorials/ode_example.html) . There the RHS of the ode is just `A * u`. I made `A` and `u` into Float32 arrays and made an ensemble problem by adding small amounts of noise on the `u0` initial condition.

```julia
A = Float32[
    1. 0 0 -5
     4 -2 4 -3
     -4 0 0 1
      5 -2 2 3]
u0 = Array{Float32}(rand(4,2))
tspan = (0.0,1.0)
f(u,p,t) = A*u
prob = ODEProblem(f,u0,tspan)
function prob_func(prob, i, repeat)
    prob.u0 .+= 0.01 * Array{Float32}(randn(4,2))
    prob
end
saveat = 0.01
trajs = 123
ensprobfull=EnsembleProblem(prob, prob_func=prob_func)
simfull = solve(ensprobfull,Tsit5(),EnsembleDistributed(), trajectories=trajs, saveat=saveat)
# now try on gpu
using DiffEqGPU
simgpu = solve(ensprobfull, Tsit5(), EnsembleGPUArray(), trajectories=trajs, saveat=saveat)

```

i think this cannot be terribly efficient since it’s not in-place modification of the `du` vector – however it does not run at all on the GPU for me, i get

```julia
InvalidIRError: compiling gpu_kernel(Cassette.Context{nametype(Ctx),Nothing,Nothing,getfield(GPUifyLoops, Symbol("##PassType#371")),Nothing,Cassette.DisableHooks}, typeof(DiffEqGPU.gpu_kernel), ODEFunction{false,typeof(f),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing}, CUDAnative.CuDeviceArray{Float32,2,CUDAnative.AS.Global}, CUDAnative.CuDeviceArray{Float32,2,CUDAnative.AS.Global}, CUDAnative.CuDeviceArray{DiffEqBase.NullParameters,2,CUDAnative.AS.Global}, Float64) resulted in invalid LLVM IR
Reason: unsupported dynamic function invocation (call to f)

```

without specifying a solver algorithm it does run – at the same speed as the CPU version.

is this expected? (it wasn’t for me, i’m just learning julia diff eq)

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [September 19, 2019, 4:35pm UTC](https://discourse.julialang.org/t/test-diffeqgpu-errors-with-unsupported-call-to-the-julia-runtime/28893/12 "2019-09-19T16:35:28Z")

</div>

> [@nilsbecker](#):
>
> i think this cannot be terribly efficient since it’s not in-place modification of the `du` vector – however it does not run at all on the GPU for me, i get

Those two facts are the same thing. The GPU code must be non-allocating: it’s a current limitation. The fact that your code is allocating the arrays means it cannot auto-GPU.

---

<div class="post-metadata">

**Author:** ![nilsbecker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilsbecker/32/12157_2.png) [@nilsbecker](https://discourse.julialang.org/u/nilsbecker)\
**Post date:** [September 20, 2019, 8:09am UTC](https://discourse.julialang.org/t/test-diffeqgpu-errors-with-unsupported-call-to-the-julia-runtime/28893/13 "2019-09-20T08:09:12Z")

</div>

ah ok, i must have missed that limitation. thanks

---

<div class="post-metadata">

**Author:** ![nilsbecker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilsbecker/32/12157_2.png) [@nilsbecker](https://discourse.julialang.org/u/nilsbecker)\
**Post date:** [September 20, 2019, 8:46am UTC](https://discourse.julialang.org/t/test-diffeqgpu-errors-with-unsupported-call-to-the-julia-runtime/28893/14 "2019-09-20T08:46:03Z")

</div>

i’ve now modified the problem to be in-place, which i assumed should work:

```julia
function fip(du,u,p,t) 
    du .= A*u
end
probip = ODEProblem(fip,u0,tspan)
ensprobip=EnsembleProblem(probip, prob_func=prob_func)
using DiffEqGPU
simgpu = solve(ensprobip, Tsit5(), EnsembleGPUArray(), trajectories=trajs, saveat=saveat)

```

the problem `probip` is reported as being in-place.

for the gpu ensemble problem, i still get the same error as above: `[...]invalid LLVM IR Reason: unsupported dynamic function invocation (call to overdub)`. without `Tsit5` it does run, as before, but slower than on CPU with `EnsembleDistributed` – does this mean that it quietly falls back to CPU computation? i’m confused.

when i `@time sol=solve(probip, Tsit5(), saveat=0.01)` this reports 116MiB of allocations. so, is my code allocating and therefore does not run on gpu? (if that’s the case, a more specific error message from DiffEqGPU would be helpful.)

i don’t see where allocation happens and how to avoid it?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [September 20, 2019, 11:55am UTC](https://discourse.julialang.org/t/test-diffeqgpu-errors-with-unsupported-call-to-the-julia-runtime/28893/15 "2019-09-20T11:55:36Z")

</div>

> [@nilsbecker](#):
>
> A\*u

This allocates. You’d need to do `mul!(du,A,u)`

---

<div class="post-metadata">

**Author:** ![nilsbecker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilsbecker/32/12157_2.png) [@nilsbecker](https://discourse.julialang.org/u/nilsbecker)\
**Post date:** [September 20, 2019, 11:58am UTC](https://discourse.julialang.org/t/test-diffeqgpu-errors-with-unsupported-call-to-the-julia-runtime/28893/16 "2019-09-20T11:58:19Z")

</div>

ah that makes sense. it’s not always obvious how clever a language is trying to be… thanks again!

---

<div class="post-metadata">

**Author:** ![nilsbecker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilsbecker/32/12157_2.png) [@nilsbecker](https://discourse.julialang.org/u/nilsbecker)\
**Post date:** [September 20, 2019, 1:55pm UTC](https://discourse.julialang.org/t/test-diffeqgpu-errors-with-unsupported-call-to-the-julia-runtime/28893/17 "2019-09-20T13:55:55Z")

</div>

still no luck. sorry for keeping you busy, but there might be a chance that this is not user error?

the following is a self contained minimal version which i tried to make non-allocating. also, the `prob_func` is not just a no-nothing function to exclude anything there (does that also have to be non-allocating?)

```julia
using LinearAlgebra, DifferentialEquations, DiffEqGPU
A = Float32[
    1. 0 0 -5; 4 -2 4 -3; -4 0 0 1; 5 -2 2 3]
u0 = Array{Float32}(rand(4,2))
tspan = (0.0f0,1.0f0)
function fip(du,u,p,t) 
   mul!(du, A, u)
end
probip = ODEProblem(fip,u0,tspan)
function prob_func(prob, i, repeat)
    prob
end
ensprobip=EnsembleProblem(probip, prob_func=prob_func)
trajs = 123
simgpu = solve(ensprobip, Tsit5(), EnsembleGPUArray(), trajectories=trajs)

```

when i run it i get a similar stack trace:

```julia
InvalidIRError: compiling gpu_kernel(Cassette.Context{nametype(Ctx),Nothing,Nothing,getfield(GPUifyLoops, Symbol("##PassType#371")),Nothing,Cassette.DisableHooks}, typeof(DiffEqGPU.gpu_kernel), ODEFunction{true,typeof(fip),UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing}, CUDAnative.CuDeviceArray{Float32,2,CUDAnative.AS.Global}, CUDAnative.CuDeviceArray{Float32,2,CUDAnative.AS.Global}, CUDAnative.CuDeviceArray{DiffEqBase.NullParameters,2,CUDAnative.AS.Global}, Float32) resulted in invalid LLVM IR
Reason: unsupported dynamic function invocation (call to overdub)
Stacktrace:
 [1] fip at In[1]:10
 [2] ODEFunction at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqBase/8uyX3/src/diffeqfunction.jl:230
 [3] gpu_kernel at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqGPU/QB1WC/src/DiffEqGPU.jl:6
 [4] overdub at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/Cassette/YCOeN/src/overdub.jl:0

Stacktrace:
 [1] check_ir(::CUDAnative.CompilerJob, ::LLVM.Module) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/CUDAnative/UWBIY/src/compiler/validation.jl:114
 [2] macro expansion at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/CUDAnative/UWBIY/src/compiler/driver.jl:188 [inlined]
 [3] macro expansion at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/TimerOutputs/7zSea/src/TimerOutput.jl:216 [inlined]
 [4] #codegen#130(::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::typeof(CUDAnative.codegen), ::Symbol, ::CUDAnative.CompilerJob) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/CUDAnative/UWBIY/src/compiler/driver.jl:186
 [5] #codegen at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/CUDAnative/UWBIY/src/compiler/driver.jl:0 [inlined]
 [6] #compile#129(::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::typeof(CUDAnative.compile), ::Symbol, ::CUDAnative.CompilerJob) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/CUDAnative/UWBIY/src/compiler/driver.jl:47
 [7] #compile at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/CUDAnative/UWBIY/src/compiler/common.jl:0 [inlined]
 [8] #compile#128 at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/CUDAnative/UWBIY/src/compiler/driver.jl:28 [inlined]
 [9] #compile at ./none:0 [inlined] (repeats 2 times)
 [10] macro expansion at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/CUDAnative/UWBIY/src/execution.jl:389 [inlined]
 [11] #cufunction#170(::String, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(CUDAnative.cufunction), ::typeof(Cassette.overdub), ::Type{Tuple{Cassette.Context{nametype(Ctx),Nothing,Nothing,getfield(GPUifyLoops, Symbol("##PassType#371")),Nothing,Cassette.DisableHooks},typeof(DiffEqGPU.gpu_kernel),ODEFunction{true,typeof(fip),UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},CUDAnative.CuDeviceArray{Float32,2,CUDAnative.AS.Global},CUDAnative.CuDeviceArray{Float32,2,CUDAnative.AS.Global},CUDAnative.CuDeviceArray{DiffEqBase.NullParameters,2,CUDAnative.AS.Global},Float32}}) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/CUDAnative/UWBIY/src/execution.jl:357
 [12] (::getfield(CUDAnative, Symbol("#kw##cufunction")))(::NamedTuple{(:name,),Tuple{String}}, ::typeof(CUDAnative.cufunction), ::Function, ::Type) at ./none:0
 [13] #launch#50(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(GPUifyLoops.launch), ::GPUifyLoops.CUDA, ::typeof(DiffEqGPU.gpu_kernel), ::Function, ::Vararg{Any,N} where N) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/GPUifyLoops/mjszO/src/GPUifyLoops.jl:125
 [14] launch at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/GPUifyLoops/mjszO/src/GPUifyLoops.jl:119 [inlined]
 [15] macro expansion at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/GPUifyLoops/mjszO/src/GPUifyLoops.jl:54 [inlined]
 [16] #12 at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqGPU/QB1WC/src/DiffEqGPU.jl:61 [inlined]
 [17] ODEFunction at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqBase/8uyX3/src/diffeqfunction.jl:230 [inlined]
 [18] initialize!(::OrdinaryDiffEq.ODEIntegrator{Tsit5,true,CuArrays.CuArray{Float32,2},Float32,CuArrays.CuArray{DiffEqBase.NullParameters,2},Float32,Float32,Float32,Array{CuArrays.CuArray{Float32,2},1},ODESolution{Float32,3,Array{CuArrays.CuArray{Float32,2},1},Nothing,Nothing,Array{Float32,1},Array{Array{CuArrays.CuArray{Float32,2},1},1},ODEProblem{CuArrays.CuArray{Float32,2},Tuple{Float32,Float32},true,CuArrays.CuArray{DiffEqBase.NullParameters,2},ODEFunction{true,getfield(DiffEqGPU, Symbol("##12#22")){ODEFunction{true,typeof(fip),UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing}},UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},Tsit5,OrdinaryDiffEq.InterpolationData{ODEFunction{true,getfield(DiffEqGPU, Symbol("##12#22")){ODEFunction{true,typeof(fip),UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing}},UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Array{CuArrays.CuArray{Float32,2},1},Array{Float32,1},Array{Array{CuArrays.CuArray{Float32,2},1},1},OrdinaryDiffEq.Tsit5Cache{CuArrays.CuArray{Float32,2},CuArrays.CuArray{Float32,2},CuArrays.CuArray{Float32,2},OrdinaryDiffEq.Tsit5ConstantCache{Float32,Float32}}},DiffEqBase.DEStats},ODEFunction{true,getfield(DiffEqGPU, Symbol("##12#22")){ODEFunction{true,typeof(fip),UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing}},UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},OrdinaryDiffEq.Tsit5Cache{CuArrays.CuArray{Float32,2},CuArrays.CuArray{Float32,2},CuArrays.CuArray{Float32,2},OrdinaryDiffEq.Tsit5ConstantCache{Float32,Float32}},OrdinaryDiffEq.DEOptions{Float32,Float32,Float32,Float32,typeof(DiffEqBase.ODE_DEFAULT_NORM),typeof(opnorm),CallbackSet{Tuple{},Tuple{}},typeof(DiffEqBase.ODE_DEFAULT_ISOUTOFDOMAIN),typeof(DiffEqBase.ODE_DEFAULT_PROG_MESSAGE),typeof(DiffEqBase.ODE_DEFAULT_UNSTABLE_CHECK),DataStructures.BinaryHeap{Float32,DataStructures.LessThan},DataStructures.BinaryHeap{Float32,DataStructures.LessThan},Nothing,Nothing,Int64,Array{Float32,1},Array{Float32,1},Array{Float32,1}},CuArrays.CuArray{Float32,2},Float32,Nothing}, ::OrdinaryDiffEq.Tsit5Cache{CuArrays.CuArray{Float32,2},CuArrays.CuArray{Float32,2},CuArrays.CuArray{Float32,2},OrdinaryDiffEq.Tsit5ConstantCache{Float32,Float32}}) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/OrdinaryDiffEq/tQd6p/src/perform_step/low_order_rk_perform_step.jl:623
 [19] #__init#335(::Array{Float32,1}, ::Array{Float32,1}, ::Array{Float32,1}, ::Nothing, ::Bool, ::Bool, ::Bool, ::Bool, ::Nothing, ::Bool, ::Bool, ::Float32, ::Float32, ::Float32, ::Bool, ::Bool, ::Rational{Int64}, ::Nothing, ::Nothing, ::Rational{Int64}, ::Int64, ::Int64, ::Int64, ::Rational{Int64}, ::Bool, ::Int64, ::Nothing, ::Nothing, ::Int64, ::typeof(DiffEqBase.ODE_DEFAULT_NORM), ::typeof(opnorm), ::typeof(DiffEqBase.ODE_DEFAULT_ISOUTOFDOMAIN), ::typeof(DiffEqBase.ODE_DEFAULT_UNSTABLE_CHECK), ::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::Int64, ::String, ::typeof(DiffEqBase.ODE_DEFAULT_PROG_MESSAGE), ::Nothing, ::Bool, ::Bool, ::Bool, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(DiffEqBase.__init), ::ODEProblem{CuArrays.CuArray{Float32,2},Tuple{Float32,Float32},true,CuArrays.CuArray{DiffEqBase.NullParameters,2},ODEFunction{true,getfield(DiffEqGPU, Symbol("##12#22")){ODEFunction{true,typeof(fip),UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing}},UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem}, ::Tsit5, ::Array{CuArrays.CuArray{Float32,2},1}, ::Array{Float32,1}, ::Array{Any,1}, ::Type{Val{true}}) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/OrdinaryDiffEq/tQd6p/src/solve.jl:352
 [20] __init(::ODEProblem{CuArrays.CuArray{Float32,2},Tuple{Float32,Float32},true,CuArrays.CuArray{DiffEqBase.NullParameters,2},ODEFunction{true,getfield(DiffEqGPU, Symbol("##12#22")){ODEFunction{true,typeof(fip),UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing}},UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem}, ::Tsit5, ::Array{CuArrays.CuArray{Float32,2},1}, ::Array{Float32,1}, ::Array{Any,1}, ::Type{Val{true}}) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/OrdinaryDiffEq/tQd6p/src/solve.jl:66 (repeats 4 times)
 [21] #__solve#334 at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/OrdinaryDiffEq/tQd6p/src/solve.jl:4 [inlined]
 [22] __solve at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/OrdinaryDiffEq/tQd6p/src/solve.jl:4 [inlined]
 [23] #solve_call#425(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(DiffEqBase.solve_call), ::ODEProblem{CuArrays.CuArray{Float32,2},Tuple{Float32,Float32},true,CuArrays.CuArray{DiffEqBase.NullParameters,2},ODEFunction{true,getfield(DiffEqGPU, Symbol("##12#22")){ODEFunction{true,typeof(fip),UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing}},UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem}, ::Tsit5) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqBase/8uyX3/src/solve.jl:40
 [24] solve_call at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqBase/8uyX3/src/solve.jl:37 [inlined]
 [25] #solve#426 at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqBase/8uyX3/src/solve.jl:57 [inlined]
 [26] solve(::ODEProblem{CuArrays.CuArray{Float32,2},Tuple{Float32,Float32},true,CuArrays.CuArray{DiffEqBase.NullParameters,2},ODEFunction{true,getfield(DiffEqGPU, Symbol("##12#22")){ODEFunction{true,typeof(fip),UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing}},UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem}, ::Tsit5) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqBase/8uyX3/src/solve.jl:45
 [27] #batch_solve#5(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(DiffEqGPU.batch_solve), ::EnsembleProblem{ODEProblem{Array{Float32,2},Tuple{Float32,Float32},true,DiffEqBase.NullParameters,ODEFunction{true,typeof(fip),UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},typeof(prob_func),getfield(DiffEqBase, Symbol("##332#338")),getfield(DiffEqBase, Symbol("##334#340")),Array{Any,1}}, ::Tsit5, ::EnsembleGPUArray, ::UnitRange{Int64}) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqGPU/QB1WC/src/DiffEqGPU.jl:66
 [28] batch_solve at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqGPU/QB1WC/src/DiffEqGPU.jl:45 [inlined]
 [29] (::getfield(DiffEqGPU, Symbol("##3#4")){Int64,Int64,Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},EnsembleProblem{ODEProblem{Array{Float32,2},Tuple{Float32,Float32},true,DiffEqBase.NullParameters,ODEFunction{true,typeof(fip),UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},typeof(prob_func),getfield(DiffEqBase, Symbol("##332#338")),getfield(DiffEqBase, Symbol("##334#340")),Array{Any,1}},Tsit5,EnsembleGPUArray})(::Int64) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqGPU/QB1WC/src/DiffEqGPU.jl:37
 [30] iterate at ./generator.jl:47 [inlined]
 [31] _collect(::UnitRange{Int64}, ::Base.Generator{UnitRange{Int64},getfield(DiffEqGPU, Symbol("##3#4")){Int64,Int64,Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},EnsembleProblem{ODEProblem{Array{Float32,2},Tuple{Float32,Float32},true,DiffEqBase.NullParameters,ODEFunction{true,typeof(fip),UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},typeof(prob_func),getfield(DiffEqBase, Symbol("##332#338")),getfield(DiffEqBase, Symbol("##334#340")),Array{Any,1}},Tsit5,EnsembleGPUArray}}, ::Base.EltypeUnknown, ::Base.HasShape{1}) at ./array.jl:619
 [32] collect_similar(::UnitRange{Int64}, ::Base.Generator{UnitRange{Int64},getfield(DiffEqGPU, Symbol("##3#4")){Int64,Int64,Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},EnsembleProblem{ODEProblem{Array{Float32,2},Tuple{Float32,Float32},true,DiffEqBase.NullParameters,ODEFunction{true,typeof(fip),UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},typeof(prob_func),getfield(DiffEqBase, Symbol("##332#338")),getfield(DiffEqBase, Symbol("##334#340")),Array{Any,1}},Tsit5,EnsembleGPUArray}}) at ./array.jl:548
 [33] map(::Function, ::UnitRange{Int64}) at ./abstractarray.jl:2073
 [34] macro expansion at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqGPU/QB1WC/src/DiffEqGPU.jl:31 [inlined]
 [35] macro expansion at ./util.jl:213 [inlined]
 [36] #__solve#2(::Int64, ::Int64, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(DiffEqBase.__solve), ::EnsembleProblem{ODEProblem{Array{Float32,2},Tuple{Float32,Float32},true,DiffEqBase.NullParameters,ODEFunction{true,typeof(fip),UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},typeof(prob_func),getfield(DiffEqBase, Symbol("##332#338")),getfield(DiffEqBase, Symbol("##334#340")),Array{Any,1}}, ::Tsit5, ::EnsembleGPUArray) at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqGPU/QB1WC/src/DiffEqGPU.jl:30
 [37] #__solve at ./none:0 [inlined]
 [38] #solve#427 at /home/bq_nbecker/isi/julia_pkg/GPUTest/packages/DiffEqBase/8uyX3/src/solve.jl:64 [inlined]
 [39] (::getfield(DiffEqBase, Symbol("#kw##solve")))(::NamedTuple{(:trajectories,),Tuple{Int64}}, ::typeof(solve), ::EnsembleProblem{ODEProblem{Array{Float32,2},Tuple{Float32,Float32},true,DiffEqBase.NullParameters,ODEFunction{true,typeof(fip),UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},typeof(prob_func),getfield(DiffEqBase, Symbol("##332#338")),getfield(DiffEqBase, Symbol("##334#340")),Array{Any,1}}, ::Tsit5, ::EnsembleGPUArray) at ./none:0
 [40] top-level scope at In[3]:15

```

---

<div class="post-metadata">

**Author:** ![nilsbecker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilsbecker/32/12157_2.png) [@nilsbecker](https://discourse.julialang.org/u/nilsbecker)\
**Post date:** [September 20, 2019, 2:22pm UTC](https://discourse.julialang.org/t/test-diffeqgpu-errors-with-unsupported-call-to-the-julia-runtime/28893/18 "2019-09-20T14:22:31Z")

</div>

another thought: this example is unusual in that `u` is not a vector but a 4x2 matrix. maybe that confuses the compilation? it sure confuses the plot recipes…

no, it’s not that. if i change `u0` to be a length-4 vector, the same error results

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [September 20, 2019, 2:38pm UTC](https://discourse.julialang.org/t/test-diffeqgpu-errors-with-unsupported-call-to-the-julia-runtime/28893/19 "2019-09-20T14:38:10Z")

</div>

Open an issue for that. I am not sure why this one doesn’t compile. `Reason: unsupported dynamic function invocation (call to overdub)` is odd since I don’t see why this needs another overdub.

---

<div class="post-metadata">

**Author:** ![nilsbecker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilsbecker/32/12157_2.png) [@nilsbecker](https://discourse.julialang.org/u/nilsbecker)\
**Post date:** [September 20, 2019, 2:39pm UTC](https://discourse.julialang.org/t/test-diffeqgpu-errors-with-unsupported-call-to-the-julia-runtime/28893/20 "2019-09-20T14:39:11Z")

</div>

ok.