I am experiencing a strange Julia crash when I import BetaML before computing a gradient operation with Zygote, but if I first do this operation, then I import BetaML (or using BetaML) and then do again the gradient operation with Zygote it works fine.
This on Julia < 1.6, in Julia 1.6 it works in both cases.
# --------------------------------------------------------------------------
# OK
using Statistics, Zygote;
function autoJacobianDifferentName(f,x;nY=length(f(x)))
x = convert(Array{Float64,1},x)
j = Array{Float64, 2}(undef, nY, size(x,1))
for i in 1:nY
j[i,:] = gradient(x -> f(x)[i], x)[1]'
end
return j
end;
pool1dDifferentName(x,poolSize=3;f=mean) = [f(x[i:i+poolSize-1]) for i in 1:length(x)-poolSize+1];
f = (x->pool1dDifferentName(x,2,f=mean));
v = f([1,2,3,4,5]);
J = autoJacobianDifferentName(f,[1,2,3,4,5])
import BetaML
J = autoJacobianDifferentName(f,[1,2,3,4,5])
# --------------------------------------------------------------------------
# Crash in Julia < 1.6, but ok in Julia 1.6
using Statistics, Zygote;
function autoJacobianDifferentName(f,x;nY=length(f(x)))
x = convert(Array{Float64,1},x)
j = Array{Float64, 2}(undef, nY, size(x,1))
for i in 1:nY
j[i,:] = gradient(x -> f(x)[i], x)[1]'
end
return j
end;
pool1dDifferentName(x,poolSize=3;f=mean) = [f(x[i:i+poolSize-1]) for i in 1:length(x)-poolSize+1] ;
f = (x->pool1dDifferentName(x,2,f=mean));
v = f([1,2,3,4,5]);
import BetaML
J = autoJacobianDifferentName(f,[1,2,3,4,5])
As I am experiencing the problem with this specific function, which is the preferred way to deal with this situation ? I would avoid to limit my package to Julia 1.6… could I skip the test for lower versions of Julia and warn users in the docstring that this function require Julia 1.6 ? Or there is a better approach ?
This is the complete stack:
lobianco@NCY-BETA-ALOBIANCO:~/.julia/dev/BetaML$ julia1.5
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.5.3 (2020-11-09)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
(@v1.5) pkg> activate --temp
Activating new environment at `/tmp/jl_lsGMKp/Project.toml`
(jl_lsGMKp) pkg> add Zygote, BetaML
Updating registry at `~/.julia/registries/General`
######################################################################## 100,0%
Resolving package versions...
Updating `/tmp/jl_lsGMKp/Project.toml`
[024491cd] + BetaML v0.4.1
[e88e6eb3] + Zygote v0.6.8
Updating `/tmp/jl_lsGMKp/Manifest.toml`
[621f4979] + AbstractFFTs v1.0.1
[56f22d72] + Artifacts v1.3.0
[024491cd] + BetaML v0.4.1
[324d7699] + CategoricalArrays v0.9.5
[082447d4] + ChainRules v0.7.59
[d360d2e6] + ChainRulesCore v0.9.37
[861a8166] + Combinatorics v1.0.2
[bbf7d656] + CommonSubexpressions v0.3.0
[34da2185] + Compat v3.26.0
[e66e0078] + CompilerSupportLibraries_jll v0.3.4+0
[9a962f9c] + DataAPI v1.6.0
[864edb3b] + DataStructures v0.18.9
[163ba53b] + DiffResults v1.0.3
[b552c78f] + DiffRules v1.0.2
[31c24e10] + Distributions v0.24.15
[1a297f60] + FillArrays v0.11.7
[f6369f11] + ForwardDiff v0.10.18
[7869d1d1] + IRTools v0.4.2
[692b3bcd] + JLLWrappers v1.2.0
[682c06a0] + JSON v0.21.1
[e80e1ace] + MLJModelInterface v0.4.1
[1914dd2f] + MacroTools v0.5.6
[e1d29d7a] + Missings v0.4.5
[77ba4419] + NaNMath v0.3.5
[efe28fd5] + OpenSpecFun_jll v0.5.3+4
[bac558e1] + OrderedCollections v1.4.0
[90014a1f] + PDMats v0.11.0
[69de0a69] + Parsers v1.1.0
[92933f4c] + ProgressMeter v1.5.0
[1fd47b50] + QuadGK v2.4.1
[189a3867] + Reexport v1.0.0
[ae029012] + Requires v1.1.3
[79098fc4] + Rmath v0.6.1
[f50d1b31] + Rmath_jll v0.2.2+2
[321657f4] + ScientificTypes v1.1.1
[a2af1166] + SortingAlgorithms v0.3.1
[276daf66] + SpecialFunctions v1.3.0
[90137ffa] + StaticArrays v1.1.0
[64bff920] + StatisticalTraits v0.1.1
[2913bbd2] + StatsBase v0.33.5
[4c63d2b9] + StatsFuns v0.9.7
[856f2bd8] + StructTypes v1.5.2
[e88e6eb3] + Zygote v0.6.8
[700de1a5] + ZygoteRules v0.2.1
[2a0f44e3] + Base64
[ade2ca70] + Dates
[8bb1440f] + DelimitedFiles
[8ba89e20] + Distributed
[9fa8497b] + Future
[b77e0a4c] + InteractiveUtils
[76f85450] + LibGit2
[8f399da3] + Libdl
[37e2e46d] + LinearAlgebra
[56ddb016] + Logging
[d6f4376e] + Markdown
[a63ad114] + Mmap
[44cfe95a] + Pkg
[de0858da] + Printf
[3fa0cd96] + REPL
[9a3f8284] + Random
[ea8e919c] + SHA
[9e88b42a] + Serialization
[1a1011a3] + SharedArrays
[6462fe0b] + Sockets
[2f01184e] + SparseArrays
[10745b16] + Statistics
[4607b0f0] + SuiteSparse
[8dfed614] + Test
[cf7118a7] + UUIDs
[4ec0a83e] + Unicode
julia> using Statistics, Zygote;
[ Info: Precompiling Zygote [e88e6eb3-aa80-5325-afca-941959d7151f]
julia> function autoJacobianDifferentName(f,x;nY=length(f(x)))
x = convert(Array{Float64,1},x)
j = Array{Float64, 2}(undef, nY, size(x,1))
for i in 1:nY
j[i,:] = gradient(x -> f(x)[i], x)[1]'
end
return j
end;
julia> pool1dDifferentName(x,poolSize=3;f=mean) = [f(x[i:i+poolSize-1]) for i in 1:length(x)-poolSize+1] ;
julia> f = (x->pool1dDifferentName(x,2,f=mean));
julia> v = f([1,2,3,4,5]);
julia> import BetaML
[ Info: Precompiling BetaML [024491cd-cc6b-443e-8034-08ea7eb7db2b]
julia> J = autoJacobianDifferentName(f,[1,2,3,4,5])
Unreachable reached at 0x7fdbb0920d30
signal (4): Illegal instruction
in expression starting at REPL[9]:1
prune! at /home/lobianco/.julia/packages/IRTools/aSVI5/src/passes/passes.jl:189
|> at ./operators.jl:834 [inlined]
#IR#15 at /home/lobianco/.julia/packages/IRTools/aSVI5/src/ir/wrap.jl:146
IR at /home/lobianco/.julia/packages/IRTools/aSVI5/src/ir/wrap.jl:142 [inlined]
_lookup_grad at /home/lobianco/.julia/packages/Zygote/lwmfx/src/compiler/emit.jl:101
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2214 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2398
#s3004#1176 at /home/lobianco/.julia/packages/Zygote/lwmfx/src/compiler/interface2.jl:20 [inlined]
#s3004#1176 at ./none:0
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2214 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2398
GeneratedFunctionStub at ./boot.jl:527
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2214 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2398
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1690 [inlined]
jl_call_staged at /buildworker/worker/package_linux64/build/src/method.c:376
jl_code_for_staged at /buildworker/worker/package_linux64/build/src/method.c:423
get_staged at ./compiler/utilities.jl:94
retrieve_code_info at ./compiler/utilities.jl:105 [inlined]
InferenceState at ./compiler/inferencestate.jl:118
typeinf_edge at ./compiler/typeinfer.jl:475
abstract_call_method at ./compiler/abstractinterpretation.jl:419
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:111
abstract_call_known at ./compiler/abstractinterpretation.jl:904
abstract_call at ./compiler/abstractinterpretation.jl:926
abstract_call at ./compiler/abstractinterpretation.jl:911
abstract_eval at ./compiler/abstractinterpretation.jl:1005
typeinf_local at ./compiler/abstractinterpretation.jl:1270
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1326
typeinf at ./compiler/typeinfer.jl:12
typeinf_edge at ./compiler/typeinfer.jl:484
abstract_call_method at ./compiler/abstractinterpretation.jl:419
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:111
abstract_call_known at ./compiler/abstractinterpretation.jl:904
abstract_call at ./compiler/abstractinterpretation.jl:926
abstract_call at ./compiler/abstractinterpretation.jl:911
abstract_eval at ./compiler/abstractinterpretation.jl:1005
typeinf_local at ./compiler/abstractinterpretation.jl:1270
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1326
typeinf at ./compiler/typeinfer.jl:12
typeinf_edge at ./compiler/typeinfer.jl:484
abstract_call_method at ./compiler/abstractinterpretation.jl:419
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:111
abstract_call_known at ./compiler/abstractinterpretation.jl:904
abstract_call at ./compiler/abstractinterpretation.jl:926
abstract_apply at ./compiler/abstractinterpretation.jl:635
abstract_call_known at ./compiler/abstractinterpretation.jl:704
abstract_call at ./compiler/abstractinterpretation.jl:926
abstract_call at ./compiler/abstractinterpretation.jl:911
abstract_eval at ./compiler/abstractinterpretation.jl:1005
typeinf_local at ./compiler/abstractinterpretation.jl:1270
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1326
typeinf at ./compiler/typeinfer.jl:12
typeinf_edge at ./compiler/typeinfer.jl:484
abstract_call_method at ./compiler/abstractinterpretation.jl:419
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:111
abstract_call_known at ./compiler/abstractinterpretation.jl:904
abstract_call at ./compiler/abstractinterpretation.jl:926
abstract_apply at ./compiler/abstractinterpretation.jl:635
abstract_call_known at ./compiler/abstractinterpretation.jl:704
abstract_call at ./compiler/abstractinterpretation.jl:926
abstract_call at ./compiler/abstractinterpretation.jl:911
abstract_eval at ./compiler/abstractinterpretation.jl:1005
typeinf_local at ./compiler/abstractinterpretation.jl:1270
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1326
typeinf at ./compiler/typeinfer.jl:12
typeinf_edge at ./compiler/typeinfer.jl:484
abstract_call_method at ./compiler/abstractinterpretation.jl:419
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:111
abstract_call_known at ./compiler/abstractinterpretation.jl:904
abstract_call at ./compiler/abstractinterpretation.jl:926
abstract_apply at ./compiler/abstractinterpretation.jl:635
abstract_call_known at ./compiler/abstractinterpretation.jl:704
abstract_call at ./compiler/abstractinterpretation.jl:926
abstract_call at ./compiler/abstractinterpretation.jl:911
abstract_eval at ./compiler/abstractinterpretation.jl:1005
typeinf_local at ./compiler/abstractinterpretation.jl:1270
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1326
typeinf at ./compiler/typeinfer.jl:12
typeinf_edge at ./compiler/typeinfer.jl:484
abstract_call_method at ./compiler/abstractinterpretation.jl:419
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:111
abstract_call_known at ./compiler/abstractinterpretation.jl:904
abstract_call at ./compiler/abstractinterpretation.jl:926
abstract_call at ./compiler/abstractinterpretation.jl:911
abstract_eval at ./compiler/abstractinterpretation.jl:1005
typeinf_local at ./compiler/abstractinterpretation.jl:1270
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1326
typeinf at ./compiler/typeinfer.jl:12
typeinf_edge at ./compiler/typeinfer.jl:484
abstract_call_method at ./compiler/abstractinterpretation.jl:419
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:111
abstract_call_known at ./compiler/abstractinterpretation.jl:904
abstract_call at ./compiler/abstractinterpretation.jl:926
abstract_call at ./compiler/abstractinterpretation.jl:911
abstract_eval at ./compiler/abstractinterpretation.jl:1005
typeinf_local at ./compiler/abstractinterpretation.jl:1270
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1326
typeinf at ./compiler/typeinfer.jl:12
typeinf_ext at ./compiler/typeinfer.jl:570
typeinf_ext at ./compiler/typeinfer.jl:601
jfptr_typeinf_ext_23344.clone_1 at /home/lobianco/lib/julia-1.5.3/lib/julia/sys.so (unknown line)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2214 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2398
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1690 [inlined]
jl_type_infer at /buildworker/worker/package_linux64/build/src/gf.c:296
jl_generate_fptr at /buildworker/worker/package_linux64/build/src/jitlayers.cpp:290
jl_compile_method_internal at /buildworker/worker/package_linux64/build/src/gf.c:1964
jl_compile_method_internal at /buildworker/worker/package_linux64/build/src/gf.c:1919 [inlined]
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2224 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2398
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1690 [inlined]
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:117
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:206
eval_stmt_value at /buildworker/worker/package_linux64/build/src/interpreter.c:157 [inlined]
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:566
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:660
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:840
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:790
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/toplevel.c:883
eval at ./boot.jl:331
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2214 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2398
eval_user_input at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:134
repl_backend_loop at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:195
start_repl_backend at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:180
#run_repl#37 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:292
run_repl at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:288
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2231 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2398
#807 at ./client.jl:399
jfptr_YY.807_45002.clone_1 at /home/lobianco/lib/julia-1.5.3/lib/julia/sys.so (unknown line)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2214 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2398
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1690 [inlined]
do_apply at /buildworker/worker/package_linux64/build/src/builtins.c:655
jl_f__apply_latest at /buildworker/worker/package_linux64/build/src/builtins.c:705
#invokelatest#1 at ./essentials.jl:710 [inlined]
invokelatest at ./essentials.jl:709 [inlined]
run_main_repl at ./client.jl:383
exec_options at ./client.jl:313
_start at ./client.jl:506
jfptr__start_53898.clone_1 at /home/lobianco/lib/julia-1.5.3/lib/julia/sys.so (unknown line)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2214 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2398
unknown function (ip: 0x401931)
unknown function (ip: 0x401533)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x4015d4)
Allocations: 76029324 (Pool: 76012494; Big: 16830); GC: 77
Illegal instruction (core dumped)