Matplotlib rcParams in module __init__() function segfault on M1

General Usage
,
1

Hi, I am converting a package from PyCall.jl to PythonCall.jl. (I am planning on calling Julia from Python in the future, and I want to test whether to use PythonCall/JuliaCall or PyCall/pyjulia. PythonCall/JuliaCall appears to be the future, so I’m pursuing that route now. However, I had a similar problem with PyCall - I just never bothered to investigate.)

So I have two packages: MyPlotExtras.jl and another that loads MyPlotExtras. A minimal example is:

module MyPlotExtras

using PythonPlot

function __init__()
    rcParams = PythonPlot.matplotlib.rcParams
end

end

and

module TestMyPlotExtras

using MyPlotExtras

end

To reproduce the problem it is important to dev the MyPlotExtras.jl package in the TestMyPlotExtras.jl package, or to have it registered in a local registry.

I am probably failing at explaining this properly, so I made a very small git repo to illustrate: GitHub - hsgg/TestMyPlotExtras: Testing repository to show how loading the PlotExtras package fails with PythonCall.jl

Only then do I get:

julia> using TestMyPlotExtras
[11742] signal (11.2): Segmentation fault: 11
in expression starting at /Users/henry/TestMyPlotExtras/TestMyPlotExtras/src/TestMyPlotExtras.jl:3
PyObject_GetAttr at /Users/henry/TestMyPlotExtras/TestMyPlotExtras/.CondaPkg/env/lib/libpython3.11.dylib (unknown line)
PyObject_GetAttr at /Users/henry/.julia/packages/PythonCall/qTEA1/src/cpython/pointers.jl:299 [inlined]
macro expansion at /Users/henry/.julia/packages/PythonCall/qTEA1/src/Py.jl:131 [inlined]
pygetattr at /Users/henry/.julia/packages/PythonCall/qTEA1/src/abstract/object.jl:60
getproperty at /Users/henry/.julia/packages/PythonCall/qTEA1/src/Py.jl:261 [inlined]
__init__ at /Users/henry/TestMyPlotExtras/MyPlotExtras/src/MyPlotExtras.jl:6
jl_sysimg_fvars_base at /Users/henry/.julia/compiled/v1.9/MyPlotExtras/dkBtT_GM8Uw.dylib (unknown line)
ijl_apply_generic at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.9.dylib (unknown line)
jl_module_run_initializer at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.9.dylib (unknown line)
ijl_init_restored_modules at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.9.dylib (unknown line)
register_restored_modules at ./loading.jl:1115
_include_from_serialized at ./loading.jl:1061
_require_search_from_serialized at ./loading.jl:1506
_require at ./loading.jl:1783
_require_prelocked at ./loading.jl:1660
macro expansion at ./loading.jl:1648 [inlined]
macro expansion at ./lock.jl:267 [inlined]
require at ./loading.jl:1611
jfptr_require_55044 at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
ijl_apply_generic at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.9.dylib (unknown line)
eval_import_path at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.9.dylib (unknown line)
jl_toplevel_eval_flex at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.9.dylib (unknown line)
jl_toplevel_eval_flex at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.9.dylib (unknown line)
jl_toplevel_eval_flex at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.9.dylib (unknown line)
ijl_toplevel_eval_in at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.9.dylib (unknown line)
eval at ./boot.jl:370 [inlined]
include_string at ./loading.jl:1903
ijl_apply_generic at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.9.dylib (unknown line)
_include at ./loading.jl:1963
include at ./Base.jl:457 [inlined]
include_package_for_output at ./loading.jl:2049
jfptr_include_package_for_output_43027 at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
ijl_apply_generic at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.9.dylib (unknown line)
do_call at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.9.dylib (unknown line)
eval_body at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.9.dylib (unknown line)
jl_interpret_toplevel_thunk at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.9.dylib (unknown line)
jl_toplevel_eval_flex at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.9.dylib (unknown line)
jl_toplevel_eval_flex at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.9.dylib (unknown line)
ijl_toplevel_eval_in at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.9.dylib (unknown line)
eval at ./boot.jl:370 [inlined]
include_string at ./loading.jl:1903
include_string at ./loading.jl:1913 [inlined]
exec_options at ./client.jl:305
_start at ./client.jl:522
jfptr__start_51456 at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
ijl_apply_generic at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.9.dylib (unknown line)
true_main at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.9.dylib (unknown line)
jl_repl_entrypoint at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.9.dylib (unknown line)
Allocations: 4925588 (Pool: 4922287; Big: 3301); GC: 7
ERROR: Failed to precompile TestMyPlotExtras [679a42ec-c8f9-48a5-807c-1ce16acee5ed] to "/Users/henry/.julia/compiled/v1.9/TestMyPlotExtras/jl_IJiDWn".
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
   @ Base ./loading.jl:2300
 [3] compilecache
   @ ./loading.jl:2167 [inlined]
 [4] _require(pkg::Base.PkgId, env::String)
   @ Base ./loading.jl:1805
 [5] _require_prelocked(uuidkey::Base.PkgId, env::String)
   @ Base ./loading.jl:1660
 [6] macro expansion
   @ ./loading.jl:1648 [inlined]
 [7] macro expansion
   @ ./lock.jl:267 [inlined]
 [8] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1611

I’m on Apple M1:

julia> versioninfo()
Julia Version 1.9.3
Commit bed2cd540a1 (2023-08-24 14:43 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: macOS (arm64-apple-darwin22.4.0)
  CPU: 10 × Apple M1 Max
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, apple-m1)
  Threads: 1 on 8 virtual cores
2

Digging further, there is a workaround that PythonPlot.jl uses. Adding that to skip precompilation of __ini__() in MyPlotExtras does seem to work. However, the PythonCall.jl author seems to say that is not a good solution.

What is a good solution?

3

If you don’t want to use the ugly ccall(:jl_generating_output, ...) to check whether Julia is precompiling, you can just check whether PythonPlot.matplotlib is initialized:

PythonCall.pyisnull(PythonPlot.matplotlib) && return
1 Like
4

The reason you get this crash is because PythonPlot skips __init__ when precompiling. This means that the matplotlib object is not properly initialised when you access it in your own __init__ function, hence the crash.

AFAIU skipping __init__ is an antipattern, precisely for this sort of reason: as soon as one package skips __init__ then all downstream packages need to as well, because downstream __init__ functions are allowed to assume all upstream packages are properly initialised. This is why I refused to do it in PythonCall.

@stevengj why does PythonPlot do this?

5

Also a direct solution to your problem is to call pyimport("matplotlib") instead of PythonPlot.matplotlib.

6

It was to work around `using PyPlot` crashes package precompilation on Julia v1.10.0-beta1 · Issue #572 · JuliaPy/PyPlot.jl · GitHub, IIRC. Probably there is a better fix, e.g. initialize matplotlib with a non-interactive backend (Agg) when precompiling.