Matplotlib rcParams in module __init__() function segfault on M1

Hi, I am converting a package from PyCall.jl to PythonCall.jl. (I am planning on calling Julia from Python in the future, and I want to test whether to use PythonCall/JuliaCall or PyCall/pyjulia. PythonCall/JuliaCall appears to be the future, so I’m pursuing that route now. However, I had a similar problem with PyCall - I just never bothered to investigate.)

So I have two packages: MyPlotExtras.jl and another that loads MyPlotExtras. A minimal example is:

module MyPlotExtras

using PythonPlot

function __init__()
    rcParams = PythonPlot.matplotlib.rcParams



module TestMyPlotExtras

using MyPlotExtras


To reproduce the problem it is important to dev the MyPlotExtras.jl package in the TestMyPlotExtras.jl package, or to have it registered in a local registry.

I am probably failing at explaining this properly, so I made a very small git repo to illustrate: GitHub - hsgg/TestMyPlotExtras: Testing repository to show how loading the PlotExtras package fails with PythonCall.jl

Only then do I get:

julia> using TestMyPlotExtras
[11742] signal (11.2): Segmentation fault: 11
in expression starting at /Users/henry/TestMyPlotExtras/TestMyPlotExtras/src/TestMyPlotExtras.jl:3
PyObject_GetAttr at /Users/henry/TestMyPlotExtras/TestMyPlotExtras/.CondaPkg/env/lib/libpython3.11.dylib (unknown line)
PyObject_GetAttr at /Users/henry/.julia/packages/PythonCall/qTEA1/src/cpython/pointers.jl:299 [inlined]
macro expansion at /Users/henry/.julia/packages/PythonCall/qTEA1/src/Py.jl:131 [inlined]
pygetattr at /Users/henry/.julia/packages/PythonCall/qTEA1/src/abstract/object.jl:60
getproperty at /Users/henry/.julia/packages/PythonCall/qTEA1/src/Py.jl:261 [inlined]
__init__ at /Users/henry/TestMyPlotExtras/MyPlotExtras/src/MyPlotExtras.jl:6
jl_sysimg_fvars_base at /Users/henry/.julia/compiled/v1.9/MyPlotExtras/dkBtT_GM8Uw.dylib (unknown line)
ijl_apply_generic at /Applications/ (unknown line)
jl_module_run_initializer at /Applications/ (unknown line)
ijl_init_restored_modules at /Applications/ (unknown line)
register_restored_modules at ./loading.jl:1115
_include_from_serialized at ./loading.jl:1061
_require_search_from_serialized at ./loading.jl:1506
_require at ./loading.jl:1783
_require_prelocked at ./loading.jl:1660
macro expansion at ./loading.jl:1648 [inlined]
macro expansion at ./lock.jl:267 [inlined]
require at ./loading.jl:1611
jfptr_require_55044 at /Applications/ (unknown line)
ijl_apply_generic at /Applications/ (unknown line)
eval_import_path at /Applications/ (unknown line)
jl_toplevel_eval_flex at /Applications/ (unknown line)
jl_toplevel_eval_flex at /Applications/ (unknown line)
jl_toplevel_eval_flex at /Applications/ (unknown line)
ijl_toplevel_eval_in at /Applications/ (unknown line)
eval at ./boot.jl:370 [inlined]
include_string at ./loading.jl:1903
ijl_apply_generic at /Applications/ (unknown line)
_include at ./loading.jl:1963
include at ./Base.jl:457 [inlined]
include_package_for_output at ./loading.jl:2049
jfptr_include_package_for_output_43027 at /Applications/ (unknown line)
ijl_apply_generic at /Applications/ (unknown line)
do_call at /Applications/ (unknown line)
eval_body at /Applications/ (unknown line)
jl_interpret_toplevel_thunk at /Applications/ (unknown line)
jl_toplevel_eval_flex at /Applications/ (unknown line)
jl_toplevel_eval_flex at /Applications/ (unknown line)
ijl_toplevel_eval_in at /Applications/ (unknown line)
eval at ./boot.jl:370 [inlined]
include_string at ./loading.jl:1903
include_string at ./loading.jl:1913 [inlined]
exec_options at ./client.jl:305
_start at ./client.jl:522
jfptr__start_51456 at /Applications/ (unknown line)
ijl_apply_generic at /Applications/ (unknown line)
true_main at /Applications/ (unknown line)
jl_repl_entrypoint at /Applications/ (unknown line)
Allocations: 4925588 (Pool: 4922287; Big: 3301); GC: 7
ERROR: Failed to precompile TestMyPlotExtras [679a42ec-c8f9-48a5-807c-1ce16acee5ed] to "/Users/henry/.julia/compiled/v1.9/TestMyPlotExtras/jl_IJiDWn".
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
   @ Base ./loading.jl:2300
 [3] compilecache
   @ ./loading.jl:2167 [inlined]
 [4] _require(pkg::Base.PkgId, env::String)
   @ Base ./loading.jl:1805
 [5] _require_prelocked(uuidkey::Base.PkgId, env::String)
   @ Base ./loading.jl:1660
 [6] macro expansion
   @ ./loading.jl:1648 [inlined]
 [7] macro expansion
   @ ./lock.jl:267 [inlined]
 [8] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1611

I’m on Apple M1:

julia> versioninfo()
Julia Version 1.9.3
Commit bed2cd540a1 (2023-08-24 14:43 UTC)
Build Info:
  Official release
Platform Info:
  OS: macOS (arm64-apple-darwin22.4.0)
  CPU: 10 × Apple M1 Max
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, apple-m1)
  Threads: 1 on 8 virtual cores

Digging further, there is a workaround that PythonPlot.jl uses. Adding that to skip precompilation of __ini__() in MyPlotExtras does seem to work. However, the PythonCall.jl author seems to say that is not a good solution.

What is a good solution?

If you don’t want to use the ugly ccall(:jl_generating_output, ...) to check whether Julia is precompiling, you can just check whether PythonPlot.matplotlib is initialized:

PythonCall.pyisnull(PythonPlot.matplotlib) && return
1 Like

The reason you get this crash is because PythonPlot skips __init__ when precompiling. This means that the matplotlib object is not properly initialised when you access it in your own __init__ function, hence the crash.

AFAIU skipping __init__ is an antipattern, precisely for this sort of reason: as soon as one package skips __init__ then all downstream packages need to as well, because downstream __init__ functions are allowed to assume all upstream packages are properly initialised. This is why I refused to do it in PythonCall.

@stevengj why does PythonPlot do this?

Also a direct solution to your problem is to call pyimport("matplotlib") instead of PythonPlot.matplotlib.

It was to work around `using PyPlot` crashes package precompilation on Julia v1.10.0-beta1 · Issue #572 · JuliaPy/PyPlot.jl · GitHub, IIRC. Probably there is a better fix, e.g. initialize matplotlib with a non-interactive backend (Agg) when precompiling.