So we have some more breadcrumbs to follow.
- Let’s watch for any memory write to the object
0x00006e36383e2cc8 in gc_assert_parent_validity (parent=0x43fd6a5f76d0,
child=0x588277d1b3c0 <jl_system_image_data+8141504>)
at /home/vchuravy/src/julia-1.11/src/gc.c:1974
1974 if (__unlikely(!jl_is_datatype((jl_datatype_t *)child_vt) || ((jl_datatype_t *)child_vt)->smalltag)) {
(rr) watch *(uintptr_t*)0x588277d1b3c0
Hardware watchpoint 2: *(uintptr_t*)0x588277d1b3c0
- Every julia object has a tag before it. If that tag is corrupted we don’t know what is stored there
(rr) p/x child_astagged
$4 = 0x588277d1b3b8
(rr) watch *(uintptr_t*)0x588277d1b3b8
Hardware watchpoint 3: *(uintptr_t*)0x588277d1b3b8
- The object is currently stored in an array, so we have the “slot” address and can watch when this object got written to this array
(rr) up
#1 0x00006e36383e47b2 in gc_mark_objarray (ptls=0x5607db3e3fc0, obj_parent=0x43fd6a5f76d0, obj_begin=0x5607dbca7498, obj_end=0x5607dbca98a0, step=1,
nptr=32370) at /home/vchuravy/src/julia-1.11/src/gc.c:2245
(rr) p slot
$5 = (jl_value_t **) 0x5607dbca7498
(rr) watch *(jl_value_t **) 0x5607dbca7498
Hardware watchpoint 4: *(jl_value_t **) 0x5607dbca7498
(rr) rc
Okay, let’s see if we hit any useful breakpoints.
Continuing.
Downloading 343.98 K source file /usr/src/debug/python/Python-3.13.5/Objects/typeobject.c
Thread 2 hit Hardware watchpoint 3: *(uintptr_t*)0x588277d1b3b8
Old value = 65703
New value = 167
set_version_unlocked (tp=0x588277d1b220 <jl_system_image_data+8141088>, version=238) at Objects/typeobject.c:972
972 tp->tp_versions_used++;
Uhm, why is Python smashing our type-tag?
(rr) bt
#0 set_version_unlocked (tp=0x588277d1b220 <jl_system_image_data+8141088>,
version=238) at Objects/typeobject.c:972
#1 assign_version_tag (interp=interp@entry=0x506a4085a5b0 <_PyRuntime+88400>,
type=<optimized out>) at Objects/typeobject.c:1164
#2 0x0000506a404b0ce3 in assign_version_tag (
interp=interp@entry=0x506a4085a5b0 <_PyRuntime+88400>,
type=<optimized out>) at Objects/typeobject.c:1154
#3 0x0000506a404b0ce3 in assign_version_tag (
interp=interp@entry=0x506a4085a5b0 <_PyRuntime+88400>,
type=type@entry=0x5607dba92b90) at Objects/typeobject.c:1154
#4 0x0000506a4056395a in _PyType_LookupRef (type=0x5607dba92b90,
name=0x506a4084e510 <_PyRuntime+39088>) at Objects/typeobject.c:5293
#5 _Py_slot_tp_getattr_hook (self=0x548276983940, name=0x321c2d527a20)
at Objects/typeobject.c:9629
#6 0x0000506a404bc3c2 in PyObject_GetAttr (v=0x548276983940,
name=0x321c2d527a20) at Objects/object.c:1261
#7 0x0000506a404f377f in _PyObject_GetMethod (obj=0x548276983940,
name=0x321c2d527a20, method=0x7ffd67ad4fb8) at Objects/object.c:1552
#8 0x0000506a404d3218 in _PyEval_EvalFrameDefault (tstate=<optimized out>,
frame=<optimized out>, throwflag=<optimized out>)
at Python/generated_cases.c.h:3748
#9 0x0000506a405a58b9 in PyEval_EvalCode (co=0x30ac6239a170,
globals=<optimized out>, locals=0x321c2d4f0800) at Python/ceval.c:604
#10 0x0000506a405e3f5c in run_eval_code_obj (
tstate=tstate@entry=0x506a40889df0 <_PyRuntime+283024>,
co=co@entry=0x30ac6239a170, globals=globals@entry=0x321c2d4f0800,
locals=locals@entry=0x321c2d4f0800) at Python/pythonrun.c:1381
#11 0x0000506a405e101b in run_mod (mod=mod@entry=0x5607daf3fe88,
filename=filename@entry=0x321c2d4f08f0,
globals=globals@entry=0x321c2d4f0800, locals=locals@entry=0x321c2d4f0800,
flags=flags@entry=0x7ffd67ad52f8, arena=arena@entry=0x30ac6230fd30,
interactive_src=0x321c2d511ae0, generate_new_source=0)
at Python/pythonrun.c:1466
#12 0x0000506a405dc35e in _PyRun_StringFlagsWithName (
str=str@entry=0x321c2d511400 "from juliacall import Main as jl; [jl.randn(5) for _ in range(10000)]\n", name=name@entry=0x321c2d4f08f0,
start=start@entry=257, globals=globals@entry=0x321c2d4f0800,
locals=locals@entry=0x321c2d4f0800, flags=flags@entry=0x7ffd67ad52f8,
generate_new_source=0) at Python/pythonrun.c:1261
We can confirm that prior to that we had a valid Julia object.
(rr) p jl_(0x588277d1b3c0)
"juliacall.ValueBase"
$6 = void
So yay? We found our culprit? For some reason python is smashing the type-tag of a Julia object…
Out of curiosity I kept reverse executing.
Thread 2 hit Hardware watchpoint 4: *(jl_value_t **) 0x5607dbca7498
Old value = (jl_value_t *) 0x588277d1b3c0 <jl_system_image_data+8141504>
New value = (jl_value_t *) 0x0
0x00006e3638378e6f in jl_genericmemory_ptr_set (m=0x43fd6a5f76d0, i=6939,
x=0x588277d1b3c0 <jl_system_image_data+8141504>)
at /home/vchuravy/src/julia-1.11/src/julia.h:1201
1201 jl_atomic_store_release(((_Atomic(jl_value_t*)*)(m_->ptr)) + i, (jl_value_t*)x);
0x00006e3638378e6f in jl_genericmemory_ptr_set (m=0x43fd6a5f76d0, i=6939,
x=0x588277d1b3c0 <jl_system_image_data+8141504>)
at /home/vchuravy/src/julia-1.11/src/julia.h:1201
1201 jl_atomic_store_release(((_Atomic(jl_value_t*)*)(m_->ptr)) + i, (jl_value_t*)x);
(rr) bt
#0 0x00006e3638378e6f in jl_genericmemory_ptr_set (m=0x43fd6a5f76d0, i=6939,
x=0x588277d1b3c0 <jl_system_image_data+8141504>)
at /home/vchuravy/src/julia-1.11/src/julia.h:1201
#1 0x00006e363837c1fd in jl_idset_put_key (keys=0x43fd6a5f76d0,
key=0x588277d1b3c0 <jl_system_image_data+8141504>, newidx=0x7ffd67acc4e0)
at /home/vchuravy/src/julia-1.11/src/idset.c:89
#2 0x00006e36383b029b in jl_as_global_root (
val=0x588277d1b3c0 <jl_system_image_data+8141504>, insert=1)
at /home/vchuravy/src/julia-1.11/src/staticdata.c:2547
#3 0x00006e36383af2c9 in jl_root_new_gvars (s=0x7ffd67accb70,
image=0x7ffd67ace040, external_fns_begin=2748)
at /home/vchuravy/src/julia-1.11/src/staticdata.c:2283
#4 0x00006e36383b4ee1 in jl_restore_system_image_from_stream_ (
f=0x7ffd67acdf10, image=0x7ffd67ace040, depmods=0x43fd6a5142f0,
checksum=18085326892769599201, restored=0x7ffd67acdc50,
init_order=0x7ffd67acdc58, extext_methods=0x7ffd67acdc60,
internal_methods=0x7ffd67acdc68, new_ext_cis=0x7ffd67acdc70,
method_roots_list=0x7ffd67acdc78, ext_targets=0x7ffd67acdc80,
edges=0x7ffd67acdc88, base=0x7ffd67acdc48, ccallable_list=0x7ffd67acdd90,
cachesizes=0x7ffd67acdcf0)
at /home/vchuravy/src/julia-1.11/src/staticdata.c:3532
#5 0x00006e36383b5716 in jl_restore_package_image_from_stream (
pkgimage_handle=0x5607db723940, f=0x7ffd67acdf10, image=0x7ffd67ace040,
depmods=0x43fd6a5142f0, completeinfo=0,
pkgname=0x43fd697287f8 "PythonCall", needs_permalloc=0)
at /home/vchuravy/src/julia-1.11/src/staticdata.c:3670
#6 0x00006e36383b5bd8 in jl_restore_incremental_from_buf (
pkgimage_handle=0x5607db723940,
buf=0x588277557900 <jl_system_image_data> "\373jli\r\n\032\n\f",
image=0x7ffd67ace040, sz=11933428, depmods=0x43fd6a5142f0, completeinfo=0,
pkgname=0x43fd697287f8 "PythonCall", needs_permalloc=0)
at /home/vchuravy/src/julia-1.11/src/staticdata.c:3729
#7 0x00006e36383b6267 in ijl_restore_package_image_from_file (
fname=0x43fd6aaba138 "/home/vchuravy/.julia/compiled/v1.11/PythonCall/WdXsa_9Ah4Y.so", depmods=0x43fd6a5142f0, completeinfo=0,
pkgname=0x43fd697287f8 "PythonCall", ignore_native=0)
So it is a global variable that we are restoring…
perhaps
I have no clue what this code is trying to do, but in any case python should not stamp over Julia’s type-tag.