Fixing the type instability also fixed the problem of crashing with GC active. It also improved the speed.
So for now, on Julia 1.11 all works fine. Still need to test on Julia 1.12.
EDIT: While it works on my desktop, it still crashes on my laptop.
I found the reason: I am using ControlPlots.jl, which is based on PyPlot.jl for live plotting. When the GC collects a Python object from another thread than the main thread, Julia crashes.
Workaround: Disable the GC during the multi-threaded operations. Not nice, because it reduces the performance a lot.
So I need to look for alternatives, either using another plotting library, or (more likely) to do plotting in a separate process.