Problems scaling jl_alloc_array_2d C API

Okay, I see the hang you had as well. Looking at the gdb stacktrace there’s a non-main thread that runs the Julia GC as a result of the jl_ptr_to_array call from Go:

(gdb) bt
#0  runtime.futex () at /usr/lib/go/src/runtime/sys_linux_amd64.s:580
#1  0x000000000042c897 in runtime.futexsleep (addr=0x4dfa38 <runtime.sched+280>, val=0, ns=60000000000) at /usr/lib/go/src/runtime/os_linux.go:50
#2  0x000000000040c419 in runtime.notetsleep_internal (n=0x4dfa38 <runtime.sched+280>, ns=60000000000, ~r2=<optimized out>) at /usr/lib/go/src/runtime/lock_futex.go:201
#3  0x000000000040c4f1 in runtime.notetsleep (n=0x4dfa38 <runtime.sched+280>, ns=60000000000, ~r2=<optimized out>) at /usr/lib/go/src/runtime/lock_futex.go:224
#4  0x000000000043e465 in runtime.sysmon () at /usr/lib/go/src/runtime/proc.go:5203
#5  0x0000000000435608 in runtime.mstart1 () at /usr/lib/go/src/runtime/proc.go:1306
#6  0x000000000043550e in runtime.mstart () at /usr/lib/go/src/runtime/proc.go:1272
#7  0x0000000000466851 in crosscall_amd64 () at gcc_amd64.S:35
#8  0x00007fffc8d49640 in ?? ()
#9  0x0000000000000000 in ?? ()

(gdb) info threads
  Id   Target Id                                Frame 
  1    Thread 0x7ffff7d56b80 (LWP 25159) "main" runtime.futex () at /usr/lib/go/src/runtime/sys_linux_amd64.s:580
* 2    Thread 0x7fffc8d49640 (LWP 25163) "main" runtime.futex () at /usr/lib/go/src/runtime/sys_linux_amd64.s:580
  3    Thread 0x7fffc8548640 (LWP 25164) "main" 0x00007ffff78d4d73 in jl_gc_collect () from /usr/lib/julia/libjulia-internal.so.1
  4    Thread 0x7fffc7d47640 (LWP 25165) "main" runtime.futex () at /usr/lib/go/src/runtime/sys_linux_amd64.s:580
  5    Thread 0x7fffc7506640 (LWP 25166) "main" runtime.futex () at /usr/lib/go/src/runtime/sys_linux_amd64.s:580
  6    Thread 0x7fffc6d05640 (LWP 25167) "main" runtime.futex () at /usr/lib/go/src/runtime/sys_linux_amd64.s:580
  7    Thread 0x7fffc621c640 (LWP 25168) "main" 0x00007ffff7d9cae2 in sigtimedwait () from /usr/lib/libc.so.6

(gdb) thread 3
[Switching to thread 3 (Thread 0x7fffc8548640 (LWP 25164))]
#0  0x00007ffff78d4d73 in jl_gc_collect () from /usr/lib/julia/libjulia-internal.so.1
(gdb) bt
#0  0x00007ffff78d4d73 in jl_gc_collect () from /usr/lib/julia/libjulia-internal.so.1
#1  0x00007ffff78d508c in jl_gc_pool_alloc () from /usr/lib/julia/libjulia-internal.so.1
#2  0x00007ffff78d5725 in jl_gc_alloc () from /usr/lib/julia/libjulia-internal.so.1
#3  0x00007ffff78a3d6a in jl_ptr_to_array () from /usr/lib/julia/libjulia-internal.so.1
#4  0x0000000000465e22 in _cgo_169e35b66d06_Cfunc_jl_ptr_to_array (v=0xc000042eb0) at /tmp/go-build/cgo-gcc-prolog:120
#5  0x000000000045e570 in runtime.asmcgocall () at /usr/lib/go/src/runtime/asm_amd64.s:667
#6  0x000000c000000180 in ?? ()
#7  0x000000c0004d6000 in ?? ()
#8  0x000000c0000200e0 in ?? ()
#9  0x00000000004dd801 in runtime.class_to_divmagic ()
#10 0x000000c000000d80 in ?? ()
#11 0x00000000000001c0 in ?? ()
#12 0x000000c000000180 in ?? ()
#13 0x000000c000000180 in ?? ()
#14 0x0000000000000000 in ?? ()

Although the Julia embedding docs don’t seem to mention it explicitly there’s some post here on the forum relating to thread-safety when embedding. Looking at Embedding Julia into multithreading apps - #10 by yuyichao it seems most API methods are thread-safe, but not when calling them from threads that were not started by Julia. In your case Go will have started the different threads, including the one that calls Julia API methods, so I suspect that is what is going on here. But perhaps @yuyichao has some more insights.