Occasional long delays in CUDA.jl

[maleadt]

@hogandp

Could you elaborate on this? Is this a limitation on the underlying libuv implementation, Julia, the scheduler, or some form of a global lock on the IO subsystem?

From talking with @vchuravy, I think the limitation is “just” I/O that’s performed using libuv, due to it’s single-threaded event loop. So IIUC channels, locks, etc do not fall under this restriction, but here we’re using AsyncCondition and Timer, both of which are implemented using libuv.

@jpdoane

During that 127ms, there is essentially no activity in any of the threads, apart from mutex polling. Specifically in the main thread, there are periodic calls to
libjulia-internal.so.1.7!get_next_task, pthread_cond_signal, pthread_mutex_trylock, and libjulia-internal.so.1.7!uv_mutex_trylock. Perhaps something is somehow locking libuv? There are a number of other threads all similarly calling get_next_task and mutex_try_lock. No threads seem to be busy with anything. Eventually, I see a read() in the main thread followed by the catch of EOF in the watchdog thread.

Sorry, I’m not familiar enough with libuv to know what could cause this. Maybe you’re calling into a library from the libuv thread, and that system library is then doing a blocking syscall?

For a solution to your issue, either we could introduce an environment variable that disables the nonblocking synchronization, or we could rework it so that it doesn’t rely on libuv. The simplest solution that comes to mind then is a busy-loop that calls yield() (to give other tasks the opportunity to execute), maybe with some exponential delay and a sleep() to not hog the CPU too much (although CUDA’s default synchronization behavior is to spin as well).