I’m currently thinking of applying for JSoC/GSoC, and I’d like to discuss a project for it, which is about task parallelism.
As discussed in https://github.com/JuliaLang/julia/issues/33762, the current scheduling algorithm (parallel depth first) seems to have a scalability issue for fine-grained tasks.
The current issues would be the following:
- It does not scale for fine-grained tasks because of lock contention
- It’s not NUMA-aware
As you might know, a lot of schedulers for task parallelism have been proposed even in these days. And I guess there is no “best” scheduler and there are always trade-offs. In some cases, work stealing schedulers would perform better.
My proposal is to make an abstraction to customize the scheduling policy in Julia, and to make it possible to publish user-defined schedulers as packages, so that users can choose the “best” scheduler for their own purpose.
I’d like to hear your opinions!