I haven’t checked in a while but has Julia fixed the problem with only using 1/2 or 1/4 of the available cpu threads on high-end (either >64 HW threads or multi-socket) windows systems?
To be clear the issue isn’t with Julia but libuv. I fixed it myself a while back and sent them the fix but got a little frustrated with conforming to their formatting standard so didn’t follow up much after and kept it for my personal use. But now I’m thinking I should follow through unless it’s been fixed.
I believe the issue isn’t libuv, but that windows doesn’t actually support more than 64 cores reasonably (but I’m not 100% usre). If you have that many cores you should be using Linux since it actually has proper schedulers and stuff IO system to work for big systems.
No it supports 128 cores or more but you need a little tiny special codein your thread creation because of 64 bit thread masks. Ive fixed it in libuv and other libs plus my personal codebase. When i run with my modified libuv, i can max out any number of hw threads and get speedups past 64 threads (obviously its hard to get LINEAR speedups at that thread count :). Here’s an article i wrote about it with (slightly incomplete) sample code:
And heres the libuv fix: Fix for win32 processor groups issue · libuv/libuv · Discussion #3457 · GitHub
I’m confused about why the code is stuffed into a Github discussion rather than as a pull request. It is very difficult to read and parse that way, especially since I would mostly be interested in the diff.
Note that there is a Julia fork of libuv:
I’m aware of what you speak since I have a multisocket Windows machine at work. My usual solution is to run an individual Julia process for each socket, especially since I usually want to be explicit about memory transfers between NUMA nodes.
Like i said git and github aren’t really a thing in my work and i was very unfamiliar - perforce has a more dominant position in aaa game dev.
I have since released some code (threading extensions to eigen GitHub - ValveSoftware/eigen: Fork of Eigen release version 3.4. Adds ability to use alternate threading systems from Open MP.) to github and am a little familiar now, another reason to follow up. Seems kind of selfish of me to be running the only julia on windows without this problem :). Plus it’s a pain to have to build my own libuv.
Ill see if i can put together a fixed version of the julia fork of libuv and submit a patch properly
Can you zip or tar up the modified source code and post it somewhere?
yes.
Give me a little bit, I need to update to the latest and merge+test the changes.
If you would like, but you seem to be having a lof of challenges with git.