Adding Hwloc as a julia dependency

During the Multithreading BoF @kpamnany and I were discussing adding Hwloc as a dependency of julia itself.
Some reasons for adding it to julia would be to simplify the cpu detection code we have. Currently there is a lot of handwavy and convoluted code to get around hyperthreading and now efficiency cores that CPUs might have. We also have to keep track of a number of relevant cpus and their features, which is a small maintenance burden.
In addition we would be able to get cache sizes and other cpu details, which might be useful for some algorithms.

There are a few long standing issues that would be fixed by this i.e Use number of physical not logical cores for auto nthreads? · Issue #43692 · JuliaLang/julia · GitHub and friends.

6 Likes

Oooh… as an HPC person hwloc is of course used frequently, and is an integral part of our builds.
HOWEVER if you are wanting to perform CPU recognition it is not the only game in town

Would Archspec be more useful GitHub - archspec/archspec: A library for detecting, labeling, and reasoning about microarchitectures
Archspec aims at providing a standard set of human-understandable labels for various aspects of a system architecture like CPU, network fabrics, etc. and APIs to detect, query and compare them.

Totally blueskying here… could we think about a native Julia implementation?

2 Likes

I think this would be awesome; it would be great to have something more official for cases like: ScanByte.jl/codegen.jl at 492af6fbdbbec3beb4381a96b9d52f69858a67ba · jakobnissen/ScanByte.jl · GitHub.

2 Likes

Archspec has python bindings, though it’s basically a big JSON file. Which could be interesting. Pure julia wouldn’t be super helpful here, because ideally we want that information at the runtime, probably even at init time. So a C lib like hwloc might be easier to integrate.

@quinnj have you seen Base.BinaryPlatforms.CPUID. Which is the code I mentioned that is a bit annoying to maintain, it exposes those strings and ways to test them quite a bit easie rthan what you’ve done in ScanByte