[ANN] AppleNeuralEngine.jl: Julia on the Apple Neural Engine

How can anybody reject such an offer! There’s no point building a second general-purpose tracer when Reactant.jl already does this so well. The tracer here is deliberately handling vectors, matrices, broadcasting, matmul and reductions only.

Here’s what could be a possible path forward. AppleNeuralEngine.jl keeps the MIL graph builder and the runtime that calls the ANE directly as its core. On top of that we’d add an optional Reactant package extension that takes Reactant’s optimized StableHLO module and lowers it to MIL. That way, much of what Reactant can trace can run on the Neural Engine, and users who don’t want XLA/MLIR as a dependency don’t have to take it on.

Most of the ANE-specific work is in the lowering:

  • precision: the ANE computes in fp16, so f32 has to be lowered explicitly, not silently;
  • layout: the ANE is fastest with [1, C, 1, S] tensors, so there’s a layout pass;
  • op coverage: we probed 195 MIL ops and 148 run on the ANE.

On M2, I did not figure out how to enable while, scatter, cumsum and the trig ops, and gather needs (u)int16 indices. If I am right, the converter should reject these with a clear message, or rewrite them with ops the ANE has (cumsum as a triangular matmul, for example). Otherwise our ANE compile fails on them, and through Core ML they’d quietly run on the CPU.

So,

  1. What’s the cleanest hook for getting the StableHLO module after your optimization passes, so a backend that isn’t XLA can take over from there?
  2. Can Reactant emit f16 directly, or would we cast at the boundary?
  3. Can loops with a known trip count stay unrolled, or be unrolled by a pass? while doesn’t run on the ANE.

Does this make sense, or do you have an alternative plan?
I just pushed some commits that were missing.

Seems like a reasonable first plan to me.

re your questions.

(1), probably just hook into code_hlo or similar [and you can extract out the mlir]
(2), yup f16 is fine
(3), either! we have an unrolling pass which runs by default when it thinks profitable, and all loops not marked with @ trace currently are unrolled by default

AppleNeuralEngine.jl now uses ObjectiveC.jl 7.2, so it works alongside the latest Metal.jl. There’s also an experimental Reactant.jl front end:

using Reactant
compile(f, x; frontend=:reactant)

following the suggestion by @wsmoses.

The op support page now lists for each op the chips it works on. Thanks to @Mathieu for the first PR, an M4 run: sin and cos compile there, but not on the M2 Max. If you have a different M-series Mac other than those already listed, and a couple of minutes to spare, please run the probe and open a PR with the two files it writes.

git clone https://github.com/pitsianis/AppleNeuralEngine.jl
cd AppleNeuralEngine.jl
julia --project=. -e 'using Pkg; Pkg.instantiate()'
PROBE_BY="Your Name" julia --project=. tools/probe_ops.jl