How can anybody reject such an offer! There’s no point building a second general-purpose tracer when Reactant.jl already does this so well. The tracer here is deliberately handling vectors, matrices, broadcasting, matmul and reductions only.
Here’s what could be a possible path forward. AppleNeuralEngine.jl keeps the MIL graph builder and the runtime that calls the ANE directly as its core. On top of that we’d add an optional Reactant package extension that takes Reactant’s optimized StableHLO module and lowers it to MIL. That way, much of what Reactant can trace can run on the Neural Engine, and users who don’t want XLA/MLIR as a dependency don’t have to take it on.
Most of the ANE-specific work is in the lowering:
- precision: the ANE computes in fp16, so f32 has to be lowered explicitly, not silently;
- layout: the ANE is fastest with [1, C, 1, S] tensors, so there’s a layout pass;
- op coverage: we probed 195 MIL ops and 148 run on the ANE.
On M2, I did not figure out how to enable while, scatter, cumsum and the trig ops, and gather needs (u)int16 indices. If I am right, the converter should reject these with a clear message, or rewrite them with ops the ANE has (cumsum as a triangular matmul, for example). Otherwise our ANE compile fails on them, and through Core ML they’d quietly run on the CPU.
So,
- What’s the cleanest hook for getting the
StableHLOmodule after your optimization passes, so a backend that isn’tXLAcan take over from there? - Can Reactant emit
f16directly, or would we cast at the boundary? - Can loops with a known trip count stay unrolled, or be unrolled by a pass?
whiledoesn’t run on the ANE.
Does this make sense, or do you have an alternative plan?
I just pushed some commits that were missing.