[ANN] AppleNeuralEngine.jl: Julia on the Apple Neural Engine

Package Announcements
,
1

Every Apple Silicon Mac has a Neural Engine (ANE), an fp16 accelerator that Apple exposes only through Core ML. AppleNeuralEngine.jl drives it directly from Julia through ObjectiveC.jl, as Metal.jl does for the GPU, with no native code to build. You write ordinary Julia, and it is traced into an ANE program:

using AppleNeuralEngine
silu(z) = z / (1 + exp(-z))
function ffn(x)  # LLaMA-style feed-forward block: RMSNorm, SwiGLU, residual
    n = x ./ sqrt.(sum(abs2, x; dims=2) ./ size(x, 2) .+ 1f-5) .* g'
    return x .+ (silu.(n * W1') .* (n * W3')) * W2'
end
x   = randn(Float16, 256, 512)
ane = compile(ffn, x)   # trace + compile
y   = ane(x)            # runs on the Neural Engine

Results on an M2 Max

16-core ANE, macOS 27.0.1, Julia 1.13.3, 8 BLAS threads. The same Julia function runs on each device; ANE times include copying inputs and outputs.

workload ANE fp16 ms GFLOP/s Accelerate fp32 ms GFLOP/s OpenBLAS fp32 ms GFLOP/s rel. error
FFN block D=512 H=1536 T=256 0.261 4629 2.646 457 3.866 312 6.5e-04
x*W 256×1024 · 1024×1024 0.148 3628 0.210 2553 0.728 737 3.3e-04
x*W 1024×1024 · 1024×1024 0.366 5859 0.791 2715 2.839 756 3.3e-04
x*W 2048×2048 · 2048×2048 1.774 9682 7.223 2378 24.062 714 3.6e-04

ANE times include copying inputs and outputs. The ANE computes in fp16 and the CPU in fp32, so each runs at its native precision.

Reproduce it and post your numbers

git clone https://github.com/pitsianis/AppleNeuralEngine.jl
cd AppleNeuralEngine.jl
julia --project=benchmark -e 'using Pkg; Pkg.instantiate()'
julia --project=benchmark benchmark/run.jl     # prints the table above for your Mac
julia --project=benchmark tools/probe_ops.jl   # checks every op on your Mac

The benchmark prints a Markdown table with your chip and macOS version: please paste it in this thread. So far everything has been tested only with Julia 1.13.3 on a M2 Max.

What runs on the ANE

On the M2 Max, 129 of 167 probed MIL operations compile and give correct results. Works: elementwise arithmetic, exp/log/sqrt/tanh/erf, activations (relu, gelu, silu, softmax, …), reductions, convolutions (strided, depthwise, int8 weights), pooling, matmul/linear/einsum, layer/batch/instance norm, scaled dot-product attention, reshape/transpose/slice/concat/split/tile, upsampling. Rejected by the ANE compiler: sin/cos/tan, pad, gather/scatter, cumsum, topk/argsort/argmax, LSTM/GRU/RNN, random ops, fp32 compute. Full table: docs/src/op_support.md.

PRs are very welcome if you get a rejected op to work, or if tools/probe_ops.jl gives a different table on a newer M chip or macOS version. The probe rewrites docs/src/op_support.md, so the PR is that file.

Caveats: it uses a private, undocumented API that any macOS update can break. It computes in fp16 only, and each call costs about 0.1–0.2 ms, so it pays off for matrix–matrix work. It is not registered yet.

Credits: this builds on the reverse engineering of the private ANE runtime in maderix/ANE, and on ObjectiveC.jl and Metal.jl. Most of the code was written together with Claude, Anthropic’s AI assistant, in Claude Code.

Repository: GitHub - pitsianis/AppleNeuralEngine.jl: Julia on the Apple Neural Engine: trace Julia functions, build MIL graphs, run them on the ANE · GitHub

Edit: Corrected a bug that underestimated the performance of Accelerate and OpenBLAS.

2

Apple M1 Max*, ANE: 16 cores (h13g), macOS 26.6.2, Julia 1.12.7, 8 BLAS threads

workload ANE fp16 ms GFLOP/s Accelerate fp32 ms GFLOP/s OpenBLAS fp32 ms GFLOP/s rel. error
FFN block D=512 H=1536 T=256 0.334 3618 2.704 447 3.909 309 6.8e-04
x*W 256×1024 · 1024×1024 0.257 2087 0.238 2257 0.770 697 3.3e-04
x*W 1024×1024 · 1024×1024 0.516 4162 0.882 2435 3.006 714 3.3e-04
x*W 2048×2048 · 2048×2048 2.391 7186 9.159 1876 23.880 719 3.6e-04

Apple M3 Ultra, ANE: 32 cores (h15g), macOS 26.6.2, Julia 1.12.6, 20 BLAS threads

workload ANE fp16 ms GFLOP/s Accelerate fp32 ms GFLOP/s OpenBLAS fp32 ms GFLOP/s rel. error
FFN block D=512 H=1536 T=256 0.224 5391 1.812 667 2.772 436 6.6e-04
x*W 256×1024 · 1024×1024 0.134 3997 0.114 4701 0.445 1207 3.3e-04
x*W 1024×1024 · 1024×1024 0.331 6489 0.374 5744 1.333 1611 3.3e-04
x*W 2048×2048 · 2048×2048 1.683 10209 3.370 5097 9.550 1799 3.6e-04

ANE times include copying inputs and outputs. CPU times are Float32.

3

Apple M5 Max, ANE: 16 cores (h17g), macOS 27.2, Julia 1.13.3, 17 BLAS threads

workload ANE fp16 ms GFLOP/s Accelerate fp32 ms GFLOP/s OpenBLAS fp32 ms GFLOP/s rel. error
FFN block D=512 H=1536 T=256 0.171 7066 1.889 639 2.202 549 6.7e-04
x*W 256×1024 · 1024×1024 0.126 4252 0.308 1741 0.425 1262 3.3e-04
x*W 1024×1024 · 1024×1024 0.247 8691 0.811 2648 1.924 1116 3.3e-04
x*W 2048×2048 · 2048×2048 1.239 13860 6.197 2772 12.047 1426 3.6e-04
4

**Apple M2 Pro**, ANE: 16 cores (h14g), macOS 27.0.1, Julia 1.13.2, 6 BLAS threads

workload ANE fp16 ms GFLOP/s Accelerate fp32 ms GFLOP/s OpenBLAS fp32 ms GFLOP/s rel. error
FFN block D=512 H=1536 T=256 0.231 5230 2.588 467 4.451 271 6.7e-04
x*W 256×1024 · 1024×1024 0.147 3642 0.208 2576 0.968 555 3.3e-04
x*W 1024×1024 · 1024×1024 0.374 5748 0.789 2720 4.285 501 3.3e-04
x*W 2048×2048 · 2048×2048 1.735 9904 7.759 2214 31.066 553 3.6e-04

ANE times include copying inputs and outputs. CPU times are Float32.

5

As you get more data points, it would be interesting to perform an ANOVA check or HSD to see if the results are platform dependent or sensitive to the system load. I see a couple of surprises among the test environments.

6

Apple M4 Pro, ANE: 16 cores (h16g), macOS 26.5.1, Julia 1.13.3, 8 BLAS threads.

workload ANE fp16 ms GFLOP/s Accelerate fp32 ms GFLOP/s OpenBLAS fp32 ms GFLOP/s rel. error
FFN block D=512 H=1536 T=256 0.167 7233 1.737 696 2.945 410 6.4e-04
x*W 256×1024 · 1024×1024 0.111 4824 0.175 3069 0.649 827 3.3e-04
x*W 1024×1024 · 1024×1024 0.255 8437 0.674 3188 2.554 841 3.3e-04
x*W 2048×2048 · 2048×2048 1.412 12167 5.044 3406 21.758 790 3.6e-04

ANE times include copying inputs and outputs. CPU times are Float32.

7

Apple M1 Pro, ANE: 16 cores (h13g), macOS 27.0.1, Julia 1.13.1, 6 BLAS threads

workload ANE fp16 ms GFLOP/s Accelerate fp32 ms GFLOP/s OpenBLAS fp32 ms GFLOP/s rel. error
FFN block D=512 H=1536 T=256 0.321 3764 2.786 434 4.674 258 6.7e-04
x*W 256×1024 · 1024×1024 0.249 2158 0.235 2284 1.031 521 3.3e-04
x*W 1024×1024 · 1024×1024 0.461 4654 1.126 1907 4.320 497 3.3e-04
x*W 2048×2048 · 2048×2048 2.324 7392 9.246 1858 34.057 504 3.6e-04
8

Apple M4, ANE: 16 cores (h16g), macOS 27.0.1, Julia 1.13.3, 4 BLAS threads

workload ANE fp16 ms GFLOP/s Accelerate fp32 ms GFLOP/s OpenBLAS fp32 ms GFLOP/s rel. error
FFN block D=512 H=1536 T=256 0.211 5720 1.975 612 4.773 253 6.5e-04
x*W 256×1024 · 1024×1024 0.141 3818 0.322 1670 1.429 376 3.3e-04
x*W 1024×1024 · 1024×1024 0.324 6620 1.132 1897 5.714 376 3.3e-04
x*W 2048×2048 · 2048×2048 1.599 10742 9.231 1861 44.787 384 3.6e-04

ANE times include copying inputs and outputs. CPU times are Float32.

9

Apple M5 Pro, ANE: 16 cores (h17g), macOS 27.0.1, Julia 1.13.3, 17 BLAS threads

workload ANE fp16 ms GFLOP/s Accelerate fp32 ms GFLOP/s OpenBLAS fp32 ms GFLOP/s rel. error
FFN block D=512 H=1536 T=256 0.179 6758 1.879 643 2.203 548 6.6e-04
x*W 256×1024 · 1024×1024 0.122 4416 0.308 1744 0.426 1261 3.3e-04
x*W 1024×1024 · 1024×1024 0.257 8355 0.813 2640 1.945 1104 3.3e-04
x*W 2048×2048 · 2048×2048 1.251 13728 6.195 2773 12.009 1431 3.6e-04
10

I was trying something else and the results are not what I expected.

julia> using AppleNeuralEngine

julia> A=rand(Float16,1000,1000); x=rand(Float16,1000);

julia> mm(A,x) = A*x;

julia> mmc=compile(mm,A,x);

julia> using BenchmarkTools;

julia> @belapsed mm($A,$x)
3.43750e-05

julia> @belapsed mmc($A,$x)
1.50875e-04

Neural Engine is over 4x slower.

11

Does compile pre-compile on definition?

12

Apple M3 Max, ANE: 16 cores (h15g), macOS 26.7.2, Julia 1.13.3, 12 BLAS threads

workload ANE fp16 ms GFLOP/s Accelerate fp32 ms GFLOP/s OpenBLAS fp32 ms GFLOP/s rel. error
FFN block D=512 H=1536 T=256 0.205 5883 2.094 577 2.976 406 6.7e-04
x*W 256×1024 · 1024×1024 0.125 4311 0.192 2796 0.506 1061 3.3e-04
x*W 1024×1024 · 1024×1024 0.348 6174 0.706 3043 1.945 1104 3.3e-04
x*W 2048×2048 · 2048×2048 1.671 10282 5.639 3047 14.592 1177 3.6e-04

ANE times include copying inputs and outputs. CPU times are Float32.

13

I am not sure what “precompile on definition” means. Does this help?

julia> using BenchmarkTools

julia> A=rand(Float16,1000,1000); x=rand(Float16,1000);

julia> mm(A,x)=A*x;

julia> mmc=compile(mm,A,x)
ANEProgram(inputs=[Float16(1000, 1000), Float16(1000)], outputs=[Float16(1000)])

julia> @belapsed mm($A,$x)
3.43750e-05

julia> @belapsed mmc($A,$x)
1.53000e-04
14

Hello Tim (@ctkelley), you are hitting two limitations at once:

  1. the ANE’s fast path is matrix–matrix, not matrix–vector — gemv on the CPU is very efficient and the ANE doesn’t have an equivalent fast lane for it;

  2. each ANE call has ~0.1–0.2 ms of fixed overhead for the IOSurface copy and dispatch, which exceeds the 34 µs your CPU takes. The ANE pays off when both operands are matrices and the compute time is well above that overhead floor — the benchmark workloads are all in that regime.

To see how your experiment scales, I tried larger sizes: the time difference gets a bit better but the fp16 overflow bites.

using LinearAlgebra, AppleNeuralEngine, BenchmarkTools

function testit(n)
         A = rand(Float16,n,n)
         x = rand(Float16,n)

         mm(A,x) = A*x
         mmc = compile(mm,A,x)
         
         b = mm(A,x)
         bc = mmc(A,x)
         return (@belapsed $mm($A,$x)) / (@belapsed $mmc($A,$x)), norm(b-bc)/norm(b)
 end

Then

julia> testit(2^10)
(0.13035633739287325, Float16(0.00786))

julia> testit(2^11)
(0.31250537875006457, Float16(0.02293))

julia> testit(2^12)
(0.27417863385653507, Float16(0.07825))

julia> testit(2^13)
(0.09426929906631279, Float16(0.0))

julia> testit(2^14)
(0.08214119742140809, NaN16)

at n=2^13, norm(b) overflows to Inf while norm(b-bc) is still finite, so you get finite/Inf == 0.0; at n=2^14 both overflow, giving Inf/Inf == NaN

15

And if I make them square matrices, the advantage comes back.

julia> testit.(2 .^(6:12))
7-element Vector{Tuple{Float64, Float16}}:
 (0.18883102493074794, 0.00099)
 (0.6223709369024856, 0.001325)
 (3.899402513776237, 0.00189)
 (16.326130428398624, 0.003206)
 (35.00449930490615, 0.0)
 (46.328273513563076, 0.0)
 (36.50367896868851, NaN)
16

The first time you call a function, there is some latency while it precompiles, But I misread and by the time mmc() is called whatever compilation is needed has already been performed for ANEProgram so my speculation is wrong.

17

Thanks @pitsianis. That all makes sense.

18

Random question, rather than build a separate tracer, why not just build it off Reactant.jl’s tracing mechanism?

19

Good question. I’m fairly new to this, so take this as my limited understanding. This package traces a Julia function and emits Apple’s MIL directly, then runs it through the ANE runtime. Reactant.jl is much more general, but its documented backends don’t include the ANE; using it here would require an ANE-specific path from Reactant’s IR to MIL, plus the runtime integration and support for the ANE-compatible subset. For the small fp16 graphs this package currently targets—broadcasts, matrix products, and reductions—a direct tracer seemed like the smaller, more focused first step. The ANE also imposes constraints on supported operations, types, and layouts, which the tracer can account for directly. If people want to run Lux or Flux models or use Enzyme gradients, a Reactant extension that lowers a supported subset to MIL could be a useful next step.