Formalizing Julia in Lean (and finding 400+ bugs in Base)

This is amazing, and thank you so much for all your hard work! How reusable is this code—is there a package or something for finding errors in Julia with Lean? That’d be awesome to see :slight_smile:

Maybe file an PR for this change? I timed it the same:

julia> @btime collect(LinRange(0.9, nextfloat(0.9), 10))
  27.814 ns (2 allocations: 144 bytes)

[Before and after 2 allocations, both 144 bytes, but should either allocate? Might be about global scope.] Maybe slightly faster, or just noise, at least not slower.

What do you mean? We should compete with np.linspace with LinRange (not with StepRangeLen?!). StepRangeLen gives a different result and also with, if you meanst something like that:

julia> @btime collect(StepRangeLen(0.9, nextfloat(0.9) - 0.9, 10))
  29.148 ns (2 allocations: 144 bytes)

Just seen in the na-digest:

From: Diana Halikias diana.halikias@nyu.edu
Date: October 27, 2026
Subject: Lean + Numerical linear algebra working group

We are running a biweekly working group over Zoom to discuss integrating
Lean and NLA research. Topics include an NLA Lean library, incorporating Lean
in combination with AI into our research workflows, proof formalization
standards for the research community, and theoretical underpinnings of Lean.
Our next meeting is Tuesday, Nov 10 at 9:30am eastern time. If you are
interested in joining the email list, please contact diana.halikias@nyu.edu. You
can also view the slides from our first meeting on the website:
dihalikias.github.io/lean-nla.html

Diana Halikias and Haim Avron

For a real Halloween fright… look at floor(x; sigdigits=1) near a power of 10! :face_with_peeking_eye:

Luckily it is fixable:

I’m more surprised that there are functions, other than printing and log10, that hard-code the base 10.

Thanks @MilesCranmer for this very impressive work.
I tried a quite similar approach than yours with Golang compiler.
See GitHub - s-celles/go-in-lean: A proof-of-concept to find correctness bugs in Go's standard library with Lean 4. · GitHub and go-in-lean: Exploring Go correctness with AI agents, Lean 4 and QEMU - Technical Discussion - Go Forum and catched some miscompilation bugs there (much less than in Julia :sweat_smile:) but I’m probably not a good bug wizard.

This reminds me of the recent critique of OpenAI’s Navier–Stokes Lean formalization ([2610.08144] Navier-Stokes lost in translation: Why Lean verification of AI autoformalisation does not guarantee correct natural language proofs). Both cases seem to expose the same fundamental weakness in AI-assisted formal verification: a formally valid proof does not necessarily establish the intended claim.

Perhaps the real challenge is no longer just verifying the proof, but independently verifying that AI is proving the right theorem in the first place.

More fundamentally, this raises a question about AI’s ability to judge what truly matters. Perhaps human judgment remains indispensable precisely because we are the ones who decide what is worth proving.

This is not the issue with the linked Navier-Stokes comment: Humans have checked that the formal claims of the Lean-proof do indeed correspond to the intended natural-language claims we want; and the Lean-kernel has checked that the formal proof is indeed correct and does prove the formal claims.

Instead the issue is: The formal lean proof is not very accessible to humans. It is accompanied with a pretty bad slop natural language proof / writeup. This writeup is also not very accessible to humans (it’s just bad writing).

The linked paper shows that the situation is even worse: The natural language writeup of the proof is not just bad prose; it is incorrect and significantly diverges from the formal Lean proof.

In other words: “dear chatgpt; please solve Navier-Stokes millenium problem. Make no mistakes.”, “sure, here you go. No mistakes: Checked in lean”, “reading the lean is too hard for me. Please ELIH (explain like I’m human)”, “sure, there you go” – and the explanation is not just hard to read, it is objectively wrong, both in that it does not explain the specific lean proof, and that the explanation does not correspond to any valid proof at all.

@MilesCranmer 's approach does have the issue you’re alluding to: We cannot economically check that the formal claims in the lean proof correspond to the natural language claims.

For example LLMs are making formal software verification feasible at a much larger scale \ Anthropic does not have that problem: Anthropic has carefully chosen to start with claims where the correspondence of “formal lean claim” and “natural language claim” is economical to check by human labor.

I don’t think that the “humans cannot economically verify the connection between formal claim and natural language claim” is catastrophic. The other issue – we actually observe something between mistake and bald-faced lie when “de-formalizing” a proof – is imo much worse.

(autoformalization is awesome – it’s RLVR, i.e. has verifiable rewards. Deformalization has no verifiable rewards, which makes it fundamentally harder. We can expect that it will forever lag behind autoformalization)

One of the longstanding critiques of Julia is that it lacks correctness because Julia doesn’t have a strong static typing and relies on dispatch that exposes a lot of surface to subtle bugs. Well, it seems that the solution is to literally brute-force it with AI agents.

Welp, I guess compute can solve a lot of problems.

I absolutely disagree with that. The lesson from this Lean exercise is that strong semantics and programmatically checkable interfaces finds bugs.
That is, Miles didn’t just throw agents at Julia, tasked with finding bugs. He had the agents leverage another language with stronger semantics than Julia in order to find this.

I mean… programmatically checkable interface designs are often proposed in Julia, but they often end up not working well enough because Julia is reliant on a permissive system. Even errors like “undefined variables” or even “no such thing exists” are only catchable runtime error and not something Julia will refuse to compile. When it does work well enough for one class of problem, it still could break generics (or even things like dual number used in autograd) and so on if the check was too conservative. Lean avoids this issue because anything that can be formally proven can be made correct. A standard static analysis tool doesn’t have that resource.

But Lean, ROCQ (or TLA+) are not programming languages “with stronger semantics”, they fall into a different category - they sit above the code and are only used for formal specification and model checking, not for running code AFAIK. Maybe there is a restricted subset of Julia that can be combined with formal verification (see Can LLMs Enable Verification in Mainstream Programming?) or bake formal verification into the compiler, see CompCert - Main page.

I think one of the issues is that formalizing and checking is undecidable in general, which is why type checkers have to be conservative or filled with proof/annotation that programmers don’t like. LLM changes this dynamic by putting the burden of proof out of programmers’ hands.

David Andrews and Chao Zhang tried formalize Python in Lean and also found 400+ bugs.

AI agents wrote 613,318 lines of Lean that copy Python’s internals line by line, tested every piece against the real Python, had a computer check every proof, and turned up 418 bug reports in Python along the way.

The closest project is julia-in-lean, which did this for the Julia language and directly inspired this one

Pretty neat. Nice one @MilesCranmer

No, Lean4 is a full programming language that can run code and do anything a normal language can, like read/write files (IO), connect to the internet, etc. (In fact, the Lean compiler itself is written in Lean.) It even has garbage collection and decent performance (~on par with Julia or C#). The main place it’d be lacking is libraries/ecosystem support for stuff outside of theorem-proving.