I think that the most valuable source of new training data, these days, comes from the history transcripts of users iterating over AI productions to perfect it. I wouldn’t be surprised that the main motivation behind, for example, the Claude Team plan for scientists, is the gathering of high quality training data, rather than improving their reputation or advancing science.
On the “AI requires human knowledge to improve” statement, I’m not confident this will hold in the medium or long term. In 2017, AlphaGo Zero was trained without any use of human knowledge, just the rules of Go and playing against itself, and was miles stronger than humans. Human knowledge started the machine for general intelligence, but I think that soon enough, LLMs agent(s swarms) will largely go beyond human knowledge and capabilities on tasks they can be trained on without the need for real-word feedback to assess correctness, like math and computer sciences, and the mathematical part of physics.
On the future of math, physics, engineering, I’m afraid these will become more and more elite disciplines for which fewer and fewer people can find jobs and be trained at a high level. I think that the question of whether we want LLMs to do something for “us” will be answered by the people that hire “us”, and unfortunately, their might be quick to realize that their money will be better spend on 10 humans and 100 computers than on a 100 humans…
If your question is entirely about the merits of replacing human labor in STEM, then it has already been answered by happening dozens of times over already. Julia among other things owes its existence to machines replacing thousands, then millions, then billions of human calculators at a time. Machine learning was already established in several fields before LLMs and agents.
I think the questions about the ethics and sustainability of pushing people out of a mental pursuit are much more difficult. For one, AlphaGo never came close to replacing the human culture because it was obvious that pitting machines against each other in games is pointless no matter how novel they are. Pure math is arguably not that far off from games. For another, it’s not 100 computers anymore, it’s data centers that privilege big tech companies and stress the local environment. As much as we like to gripe about the pace and complications of human labor, it’s green energy.
Unfortunately, mostly not. Human labor relies on food and goods (including the military goods used in wars), and producing those is the leading cause of habitat loss. On top of that, we generate billions of tons of waste, much of which is plastic, and we are responsible for the biodiversity crisis. Automation doesn’t eliminate that footprint: people who lose their jobs to AI still need food and goods, so the AI’s energy use is added to ours rather than replacing it.
True, I was mostly thinking about the fact we eat to fuel our brains and muscles at work, but I guess separating that from everything else in our life wasn’t realistic. The energy we use just for temperature at desk jobs hasn’t been green for a while. It’s still worthwhile to put living humans to work instead of using up more electricity, but I have no real idea what fraction of total energy expenditure that involves.
My own take on pure math and AI comes from decades of having a secret weapon: computation. I computed to find out what was true, then proved what I could. APL, then MATLAB, then Julia (with some help from Mathematica) shortened my time to thought. Many of my colleagues used computers for email and LaTeX, and never felt computation was at their fingertips.
So for me, the fun was always in the exploration, and the proof was often the toll I paid afterward. Now my favorite tool is on steroids, and the toll is getting cheaper. I find that exciting. Technical mastery of the old kind might matter less. What will matter more is how we use these new tools in creative ways we haven’t yet seen. I wonder whether my colleagues would be so discouraged if they had had the same computational life.
Whether universities and society will keep supporting mathematics is a separate question, and there I share Steven’s worry. But I don’t think the mathematics itself has died.
AI is also creating whole new approaches to applied mathematics, but that’s a subject for another day.
The point is that there’s nothing left to do. What’s the point of doing the exploration yourself when the AI can do it faster and better? Sure, you can do it for fun but then you’re a child playing in a sandbox, you’re not doing mathematics.
Hm, in the same sense you could ask: What is the sense in walking when cars exist? What is the sense in buying a car when planes exist?
On the other hand, LLMs (there is no intelligence, just statistics) are helpful in several areas, finding edge cases is where I found them reasonable to find them (definetly not to solve them). LLMs found counter examples here and there – as of now at least suspected and probably correct ones. Do they explain them well? No. Did they find a proof that is not a counter example? No.
Sure, it is unclear where all this will lead, and we should all be careful in critically reflect usage of LLMs and discuss the social effect on that, but “doomerism” does not help. Same as with walking versus driving, just because cars exist, we should not all but of our legs.
I do math – and programming – both as a job and for fun, but I do not have sand in my office – maybe besides gravel that will soon join our sidewalks in icy winter again.
I was being imprecise: I wasn’t talking about the present capabilities of LLMs, but near-future ones. Right now for example they suck at theory-building, and that’s what I’m writing a paper on. I just hope I manage to finish it before LLMs take it from us as well.
That’s not my experience at all. What we’ve seen so far is that the AI agents are good at pushing against boundaries that were already well-probed by humans. In my own daily work I’ve seen evidence of limited creativity from the agents, but most of the genuine breakthroughs are still coming from my own head. What I’m finding is that backburner applied math projects that have been sitting unfinished (for person-power reasons) on my hard drive for, in some cases, up to 15 years, it’s now time to bring them to fruition and get them out the door. I am finally optimistic I can get some of these published.
Agreed, though, that the future is unpredictable. Unless you believe in magic, there’s no reason to doubt that eventually they will do the more creative parts too. I am pushing pretty hard to get a bunch of stuff released soon in part because I don’t know what the future holds.
I have not yet seen any useful creativity in LLMs (I will not call them intelligent, that is not well-defined and I do not want to use marketing speech from these companies).
For math beyond exercise tasks, where it gets numbers correct by now, I did not find it useful. Exploration into literature maybe, when carefully checked against.
And concerning help with old projects that wait to the person-power – I think it is dangerous to try to rush things with LLMs, one still has to careful check what these produce. I currently “only” do some bug fixing on my Julia packages, and there it is already exhausting. It finds bugs, but fixing them it is really not good at. So I would not trust it with research tasks. No my research is not as high-level that I would fear it being “stolen”, but what I want to put out in research papers should be helpful, true, and readable.
So I prefer to take a slower route and be even more thorough. Maybe that is not the general trend we see, where everything has to be rushed – right. I stopped doing that a bit by now.
I think this common viewpoint has been thoroughly refuted by recent events, at least regarding to frontier and currently-internal models.
If proving Navier-Stokes-blowup, quasi-Riemann, O(2.25) matmul, little-o N log N FFT, etc is not proof of intelligence… then what would “intelligence” even mean?
It’s not like these proofs are inhuman SAT-solver-style. They are “real” (the writeups are atrocious, though).
The LLMs “partially understand” the related mathematical literature, in the most objective sense there could be: Reading (i.e. adding-to-context-window, even without adding-to-training-data for gradient descent) related papers helps the LLM develop novel proofs for related theorems.
[…] no finite set of ‘impressive’ observations about AIs/LLMs establishes that they exhibit a (human) cognitive capacity. This link can only be theoretically established; i.e., if we propose that a machine has a cognitive capacity X, we must also formally charactise X such that the machine’s capacity Y can be mathematically proven to be X \equiv Y under relevant and naturalistic conditions, using appropriate proof techniques (see, e.g. Blokpoel & van Rooij, 2021, Chapter 5).
Agreed that checking is non-optional. There’s nothing good to say about having an AI agent do all the math and writing and then hitting “submit”: it is pollution, pure and simple. If the AI has discovered something important that you might not have been able to do on your own, at least take the time to write it up in a way that makes sense to humans. arXiv must never become an interprocess communication protocol for agents: it is by us and for us.
Your comment is so divergent with my own experience that it makes me wonder if we are using the same agents.
My experience with Fable and Opus 5.5 indicates that they are roughly as likely to make errors as I am (caveat: I only use them for tasks where I have concluded they are useful). For fun, just a day ago I had AI “review” a peer-reviewed publication of mine, https://www.cell.com/iscience/fulltext/S2589-0042(26)00083-0, and ask it to find errors or holes in any of our statements. The content of that manuscript was 98% finished before the agents became good, and everything in it was carefully checked (read and edited word-for-word, slowly) by me, though I know my co-author used AI to help with the English language issues so I can’t honestly say it was 100% AI-free. Despite our care, it found a minor (not conclusion-altering), verifiable hole in one of our statements, one that I checked during my review. I need to do a bit more checking but then will likely submit a correction to the journal.
What I find is that the AI agents are good at some things I’m less good at, and vice versa. It makes them somewhat frustrating to work with sometimes, but also a good collaborator.
a strict rule not to use git – even hard coded, so it has to ask permission if it comes to the idea to do so – is broken about 1-2 times a day with a popup asking for permissions
instructed to propose fixes for bugs or errors or inaccuracies that align with the overall scheme of a package, it still proposes to often e.g. document the wrong functions
when it “feels” a method is missing, it proposes how to fix that, makes up some math (numerical differential geometry). Even when instructed that only formulae from published literature (arXiv, better DOI) are allowed – it makes up wrong formulae that are not correct
this “feels” is, that it considers that function missing a “bug” – while in reality it is much more likely that function does not exist, because the corresponding function does not yet exist in literature, like for example the logarithmic map on the Stiefel manifold.
(Yes, also Fable / Opus 5.5 – yes not the free model)
And when pointed to such a mistake and asked to reiterate, the main thing it does, is trying to agree with what I said. Then changes the conclusion back and forth, that e.g. it is a bug, it is not a but, it is a bug,…
So pointing to potential bugs and finding the small bugs, there its okay-ish.
For example also sign errors in exercise problems for my current math course for engineers – those an LLM finds easily and proposes the correct fixes. So generating TeX formulae there, that works ok.
And even then, in my opinion, frustrating to work with, because even then it does not stick to rules that are set up (e.g. would still ask for git permissions every now and then).
Agreed that they are not good at following rules strictly. I’ve not tried to block them from using git, and I can imagine they are very thoroughly rewarded for using it during their training (just as we are), but I’ve found that my devops rules have worked fairly well to limit egregious behavior. Early on I had an agent reply to a user or two directly on GitHub (aka, impersonating me) which is not OK, but that guidance in my CLAUDE.md has been more effective than I was anticipating. I was kind of expecting I’d have to create a second GitHub account just so my bot could check CI failures and the like while blocking it from using my own account, and that really hasn’t proven to be necessary.
For some of your other points: I’ve seen all of those, but it sounds like you may be experiencing them to a greater extent than I do? Or we just have different tolerances for agent failings? While I bet you know everything I do, for the record there are a few habits I use that I consider quite critical, mostly around context-window management:
before adopting a formula or method from the literature, I ask the agent to delegate a subagent to read the full text of a manuscript, with a clear directive on what it’s looking for, and then write a compact summary for the parent agent to read. This keeps the implementing context short while providing the guarantees of a full-text read.
it’s really really critical to keep your context windows short. I get nervous when they get above 250k despite having a ceiling of 1M, and I check /context obsessively during interactive sessions. Aim for each session to produce a single deliverable, and then shut them down and start fresh for the next.
agents have U-shaped attention and perform badly in the presence of distractors. (The frontier moves so quickly these may be partially outdated, but personal experience suggests they are still relevant.) If you start changing direction midstream during a session, everything falls to pieces: they may note the change in the next few prompts but then go back to the original plan a few steps later, leading to a weird frankensteinian combination of code/prose/approach/whatever. Just trash all the changes from that session, it is worse than useless, and start in a fresh session with unambiguous direction.
I’ve been having an existential crisis with this. This time last year i was using github copilot as fancy autocomplete and to catch typos in my PRs. I had also started using it to document APIs, but that wasn’t very effective.
I’m now directing agents to implement new capabilities, find bugs, help me write examples, add and check against original references, improve performance, increase test coverage, formalise interfaces, improve type stability, help me generalise algorithms, etc.
I could have done all this, but it would have taken me years, and the result would have been buggier and needed multiple refactors to get just right. Now if i don’t like an implementation i don’t have to spend a few days to a week painstakingly tracing all the things i need to change, i just tell the thing what i want and it goes away and does it in a hour following the rules i have set up.
I’m not using it for solving new problems, but for pushing the frontiers and integrating previously out-of-scope developments. It’s been like having an army of experts working on something that took me hundreds or thousands of hours to painstakingly craft on my own.
I set up a slew of standards and workflows to monitor and ensure quality. They measure and test things so the AI knows when it’s going astray. And it fixes the issues if they do. It now feels irresponsible to write code myself for a package that is meant to provide real value to users. These clankers are much more thorough, fast and accurate than me. They are better than me at following the architecture, rules and standards i had set up for myself. They’re also quite good at spotting ways to improve on my ground rules.
I feel both trapped and freed at the same time. Freed to persue other things, but trapped in maximising the use of these tools to produce the best free tool that i can.
Two important points here: 1. Which model you are using matters a lot, frontier models perform significantly better than the ‘free’ options. And even between paid models there can be large differences. 2. Due to the development speed what year those experiences occurred also matters a lot. I would disregard any experience before 2026 entirely because the models have changed so much since then. And yes this is big tech marketing, but there is also truth to it. I was skeptical and I didn’t think it was possible until last year, but the models are still improving.
This is not strictly speaking the fault of the AI. The way I understand it is the architecture is basically a super competent LLM at the core, that is restricted by a ‘harness’. Basically rules, automation, and often other models, simply to keep it in line and reduce hallucinations so forth. Anthrophic’s auto mode has a classifier that will block many commands that carry risk, for example I often see it blocking a web search that is too wide. To some degree you can also control the harness yourself, which is what Tim was talking about.
While I am far away from pure math or mathematical proofs as a regular professional activity, the topic and its repercussions trigger me as a human being that ultimately is affected by mathematics and its use. So allow me to come up with just a few simple observations and maybe a different perspective on the “mathpocalypse.”
Paul Erdõs (if I remember this somewhat correctly) dreamed of The Book containing the most beautiful mathematical proofs. If AI were to discover it (I understand the current proofs are not really beautiful) eventually, would that really diminish mathematics? Wouldn’t we be delighted in finding that book after all?
I understand that proofs need to be understandable and that the “road” to the final QED and its intermediary discoveries are what may really count (at least in pure math). But why stop at mathematicians? Today’s societies already expect ordinary citizens to simply “trust” scientific results they cannot fully understand. (We all currently see where that is going…)
Couldn’t it be that AI offers a remarkable opportunity to democratize scientific understanding and overcome the increasing fragmentation of knowledge that, for example, Robert Musil lamented in his “Der Mann ohne Eigenschaften” (e.g., one is an expert in an increasingly diminishing area, while being rather dumb and mediocre outside it)? Science started out as “mere” philosophy and then split into a plethora of extremely specialized fields. Maybe we are on the verge of becoming Renaissance thinkers again, who don’t know everything but can speculate and inquire more freely across the borders of disciplines?
Some of this strikes me as a somewhat hopeful prospect.
If progress continues at this rate, much of the new AI-assisted mathematics and science may soon exceed what the human mind can fully understand. People will need AI to verify results they won’t be able to comprehend. This new kind of science is arguably driven by productivity and it is understandable that many may find this a frightening prospect. Even the most talented scientists may be outpaced by those with the most computing power.