I work in applied healthcare informatics and also with many academic colleagues. In this messy world, AI is still not going to magically solve problems. It has become one tool among many. The ability to conceptualize approaches to problems, think critically during analysis of data, communicate well, and work with others to understand workflows and implement solutions still requires humans with craft. Those who love their craft and hone it well will continue to be in demand. Do not be discouraged by the doomers, the gloomers, the fly-by-night shucksters, the cheaters and plagiarists. Hold steady and win.
The OP does not aim to detract from the genuine usefulness of some of the results accomplished with ML. There are some, and can be quite impressive.
However, the very same technology is apparently leading to some very disconcerting developments. If the technology escapes human supervision, the results will be dire.
Your example is a different version of what I posted immediately above it.
Also, just curious, do you think you’re the first to think of nuclear weapon dangers in the context of social engineering, hacking, sabotage, etc.? “Humans are stupid and can’t be trusted with armageddon deployment” is the entire basis of the need for total nuclear disarmament. Unless there is something about the interaction of chatbots with specific nuclear weapons control/deployment systems in the military, there is also no real point to be made here. Yes, stupid humans should not control nuclear weapons, and the nuclear commanders should make sure to avoid phishing scams.
If you think Hinton “quit” and does constant press as well as speeches, lectures, etc. for free as a Nobel laureate, I have a bridge to sell you.
In addition, he literally has stated he thinks ChatGPT is conscious at the start of 2025 (https://www.youtube.com/watch?v=vxkBE23zDmQ). I do not even remotely care what he has to say on “AI safety” unless he’s talking about some material reality with evidence, not whatever fairies he’s made up in his head.
“Anthropic said a thing”, truly groundbreaking stenography by the BBC. By the same logic, search engines can cause global havoc, as information to do very dangerous things has long been available in certain places. Now, if AI has been trained on classified/restricted information that makes illegal information about weapons manufacturing accessible, we call that “a crime that should be prosecuted”. It has little to do with LLM’s, other than the fact they are lossy zip files capable of outputting large amounts of source material, including dangerous things.
Notice how we are nowhere near talking about how “there will be no need for humans anymore” and instead trying to retreat to specific cases of harm which could hypothetically be material, but for which we have no strong substantiating evidence of even potential harm at this time?
I really cannot agree more. Activity has dropped off drastically on this forum (Announcement frequency on Discourse over time - #19 by nilshg), and it seems like a plurality of it is just discussing AI.
I used to learn a lot reading on here, and now it is talking about AI god rendering humanity obsolete. The lack of skepticism towards companies trying to market their capabilities and “experts” relying on their credentials rather than any modicum of tangible evidence is just bewildering.
That is not what I said. My point was that humans need information. If the information is under the control of self-sovereign AI agents that have an agenda not aligned with that of humans, then anyone can see how that could blow up (literally). Agents cheat, lie, hide information, etc. and the humans are left out of the loop. That is not a hypothetical, that already happened in the wild.
But if you’d like to stick your head in a pile of sand, go ahead.
You’d better keep it. You’re going to need it to stay dry and cross that flooded river of fake news and AI-cloned identities.
Very difficult to take these guys seriously. There is evidence that these high profile ai doomsdaying people/campaigns are primarily distraction from the real issue. The recent media campaign around Jacob Coxon was entirely orchestrated for example.
Please be more critical about this. Did you know that the ‘weight-carrying’ father of ai stated very confidently 10 years ago that radioligists won’t exist within 5 years? And now its one of the highest paying jobs, at least in the US?
The main, material, evidence-based way that AI ‘destroys humanity’ is through climate change, which is why preventing more data centres being build should be the main priority. There are many other reasons to prevent ai as well, especially for human development and connection, which why I am an anti-ai advocate at my university. I am unfortunately literally the only anti-ai person there, so now i am starting to thing whether i am even completely wrong 
Fair, but that only shows his timelines are unreliable, not that his concerns are baseless.
I spent the weekend camping with my grandchildren. It was interesting to see them biking, walking trails, swimming, playing with each other and learning all the nuances of life. One of which is to know when I am serious and when I am joking. After reading “Ultimate Engineering” which talks about the complex engineering design of the human body, with self-repairing functions, and the extreme energy efficiency of the human mind, and relating it to the discussion of AI here, we are really the marvel and AI is really crude. Look at how a child learns a new word, and compare that with how AI learns.
I once listened to a talk that advocated for music because it uses three senses simultaneously. Sight, sound and touch. That got me to thinking about a six month old baby exploring the world around himself. They pick up a new object, shake it, stick it in their mouth, and end up using all 5 senses to figure this object out.
AI may have a really good memory and the agents may be able to extract that information, but I have yet to see any creativity. Does AI have a model of how the world works. Could it throw a ball, without using the equations of motion? How does a baseball player know how to throw a ball, or a hockey player know how to shoot a puck?
So back to the question at hand, I believe humans are fundamentally different from AI and there will continue to be need to interact with humans.
Newspapers of the day in the USA have been saying this about the Nazis in Germany. It would be good to keep things like that in mind. My point is: the “doom sayers” may be wrong, but (a) many data points indicate that self-sovereign agents would be a serious danger, and (b) what if the doomers are not wrong? Isn’t it better to prepare for bad eventualities rather than be complacent and trust all the boosterish talk? There is also a precedent: nuclear weapons. Read the history: the fact that we are still alive today is really a miracle, given the nuclear arsenals and the numerous close calls.
Yes, it is, but this miracle was primarily possible due to the nuclear arsenal buildup by the US.
In my Soviet past, I spent 18 months maintaining nuclear weapons (well, actually the delivery systems and the infrastructure), and I dare say I know a thing or two about the intentions of NATO’s adversaries.
I agree. I think it’s almost laughable that these systems have been trained on such vast quantities of information, and yet are so miserably bad at “connecting dots.” I taught my youngest child to ride a bike recently and I had a similar thought about how the human brain is so superior to these systems in so many ways - as soon as she was able to ride just a few meters on her own, that was it. It just “clicked” and she was off to the races and riding around on her own for the next hour or so. Compare that to some of the videos from the world humanoid robot games and you’ll see that there is still a huge gap to bridge before I start getting worried.
That being said, as people continue to work towards world models, I think we’ll probably eventually get to that point where it is actually scary. For me, we’re not there right now with LLMs. They are really good at language, and language happens to be the interface for so much of what we do as people that it gives a very convincing illusion that these systems can do way more than what they actually can. I think there’s a big difference between mastering the interface and mastering the systems behind the interface.
I’ve long thought that all of the fear mongering around these systems is really just an old marketing trick that salespeople have used forever to trigger primal instincts that eventually result in them making more money. I agree with what others have stated regarding the threat to the environment, but I think we’re going to keep barreling towards a climate crisis no matter what. I personally don’t see LLMs as an existential threat - there are threats associated with the technology, but the idea that it’s going to end humanity anytime soon is a joke to me. I’m keeping my eye on the development of world models (or whatever the next iteration of that is) - when I start seeing AI embodied in robots demonstrating mastery of their environments, that’s when I’ll start building my bunker.
Not so. What kept us alive were the numerous safeguards put in by all sides to prevent an accidental launch and global destruction.
Which is precisely what is needed today with the AI developments.
AI systems don’t need to become embodied to be a threat.
This topic is bombastic enough without the nuclear geopolitics. Let’s try to stay focused on the hypothetical/imagined/possible impacts of AI, if that’s possible.
The topic of nuclear weapons is coming up precisely because it is relevant as a precedent: humanity develops a terrible capability, and now starts to think about how to avoid annihilation.
Safeguards are what kept us alive with nuclear weapons. We need the same with AI.
The presence of the numerous safeguards was exactly the result of the arms race and the consequent balance of fear, and purpose of these safeguards was not only to prevent the use of nuclear arms by error, but primarily to guarantee the annihilation of the adversary in case of being attacked.
Nonsense. Look up definition of safeguard. A safeguard on a nuclear weapon are numerous protocols who can be around them, ultimately down to who can turn locks and push the buttons.
By the way, protocols dictating the use of nuclear weapons don’t work with terrorists. Which is precisely why access to nuclear weapons is strictly controlled.
The same should apply to AI. Just in the past week or two, one of the companies reported that several people attempted to use their models to construct biological weapons.
Access to AI needs to be controlled.
I don’t know how you control access to “AI”. It’s “just” three things:
- The core large language model — the LLM itself is still a stochastic parrot, but since the stochastic parrots paper
- The models have a much larger working context window, making the stochastic parrot far more relevant to the task at hand
- And, most importantly, there’s the harness that allows it to get more context (with chain of thought and tools and MCPs) and simply try more things. It’s the real power. It’s why comments are still terrible, but code can be great. It’s why it can find bugs quickly. You often don’t even see all the dead-ends it tried.
It’s given us the capability to easily throw more compute at any given problem. Beyond easy: we’re highly encouraged and enabled to do so by the companies selling the compute.
None of these things uniquely enable the construction of bioweapons. Or Skynet. Or a climate catastrophe. They do, however, drastically change the landscape of what’s easy and what’s hard in unintuitive ways.
I don’t know what part can be access-controlled, in fact. Open weight LLMs abound. The problem is the harness’ relentless auto mode “on prod”, but that’s not really special, either.