root is non interactive on my box… its about will you pay for local autonomous security for the things your concerned and post about, or help develop the technology that gates the agents both are valid paths.
I started my career working for the R&D department of a big newsprint company, when newsprint was a thing. A salesman took me to a pressroom and it was an interesting experience. It taught me who the customer of the newspaper was, in a business sense. The advertisers pay the bills, so they are the customer. And what does the pressroom need to do to pay the bills? The higher the circulation, the more the advertiser pays. And what drives circulation? Good stories, emotion, panic…
I expect clicks have largely replaced circulation, at least up until AI news, so the model has morphed but has the same drivers.
Think back on the big events in the news and try to relate it to this business model.
Almost all of the information we’re getting is spin. You may not be the AGI-hungry investors those companies would consider allies, but they can acknowledge you as their preferred opposition. Several times by now, you’ve linked articles about the dangers of frontier models that are explicitly parroting a narrative from the offending companies. When people looked even a little closer into those narratives, we have found alarmism and self-aggrandizement that makes the opposition look hysterical and the companies look cutting-edge at once. The public conversation has shifted away from specific abuses and crimes to the companies’ preferred misdemeanors to AI being broadly scary, which is far less feasible to regulate. Senators can introduce as many bills as they can to ban “superintelligence” and “systems that humanity cannot fully control”; that’ll either be stuck in endless debate over legal definitions or accomplish nothing of consequence. Meanwhile, there are laws about copyright, cyberattacks, nonconsensual deepfakes, etc. that needed a lot more debate among legislators and their constituents over scope, liability, and enforcement.
Everyone seems to agree that a model escaping human control is a distinct possibility.
If that happens, all other concerns will not be of much weight anymore.
Distinct possibility? It has not only already happened, it is routine. There is no firm line between LLM hallucinations and all these disclosed cases of agents going rogue, and some cases explicitly describe agents operating under hallucinated assumptions. The very idea that the only harm that matters hasn’t occurred quite yet is exactly what these companies want us to think. Of course, what matters is a matter of opinion, but while anybody is free to fall in line with the companies’ propaganda, the families of people who have died taking advice from chatbots would reasonably disagree. If you insist on a doomsday scenario above all else, I have already provided news coverage of two nuclear-armed nations almost coming into direct conflict over hallucinated nuclear weapons materials in an intelligence report. Funny how the industry repeatedly encourages discussion about Skynet taking over nukes, but not that.
Not everyone. In any case, it depends on the definition of “escaping human control.” If I give an agent full access with admin privileges to my command line, there is a (very low) chance that it (he? she?) will erase my disk. That’s unpleasant, but usually not a matter of life and death. Besides, I wouldn’t grant it admin privileges in the first place. And I have a backup on a physically disconnected hard disk. And important files are synchronized to Dropbox, etc. In the few published cases where AI destroyed somebody’s database, the actual problem was sitting in front of the monitor.
There is a much higher chance of hackers doing bad things with the help of AI, but then it was hackers, not AI.
For really catastrophic scenarios, I fail to see what a model could actually do - or why.
Generally, I see the probability of very bad guys doing very bad things with help from AI at 100%, but that is about the bad guys, not about AI. And you actually will need AI to be able to stop them.
I have used AI system with some “success” for a few months. I have also seen limitations in them.
My experience is that you really need to know what you are doing/asking the AI system about in order to use it successfully: you need to know whether the response makes sense or not. So I tell my students: feel free to use it, but I really think you should master the topic before you use AI.
Can AI systems “kill us all”? I suspect that AI systems can help some bad actors to increase their chances of developing terror weapons. How big a risk this is, I am not the right person to say.
Personally, I am more concerned with the ideas of “agentic AI”. My impression is that “agentic AI” will make observations from the information it has access to. Then it will take some action. The response to that action is then observed, etc.
This creates a feedback loop. And feedback loops can go unstable. The economic crash in 1987 supposedly was a result of many people using stock marked prediction models that had been “trained” on data during development, and which did not contain the effects of widespread adoption.
Use of agentic AI perhaps in combination with ill will can most likely bring down banking systems, vulnerable industrial production, hospitals, food distribution, etc.
I don’t think the “killer feature” is soldier robots going berserk. In late April 2025, the electricity system went down in Spain and Portugal. Without electricity, elder people in their wheel chairs could not use elevators to get back to their apartments. Groceries and restaurants did not sell food, because the payment system didn’t work. It took some 12 hours to get electricity running, and I had to starve for 24 hours. Not a big deal, but a more severe crash could easily take down important systems that we base our life on for days or weeks.
I think there is a distinct possibility that we are not talking about the same thing. AI systems going off to do bad things is routine. But an AI systems going off for good (that is not being responsive to human control; or not responsive in the right way – “go away” does not count as being responsive), that hasn’t happened (yet).
What!? You are some of the smartest most creative and inventive people I know - and I know many - so, although I came here to look into
? can I store an SVector representing an NDim tuple into a Matrix to operate efficiently as a database search key ? ;
I am instead going to try to summarize my oodles of notes on don’t be fooled or manipulated
by (Oligopoly want to be Monopoly) Ai companies marketing hype and doomerism in their attempt at Regulatory Capture to limit the competitive threat of (Chinese, NVIDIA, Google) local open source weights, and the real Dangers of Cognitive Surrender of humans to Ai.
I have oodles of notes( too many really 
), but here is a A good start:
Some relevant posts from Facebook :
Rediscover self-organised learning environments (SOLEs), after two decades of published research, but “not invented here”
Cognitive laziness → Cognitive surrender → Non-thinking pawns
Very valid concerns. Figuring out problems by yourself is like exercising a muscle.
Sure if you have an exoskeleton you can lift a heavier weight.
But when that exo is off-line you better have well trained muscles.
(see recent China study on AI vs homework and exam grades)
And the other point point of empowering curiosity and creativity is very good too.
Again students who can think for themselves stand better in this area.
and MIT Report – AI and Education - Some real Dangers of Cognitive Surrender of humans to Ai
https://aiandeducation.mit.edu/report/
2.7. Augmentation not automation
As AI-enabled technologies become more capable, it will be possible – and tempting – to offload more “thinking” tasks to them; not surprisingly, students told us the temptation was greatest when they feared they would miss a deadline. There are early signals, however, that overreliance on chatbots can have a range of significant negative consequences
– diminishing critical thinking,
- weakening memory,
- eroding confidence, and
- undermining mastery.
Getting the right answer from a chatbot can create the illusion of learning – but it can also trigger “cognitive surrender”4, where students fall back on AI at the first hint of struggle.
If that’s what you’re really afraid of, then that’s not new to software either. People routinely resort to very brute hardware and network shutdowns. The kind of runaway automated selloffs, like Black Monday as mentioned by BLI, aren’t even that sophisticated or unusual; the KOSPI crashed even harder in June and July in a loop that not enough people anticipated from margin traders abusing a seemingly innocuous kind of security approved weeks earlier. Of course, kill switches are much harder to pull off in highly distributed systems using the same data centers as other critical systems. There is a worthwhile conversation to be had, but again, the industry wants everyone to talk about increasingly science-fictional difficulties instead of the current harms and lack of accountability. I guarantee if legislators introduced consequences for the human overseers of rogue AI, we will suddenly hear a lot more about addressing negligent cybersecurity failures than AGI kill switches.
Let’s hear from the other side: Huang said on Sunday that “2030 is not going to be the end of the world. There is 0% chance that’s going to be the end of the world. Scaring people is unnecessary. It is irresponsible."
A totally disinterested and unbiased statement, of course.
Here’s not “the other side” but rather a completely different perspective: LLMs are real, AI is fake from Cory Doctorow.
In particular, I completely agree with his analysis that these recent “hacker agents” did not go “rogue” or set their own agendas.
Which is not to say that the OpenAI/Hugging Face hack is nothing. It’s something, all right: but it’s a specific something, with an explicable, even foreseeable trajectory. Once you understand that these are chatbots that were designed to complete challenges like this, using tactics like this, you can understand that the chatbots didn’t “go rogue.” They did what they were designed to do, and because OpenAI ran them with inadequate supervision (without a “human in the loop” that checked each iteration through the Python loop to ensure it hadn’t gone off the rails), they trashed a competitor’s servers.
Designing autonomous, malicious software is generally considered irresponsible and dangerous.
The fact that OpenAI is making it much easier for unskilled people to break into and damage servers is indeed very bad news, but it’s not new bad news.
The bits of AI that scare me are the bits that it uniquely unlocks. Like enabling bad actors to overwhelm authentic humans in online content. Like enabling bad actors to unleash an agent swarm to gather up all the PII that’s been leaked about everyone over the years and start making compelling social engineering calls to take over everyone’s bank/voting/retirement/life accounts. This used to be something that only “prominent” (extremely wealthy) or “soft” targets (e.g., untrained elderly) needed to worry about. These things used to be done poorly and/or with real people on the other side. Now they can be automated at a higher level. The same is true of this next-level “script kiddie.”
Of course, Huang’s chip-selling ship is sailing.
You’re doing a whole lot of anthropomorphizing of chatbots.
In the recursive prompt loop that actually makes an “agent” run, where is the “they”? Where’s the intent? What part of running the LLM recursively in a looping harness with elevated OS privileges gives it that property?
Yes, running LLMs recursively with powerful privileges and powerful software harnesses can allow them to do whacky things. Playing a game of stochastic telephone with a powerful tool that has no capability of normative judgment is very stupid. That goes for OpenAI et al, who committed negligent crimes by choice, as well as anyone else who chooses to attach a lawnmower blade onto a loose firehose and lets it wave around. “I’m shocked it tried to slash the mailman!”, area hoseblade exec says.
“Agents cheat, lie, hide information, etc. and the humans are left out of the loop” - if you’re talking about the marketing press releases about the Huggingface hack, etc. -
-
See above. The chatbots do not have agency. They generate plausible answers based on training data. Everything the agents did/“said” in these hacks is extremely plausible based on their training data. The people promoting anthropomorphized narratives are generally associated with pseudo-religious TESCREAL ideologies.
-
OpenAI et al CHOSE to leave humans out of the loop. No one forced them to recursively run chatbots with powerful hacking harnesses recursively for long amounts of time, and it is unsurprising that these actions resulted in whacky detours and crimes. LLM outputs CAN be monitored, and companies have chosen not to do so on these hacking benchmarks. If you install an LLM/“agent” on your PC, it will do nothing until you prompt it to do so. Once you do so, it will generate plausible outputs. If you allow recursive prompting, it is likely to chain incorporating a variety of weird outputs into later prompts.
I’ve been saying for literal years that the use case for “AI” is spam/misinformation. Indeed, most of the internet now is likely LLM slop. That is tangible and obviously bad. There are a lot of tangible and obviously bad things about LLMs and current “AI” movements. Further automating identity theft is also obviously bad. Spam is largely used to steal from vulnerable people, and heavily botted social media ecosystems / slop “news” farms/websites have enormously polluted the internet already.
My criticisms in this thread are focused on unsound and always-vague deferral to anthropomorphized narratives about LLMs and us “losing control”, which are utterly dominated in the recent news cycle by TESCREAL ideologies. Note how “misinformation and spam firehoses are bad” is rather distinct from suggesting the world is probably ending, “There will be no need for humans anymore”, and thinking ChatGPT is sentient.
For example, this is basically verbatim what rationalists/EA/TESCREAL ideologies also say:
They use the phrase “Pascal’s mugging” to describe the decision making, and generally they radically reject empiricism in favor of in-group thought experiments.
Coxon, for instance, is a (lesswrong) rationalist, in addition to the fact that his press cycle was not even remotely organic (as @Datseris mentioned). Numerous people clocked him from the quote “crunch time for humanity”, a Yudkowsky phrase, but he also responded on Twitter to someone with a confirmation of his idolization of Rationalists. These guys have been around for decades and have lots of in-group words/phrases/tropes that they refer to constantly.
TESCREAL beliefs are deeply unserious and based on decades of what is mostly fanfiction writing; feel free to read recent work by Cal Newport and Adam Becker about the specific beliefs, claims, and egregiously bad epistemic standards of the TESCREAL bundle. Hinton can think GPT 4 or whatever is conscious if he likes; his claims will be preceded by actual evidence if they are worth caring about.
“Alignment”, for instance, is basically a thought experiment created by Yudkowsky based on anthropomorphizing computers and having utterly magical ideas about “intelligence”. Yudkowsky is a deeply unserious middle school dropout who started a (harmful) Bay Area pseudo-religious AI god/doom cult, getting notoriety based on his Harry Potter fanfiction, and his ideas are now extremely prevalent at the top of the major AI companies. Presented without further comment:
Correct - currently we have very powerful and extremely subsidized brute force tools that can enable bad actors to do certain things to a worse or broader degree. Large matrices of training weights, drum roll plz, have no agency, but with enough compute and well-written harnesses they can arrive at plausible (and sometimes harmful) outputs.
We also have governments that treat capital as a sovereign, holy entity, so any sort of sensible limits on monopolistic practices, collusion (see: the RAM cartel engaging in likely price fixing and driving up prices 500%: I knew we weren't crazy! Samsung, SK Hynix and Micron just got sued — accused of using 'AI' to fix prices and screw your wallet in RAMpocalypse | Tom's Guide), and criminal negligence (recent hacks, horrifying cases of chatbots helping individuals end their lives) are not in the cards right now. Mix it in with “innovation” nationalism / sinophobia (in the US at least), and here we are.
Positively refreshing to see people talking about the actual nuts and bolts as opposed to TESCREAL magic and buzzwords.
Yes, you are correct sir 
essentially confirmed here by: Just follow the money here with Tom Bilyeu who is a very successful businessman and he knows about successful product launches, company stock IPO for public markets, etc. and so he is entirely credible here that American Ai companies based on LLM have billions of incentives to develop, market and sell a narrative of fear mongering for government Regulatory Capture to create a “legal” moat against competition for local open source open weight LLMs from China Qwen, Kimi ; NVidia Nemotron ; Google Gemma etc. for all the details about ‘To understand why - Just follow the money incentives’ start at &t=7m here https://youtu.be/87uXrdOdZ5o?si=i7VIsJbDW6X6DioR&t=423
Correct - the current “doom hype” has netted AI companies many millions of dollars in free/“earned” media. There’s distinctly “excitement/edgy/scary/cool/captivating” feelings on the flip side of the “doom” coin, and in TESCREAL movements it is “AI utopia” vs. “AI doom”. And unfortunately most people just don’t have the technical background nor knowledge in the absurdities of these formerly-obscure and irrelevant TESCREAL ideologies, so the marketing has a lot of “organic” traction.
Crushing competition in open source and other countries, creating fear/hype to pump IPOs - “the purpose of a [hype cycle] is what it does.”
Edit: I’ll add this for the topic generally - here is a measured and materialist summary/critique of the “hot AI doom summer” that we’ve had: Don’t be fooled by this summer of AI hype | MIT Technology Review
Further edit: a 2nd article that’s relevant - the private AI firms are knowingly and intentionally engaging in unethical and criminal behavior, probably in large part because they know they will not be held accountable: ‘Doom Loop’: OpenAI and Microsoft Admits LLMs Are Destroying the Web and Built on Theft
Hmm. How about this?
Knowing that the potential for a disaster is real, the AI companies start “scaring” people. Then, the people who point out the dangers are shouted down and drowned in cries of “doom hype” and “TESCREAL naives” etc. So, the population does not know what to believe, and does not take warnings about AI catastrophic impacts seriously. Mission accomplished.
This is why the population needs to know how to think critically, to try and understand the technology, so that they can separate the hype from fact. This is why rigorous teaching emphasizes going to the source material. It is easy to cherry pick quotes and believe the most charismatic orators.
We can all start by having dinner table conversations with our children, or grandchildren, and encouraging them to learn to think critically for themselves, and how to evaluate the reliability of the information that they receive. And for the teachers in this forum there is an even greater responsibility.
Opinion # Don’t be fooled by this summer of AI hype
Breathless claims about Ai NOW is AGI with
new capabilities fall apart pretty quickly under scrutiny.
MIT Technology Review
Ps> Thanks for sharing @harvestdeparture729 , I re-posted for emphasis.
Another solid article on the topic posted in the last hour! Why AI companies can’t be trusted to self-regulate | Nature
At least, I have generally high confidence in Dr. Khaalf’s analysis - unfortunately I cannot read the whole article due to the paywall (at the moment). She is generally quite grounded and specific in her analysis.