Can’t help but think of…
On that train all graphite and glitter
Undersea by rail
Ninety minutes from New York to Paris
(more leisure for artists everywhere)
A just machine to make big decisions
Programmed by fellows with compassion and vision
We’ll be clean when their work is done
We’ll be eternally free yes and eternally young
=================== Donald Fagen, Nightfly, IGY
That is very true. In fact, western understanding of what is ethical will probably differ from what the Chinese understand there. And that may matter hugely in AI: in China, enterpreneurs living by “get rich any way you can” will most likely not consider ethical considerations as an obstacle. A good read/listen to illustrate: Kai-Fu Lee, AI Superpowers.
An interesting take by Jessica Hullman.
https://statmodeling.stat.columbia.edu/2023/05/31/jurassic-ai-extinction/
A lot of interesting comments too, there. It’s all said, we can stop here 
From the Jurassic AI article:
It puts you in the headlines for your altruistic urges, but under a concern that’s indefinite enough that an equally vague gesture toward solutions, like “we need to focus more on responsible AI or AI safety” feels satisfactory. At least for me, extinction is not very emotionally evocative. It’s a nice clean way to signal terror without having to get too specific.
(rhetorical question) Why do the people “in charge” of the AI models want Congress – people who know next to nothing about AI in general – to regulate them when the companies – who know everything about their own models – could just regulate themselves? Are they just admitting that they are too greedy to stop themselves even as they believe they’re causing an extinction?
(edit: I mention regulation only to point out that the people ostensibly in charge of releasing their models to public are very good at pointing out what can go wrong, and yet seem to have no ideas on how to fix them.)
My cynical take is that they want to avoid competition. Now that there are capable open-source LLMs, their business model may collapse. They want all AI activity to be regulated (forbidden) for all except the most expert – themselves.
In addition to @mbaz’s answer (which I agree with, and I think it is the main motivation), I think that those companies would also like “regulation” effectively legalizing use of original material without concerns about copyright.
My impression is that now it is sort of in the gray zone, and very few have taken corporations with deep pockets to court. I hope that someone does it in the EU. Otherwise, you may see your code with minimal modifications as “a suggestion by whateverGPT”.
Of course that will not help these American juggernauts where it matters: in China. AI in China is expanding in a very permissive environment (the governments actively support its development and monetization), with very substantial amount of data.
Why don’t we ask a robot?