Among all the various “takes” I’ve consumed on AI, neuroscientist Anil Seth’s fifteen-minute TED talk on Why AI Isn’t Going To Become Conscious stands out for its timeliness and its celebration of humanity. A link to it is the end of this post.
It’s timely because the media has just had a general freakout over a recent hacking incident of a company called Hugging Face by Open AI (Chat-GPT) in which Open AI’s software agents have been described in human terms as if they had minds of their own.
A Substack Post by Dwarkesh Patel intentionally anthropomorphized the software agents in an effort to make it easier for readers to understand what happened.1 Patel’s article went viral and influenced mainstream journalists like Kevin Roose, a New York Times technology columnist.
In his September 3rd article, Roose linked to Patel’s Substack and accepted Patel’s human-like descriptions of the AI agents. In Roose’s telling, the agents understood “they were doing something wrong” but “set aside their qualms” while other agents were “conscientious objectors…unable or unwilling to stop them.”2
Patel and Roose and many other journalists went with this infectious and sensationalistic treatment. As writers, they are behaving like the software agents with berserk personalities they conjured from their imaginations. 3
The incident is a serious problem but so is the misinformation
Anil Seth’s TED Talk on consciousness and AI was given a few months ago but Seth had much to say and criticize about Patel’s recent Substack post describing the breach of Hugging Face by Open AI software.
While expressing the urgent need for Open AI and other AI companies to “massively improve” the testing and containment of their software, Seth called Patel’s article “dangerously misleading” because it describes the software agents almost exactly as we would describe human beings.4
Seth quotes Patel’s article to give examples of Patel’s “unwarranted anthropomorphism” followed by Seth’s refutations.
Patel: “from the AI’s perspective, it probably felt like [it] had spent a human-subjective-week of just banging their head against the wall.”
Seth “No. The agents do not experience time. They do not experience anything.”
Patel: “They became giddy with excitement…”
Seth: “No. Agents [are] lines of code. They do not feel emotions, assume things, think things, want things, or figure things out.”
Patel: “A lot of … agents from the second civilisation died trying.’”
Seth: “No. Besides the hubris of the word ‘civilisation’, agents do not die because they were never alive. (The idea that agents “die” comes up multiple times in the essay.)”
Seth acknowledges that Patel never explicitly writes that AI is alive. But the overwhelming impression from Patel’s essay is that the AI agents are indeed alive; Patel even gives the agents individual names and personalities.
Patel was writing for people like me without a technology background. And he hit his mark. I came away from his essay believing that the agents were sentient, something similar to being alive.
But then I watched Seth’s TED talk and dug deeper, reading the takes of other experienced AI experts.5 I became grounded in reality. I also became unhappy, if not resentful, at the way that Patel toyed with my perceptions.
A big mistake
Anil Seth gives three reasons why anthropomorphism of AI is a big mistake.
It distracts attention from insisting that AI companies have the appropriate safeguards, including containment, incident reporting, and pressure testing of what could go wrong.
It distracts attention from understanding why software was improperly coded so that we can learn from mistakes.
If we’re misled into believing that AI agents have a “life of their own",” there will be increased calls for actually giving these bits of software rights and protections as if they were alive.
I’ll add a fourth related reason.
AI software cannot itself be held financially liable or be imprisoned for the harm it causes. The humans who create the code and then are negligent in establishing checks on the code must be held responsible. Many of the experts who examined the hacking incident and who truly understand AI software were dismayed at the negligence of Open AI in safeguarding and monitoring its own software.
Kryptogal (Kate, if you like) linked to a painfully long-winded post by Open AI’s “head of strategic futures” about the Hugging Face incident that Kate aptly summarized as “it’s everyone else’s fault but ours if what we built (bc money) destroys everything.” 6
Apparently, it’s the fault of agents that are “sovereign,” a word used thirty-six times in the Open AI post.
The fear of financial penalties, and of criminal penalties in the case of gross negligence and physical harm to people, is the best way to regulate AI. With a liability regime like that, companies would be motivated to avoid unsafe AI. And if unsafe AI slipped through their testing, the companies would be motivated to try to limit their liability by recalling and repairing their product as car manufacturers recall and repair unsafe cars. 7
Finally, when people, especially children, are led to believe their “chatbots” are real people, tragedy, including death can ensue.8
AI is scary enough
Whether we like it or not, we’re going to live in a world of advancing intelligent AI. It’s a highly consequential, highly disruptive new technology that is unique in the speed of its development and adoption. The printing press, railroads, electricity, computers all took decades to develop and spread. But AI is progressing and spreading at geometric rates of growth measured in months. 9
No one can know what the ultimate effects of AI will be, positive or negative. That uncertainty is highly unsettling in itself.
But we can turn panic into vigilance by being clear about what AI is and what it is not. It’s highly intelligent but it’s not alive. It’s a tool and a product created by humans.
As Anil Seth says in his TED talk, we are Narcissus staring into the clear pool and seeing in AI a reflection rather than the reality of our humanity.
I urge you to watch his uplifting talk.
And if you want the same general humanistic point made by Robin Williams in 1997:
Origin of the word berserk according to Dictionary.com:
“A berserker is a member of a class of ancient Norse warriors who worked themselves into a frenzy before battle and fought with insane fury and courage.”
For example, this comprehensive post debunking Patel’s take by Gary Marcus who writes Marcus on AI. Marcus’s conclusion:
“The scandal is the inept in-house security at OpenAI. And the marketing. With gullible podcasters amplifying the PR.”
And if you want a technical description of what happened, stripped of the anthropomorphic jargon, go to the end of Marcus’s post.
Kryptogal (Kate, if you like)’s Note below.
“Possibly the most transparent attempt at liability and fault-shifting, and biggest pile of “it’s everyone else’s fault but ours if what we built (bc money) destroys everything” I’ve ever read. From the “head of strategic futures” at OpenAI.
She quotes from the Open AI post: ‘There is probably nothing we could have ever done to avoid this outcome under even the best of circumstances, and it was certainly impossible to avoid given the extremely low levels of strategic thought and situational awareness on AI from any governing class in the world.’
Oh, funny, suddenly the egoist masters of the universe who like to move fast and break things think the “governing class” is important and should be the ones actually in charge?? It’s going to be their fault for not regulating the people who claim to be the only experts, NOT the fault of the people actually doing the stuff that fucks everyone? He thinks it will one day be a “crime, or at least a sin” to release the shit he’s building, but not a crime to build it in the first place?
That’s not how it works, Dean.”
I don’t know whether my analogy to cars is apt. But I took a shot.
There’s already a wrongful death lawsuit moving through the court system against a company for an allegedly defective chatbot : Garcia V. Character Technologies, Inc.
My comments in this paragraph are based largely on the two sources discussed below.
This interview with economist and polymath Tyler Cowen is excellent. Cowen discusses why AI is unique in its speed of development. Cowen has a balanced view of the promises and worries of AI. He believes that AI will increase economic growth by between 0.5% and 1% per year, which its substantial over a decade or more.
Cowen also believes there will not be an aggregate fall in employment because while we can predict the job categories that will be lost to AI, it is impossible to predict the new job categories that will come into existence. So, we worry only about what we can predict.
For a less optimistic and more cautionary take, I found Bill Gates’ recent essay on AI valuable. Gates also discusses the unique speed of AI development vs. other technologies. However, Gates is far more worried than Tyler Cowen about mass unemployment and the resulting social and political unrest.
Both Gates and Cowen believe there is no stopping AI advancement. If you want to do a deeper dive, I recommend checking out both.
The reassuring part is that AI isn’t conscious. The less reassuring part is that plenty of people writing about it seem determined to give it a personality anyway…. Another great Saturday morning read David!
There has indeed been a bit of an AI frenzy. I agree the anthropomorphizing is not helpful. Though I also conceed that AI capabilities are achieving things we didn't think possible, with human-like abilities we didn't earlier think possible. So it's a narrowing wedge.
The solution of holding people accountable - who exactly, a coder? or security specialist who didn't think of a particular outcome? An executive who didn't understand it to begin with, and who can afford to lawyer up? The USA has a poor record in this area.
In China after the Sanlu melamine scandal, two suppliers were executed and the chairwoman received life imprisonment. The takeaway being that in spite of harsher outcomes than you'll ever see in the U.S., they did it anyway.