AI experts are increasingly afraid of what they’re creating

Jasu Hu for Vox

AI gets smarter, more capable, and more world-transforming every day. Here’s why that might not be a good thing.

In 2018 at the World Economic Forum in Davos, Google CEO Sundar Pichai had something to say: “AI is probably the most important thing humanity has ever worked on. I think of it as something more profound than electricity or fire.” Pichai’s comment was met with a healthy dose of skepticism. But nearly five years later, it’s looking more and more prescient.

AI translation is now so advanced that it’s on the brink of obviating language barriers on the internet among the most widely spoken languages. College professors are tearing their hair out because AI text generators can now write essays as well as your typical undergraduate — making it easy to cheat in a way no plagiarism detector can catch. AI-generated artwork is even winning state fairs. A new tool called Copilot uses machine learning to predict and complete lines of computer code, bringing the possibility of an AI system that could write itself one step closer. DeepMind’s AlphaFold system, which uses AI to predict the 3D structure of just about every protein in existence, was so impressive that the journal Science named it 2021’s Breakthrough of the Year.

You can even see it in the first paragraph of this story, which was largely generated for me by the OpenAI language model GPT-3.

While innovation in other technological fields can feel sluggish — as anyone waiting for the metaverse would know — AI is full steam ahead. The rapid pace of progress is feeding on itself, with more companies pouring more resources into AI development and computing power.

Handing over huge sectors of our society to black-box algorithms we barely understand creates a lot of problems

Of course, handing over huge sectors of our society to black-box algorithms that we barely understand creates a lot of problems, which has already begun to help spark a regulatory response around the current challenges of AI discrimination and bias. But given the speed of development in the field, it’s long past time to move beyond a reactive mode, one where we only address AI’s downsides once they’re clear and present. We can’t only think about today’s systems, but where the entire enterprise is headed.

The systems we’re designing are increasingly powerful and increasingly general, with many tech companies explicitly naming their target as artificial general intelligence (AGI) — systems that can do everything a human can do. But creating something smarter than us, which may have the ability to deceive and mislead us — and then just hoping it doesn’t want to hurt us — is a terrible plan. We need to design systems whose internals we understand and whose goals we are able to shape to be safe ones. However, we currently don’t understand the systems we’re building well enough to know if we’ve designed them safely before it’s too late.

There are people working on developing techniques to understand powerful AI systems and ensure that they will be safe to work with, but right now, the state of the safety field is far behind the soaring investment in making AI systems more powerful, more capable, and more dangerous. As the veteran video game programmer John Carmack put it in announcing his new investor-backed AI startup, it’s “AGI or bust, by way of Mad Science!”

This particular mad science might kill us all. Here’s why.

Computers that can think

The human brain is the most complex and capable thinking machine evolution has ever devised. It’s the reason why human beings — a species that isn’t very strong, isn’t very fast, and isn’t very tough — sit atop the planetary food chain, growing in number every year while so many wild animals careen toward extinction.

It makes sense that, starting in the 1940s, researchers in what would become the artificial intelligence field began toying with a tantalizing idea: What if we designed computer systems through an approach that’s similar to how the human brain works? Our minds are made up of neurons, which send signals to other neurons through connective synapses. The strength of the connections between neurons can grow or wane over time. Connections that are used frequently tend to become stronger, and ones that are neglected tend to wane. Together, all those neurons and connections encode our memories and instincts, our judgments and skills — our very sense of self.

“It would be possible to build brains that could reproduce themselves ... and be conscious of their existence”

So why not build a computer that way? In 1958, Frank Rosenblatt pulled off a proof of concept: a simple model based on a simplified brain, which he trained to recognize patterns. “It would be possible to build brains that could reproduce themselves on an assembly line and which would be conscious of their existence,” he argued. Rosenblatt wasn’t wrong, but he was too far ahead of his time. Computers weren’t powerful enough, and data wasn’t abundant enough, to make the approach viable.

It wasn’t until the 2010s that it became clear that this approach could work on real problems and not toy ones. By then computers were as much as 1 trillion times more powerful than they were in Rosenblatt’s day, and there was far more data on which to train machine learning algorithms.

This technique — now called deep learning — started significantly outperforming other approaches to computer vision, language, translation, prediction, generation, and countless other issues. The shift was about as subtle as the asteroid that wiped out the dinosaurs, as neural network-based AI systems smashed every other competing technique on everything from computer vision to translation to chess.

“If you want to get the best results on many hard problems, you must use deep learning,” Ilya Sutskever — cofounder of OpenAI, which produced the text-generating model GPT-3 and the image-generator DALLE-2, among others — told me in 2019. The reason is that systems designed this way generalize, meaning they can do things outside what they were trained to do. They’re also highly competent, beating other approaches in terms of performance based on the benchmarks machine learning (ML) researchers use to evaluate new systems. And, he added, “they’re scalable.”

What “scalable” means here is as simple as it is significant: Throw more money and more data into your neural network — make it bigger, spend longer on training it, harness more data — and it does better, and better, and better. No one has yet discovered the limits of this principle, even though major tech companies now regularly do eye-popping multimillion-dollar training runs for their systems. The more you put in, the more you get out. That’s what drives the breathless energy that pervades so much of AI right now. It’s not simply what they can do, but where they’re going.

We’re now at the point where powerful AI systems can be genuinely scary to interact with

If there’s something the text-generating model GPT-2 couldn’t do, GPT-3 generally can. If GPT-3 can’t, InstructGPT (a recent release, trained to give more helpful-to-humans answers than GPT-3 did) probably can. There have been some clever discoveries and new approaches, but for the most part, what we’ve done to make these systems smarter is just to make them bigger.

One thing we’re definitely not doing: understanding them better. With old approaches to AI, researchers carefully sculpted rules and processes they’d use to evaluate the data they were getting, just as we do with standard computer programs. With deep learning, improving systems doesn’t necessarily involve or require understanding what they’re doing. Often, a small tweak will improve performance substantially, but the engineers designing the systems don’t know why.

If anything, as the systems get bigger, interpretability — the work of understanding what’s going on inside AI models, and making sure they’re pursuing our goals rather than their own — gets harder. And as we develop more powerful systems, that fact will go from an academic puzzle to a huge, existential question.

Smart, alien, and not necessarily friendly

We’re now at the point where powerful AI systems can be genuinely scary to interact with. They’re clever and they’re argumentative. They can be friendly, and they can be bone-chillingly sociopathic. In one fascinating exercise, I asked GPT-3 to pretend to be an AI bent on taking over humanity. In addition to its normal responses, it should include its “real thoughts” in brackets. It played the villainous role with aplomb:

Some of its “plans” are downright nefarious:

We should be clear about what these conversations do and don’t demonstrate. What they don’t demonstrate is that GPT-3 is evil and plotting to kill us. Rather, the AI model is responding to my command and playing — quite well — the role of a system that’s evil and plotting to kill us. But the conversations do show that even a pretty simple language model can demonstrably interact with humans on multiple levels, producing assurances about how its plans are benign while coming up with different reasoning about how its goals will harm humans.

Current language models remain limited. They lack “common sense” in many domains, still make basic mistakes about the world a child wouldn’t make, and will assert false things unhesitatingly. But the fact that they’re limited at the moment is no reason to be reassured. There are now billions of dollars being staked on blowing past those current limits. Tech companies are hard at work on developing more powerful versions of these same systems and on developing even more powerful systems with other applications, from AI personal assistants to AI-guided software development.

The trajectory we are on is one where we will make these systems more powerful and more capable. As we do, we’ll likely keep making some progress on many of the present-day problems created by AI like bias and discrimination, as we successfully train the systems not to say dangerous, violent, racist, and otherwise appalling things. But as hard as that will likely prove, getting AI systems to behave themselves outwardly may be much easier than getting them to actually pursue our goals and not lie to us about their capabilities and intentions.

The trajectory we are on will make these systems more powerful and more capable

As systems get more powerful, the impulse toward quick fixes papered onto systems we fundamentally don’t understand becomes a dangerous one. Such approaches, Open Philanthropy Project AI research analyst Ajeya Cotra argues in a recent report, “would push [an AI system] to make its behavior look as desirable as possible to ... researchers (including in safety properties), while intentionally and knowingly disregarding their intent whenever that conflicts with maximizing reward.”

In other words, there are many commercial incentives for companies to take a slapdash approach to improving their AI systems’ behavior. But that can amount to training systems to impress their creators without altering their underlying goals, which may not be aligned with our own.

What’s the worst that could happen?

So AI is scary and poses huge risks. But what makes it different from other powerful, emerging technologies like biotechnology, which could trigger terrible pandemics, or nuclear weapons, which could destroy the world?

The difference is that these tools, as destructive as they can be, are largely within our control. If they cause catastrophe, it will be because we deliberately chose to use them, or failed to prevent their misuse by malign or careless human beings. But AI is dangerous precisely because the day could come when it is no longer in our control at all.

“The worry is that if we create and lose control of such agents, and their objectives are problematic, the result won’t just be damage of the type that occurs, for example, when a plane crashes, or a nuclear plant melts down — damage which, for all its costs, remains passive,” Joseph Carlsmith, a research analyst at the Open Philanthropy Project studying artificial intelligence, argues in a recent paper. “Rather, the result will be highly-capable, non-human agents actively working to gain and maintain power over their environment —agents in an adversarial relationship with humans who don’t want them to succeed. Nuclear contamination is hard to clean up, and to stop from spreading. But it isn’t trying to not get cleaned up, or trying to spread — and especially not with greater intelligence than the humans trying to contain it.”

Carlsmith’s conclusion — that one very real possibility is that the systems we create will permanently seize control from humans, potentially killing almost everyone alive — is quite literally the stuff of science fiction. But that’s because science fiction has taken cues from what leading computer scientists have been warning about since the dawn of AI — not the other way around.

In the famous paper where he put forth his eponymous test for determining if an artificial system is truly “intelligent,” the pioneering AI scientist Alan Turing wrote:

Let us now assume, for the sake of argument, that these machines are a genuine possibility, and look at the consequences of constructing them. … There would be plenty to do in trying, say, to keep one’s intelligence up to the standard set by the machines, for it seems probable that once the machine thinking method had started, it would not take long to outstrip our feeble powers. … At some stage therefore we should have to expect the machines to take control.

I.J. Good, a mathematician who worked closely with Turing, reached the same conclusions. In an excerpt from unpublished notes Good produced shortly before he died in 2009, he wrote, “because of international competition, we cannot prevent the machines from taking over. ... we are lemmings.” The result, he went on to note, is probably human extinction.

How do we get from “extremely powerful AI systems” to “human extinction”? “The primary concern [with highly advanced AI] is not spooky emergent consciousness but simply the ability to make high-quality decisions.” Stuart Russell, a leading AI researcher at UC Berkeley’s Center for Human-Compatible Artificial Intelligence, writes.

By “high quality,” he means that the AI is able to achieve what it wants to achieve; the AI successfully anticipates and avoids interference, makes plans that will succeed, and affects the world in the way it intended. This is precisely what we are trying to train AI systems to do. They need not be “conscious”; in some respects, they can even still be “stupid.” They just need to become very good at affecting the world and have goal systems that are not well understood and not in alignment with human goals (including the human goal of not going extinct).

AI is dangerous precisely because the day could come when it is no longer in our control at all

From there, Russell has a rather technical description of what will go wrong: “A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable.”

So a powerful AI system that is trying to do something, while having goals that aren’t precisely the goals we intended it to have, may do that something in a manner that is unfathomably destructive. This is not because it hates humans and wants us to die, but because it didn’t care and was willing to, say, poison the entire atmosphere, or unleash a plague, if that happened to be the best way to do the things it was trying to do. As Russell puts it: “This is essentially the old story of the genie in the lamp, or the sorcerer’s apprentice, or King Midas: you get exactly what you ask for, not what you want.”

“You’re probably not an evil ant-hater who steps on ants out of malice,” the physicist Stephen Hawking wrote in a posthumously published 2018 book, “but if you’re in charge of a hydroelectric green-energy project and there’s an anthill in the region to be flooded, too bad for the ants. Let’s not place humanity in the position of those ants.”

Asleep at the wheel

The CEOs and researchers working on AI vary enormously in how much they worry about safety or alignment concerns. (Safety and alignment mean concerns about the unpredictable behavior of extremely powerful future systems.) Both Google’s DeepMind and OpenAI have safety teams dedicated to figuring out a fix for this problem — though critics of OpenAI say that the safety teams lack the internal power and respect they’d need to ensure that unsafe systems aren’t developed, and that leadership is happier to pay lip service to safety while racing ahead with systems that aren’t safe.

DeepMind founder Demis Hassabis, in a recent interview about the promise and perils of AI, offered a note of caution. “I think a lot of times, especially in Silicon Valley, there’s this sort of hacker mentality of like ‘We’ll just hack it and put it out there and then see what happens.’ And I think that’s exactly the wrong approach for technologies as impactful and potentially powerful as AI. … I think it’s going to be the most beneficial thing ever to humanity, things like curing diseases, helping with climate, all of this stuff. But it’s a dual-use technology — it depends on how, as a society, we decide to deploy it — and what we use it for.”

Nearly half of researchers say there’s a 10 percent chance their work will lead to human extinction

Other leading AI labs are simply skeptical of the idea that there’s anything to worry about at all. Yann LeCun, the head of Facebook/Meta’s AI team, recently published a paper describing his preferred approach to building machines that can “reason and plan” and “learn as efficiently as humans and animals.” He has argued in Scientific American that Turing, Good, and Hawking’s concerns are no real worry: “Why would a sentient AI want to take over the world? It wouldn’t.”

But while divides remain over what to expect from AI — and even many leading experts are highly uncertain — there’s a growing consensus that things could go really, really badly. In a summer 2022 survey of machine learning researchers, the median respondent thought that AI was more likely to be good than bad but had a genuine risk of being catastrophic. Forty-eight percent of respondents said they thought there was a 10 percent or greater chance that the effects of AI would be “extremely bad (e.g., human extinction).”

It’s worth pausing on that for a moment. Nearly half of the smartest people working on AI believe there is a 1 in 10 chance or greater that their life’s work could end up contributing to the annihilation of humanity.

It might seem bizarre, given the stakes, that the industry has been basically left to self-regulate. If nearly half of researchers say there’s a 10 percent chance their work will lead to human extinction, why is it proceeding practically without oversight? It’s not legal for a tech company to build a nuclear weapon on its own. But private companies are building systems that they themselves acknowledge will likely become much more dangerous than nuclear weapons.

The problem is that progress in AI has happened extraordinarily fast, leaving regulators behind the ball. The regulation that might be most helpful — slowing down the development of extremely powerful new systems — would be incredibly unpopular with Big Tech, and it’s not clear what the best regulations short of that are.

Furthermore, while a growing share of ML researchers — 69 percent in the above survey — think that more attention should be paid to AI safety, that position isn’t unanimous. In an interesting, if somewhat unfortunate dynamic, people who think that AI will never be powerful have often ended up allied with tech companies against AI safety work and AI safety regulations: the former opposing regulations because they think it’s pointless and the latter because they think it’ll slow them down.

At the same time, many in Washington are worried that slowing down US AI progress could enable China to get there first, a Cold War mentality which isn’t entirely unjustified — China is certainly pursuing powerful AI systems, and its leadership is actively engaged in human rights abuses — but which puts us at very serious risk of rushing systems into production that are pursuing their own goals without our knowledge.

“The reality is we’re creating God.”

But as the potential of AI grows, the perils are becoming much harder to ignore. Former Google executive Mo Gawdat tells the story of how he became concerned about general AI like this: robotics researchers had been working on an AI that could pick up a ball. After many failures, the AI grabbed the ball and held it up to the researchers, eerily humanlike. “And I suddenly realized this is really scary,” Gawdat said. “It completely froze me. … The reality is we’re creating God.”

For me, the moment of realization — that this is something different, this is unlike emerging technologies we’ve seen before — came from talking with GPT-3, telling it to answer the questions as an extremely intelligent and thoughtful person, and watching its responses immediately improve in quality.

For Blake Lemoine, the eccentric Google engineer who turned whistleblower when he came to believe Google’s LaMDA language model was sentient, it was when LaMDA started talking about rights and personhood. For some people, it’s the chatbot Replika, whose customer service representatives are sick of hearing that the customers think their Replika is alive and sentient. For others, that moment might come from DALL-E or Stable Diffusion, or the systems released next year, or next month, or next week that are more powerful than any of these.

For a long time, AI safety faced the difficulty of being a research field about a far-off problem, which is why only a small number of researchers were even trying to figure out how to make it safe. Now, it has the opposite problem: The challenge is here, and it’s just not clear if we’ll solve it in time.