The AI Slowdown Debate Crashed Salesforce's Party

Primary Journalistic Dispatch & Direct Reporting

Benioff roamed the audience while delivering his Dreamforce sermon, followed by a teleoperated camera on wheels that beamed his face to screens hanging from the cavernous room's ceiling. Within minutes, he ordered up a graph of Salesforce's annual revenue, which the company projects will top $46 billion in 2027, thanks in part to demand for its AI-powered products. The audience applauded. But hanging over the whole event was a dark question: Is artificial intelligence really worth risking all our lives for?

Between flashy software demonstrations of agents building dashboards, conference-goers watched Silicon Valley leaders debate whether the AI industry should keep barreling ahead or if the government needs to slow AI progress to prevent catastrophic risks stemming from the technology.

Last week, AI researcher Jacob Coxon resigned from Anthropic and publicly warned that companies racing toward self-improving AI systems were "gambling with our lives." By the weekend, other Anthropic employees were echoing his concerns, and CEO Dario Amodei was urging world leaders to help "pace the frontier" of AI development. Other AI CEOs then began endorsing the idea, including Amodei's greatest rival, OpenAI's Sam Altman.

In-Depth Developments & Factual Context

But not everyone is on board with the idea. David Sacks, a White House AI adviser, brushed off Amodei's suggestions and fears of AI doom as "just another bid for regulatory capture." President Trump went further this week, calling fears about existential AI risk " a hoax ." Despite the challenging political environment, Congress is trying to capitalize on the moment and push ahead with legislation to regulate the AI industry .

Onstage, 6-foot-5 Benioff towered over Anthropic's CEO, vigorously shaking Amodei's hand just before asking for his latest assessment of AI progress. While this is undoubtedly Benioff's show, Anthropic has become a critical partner helping usher Salesforce into the AI era.

"Let's say you're running a car company, and another car company, not yours, they have some kind of safety incident," Amodei said, making a not-so-subtle reference to OpenAI's Hugging Face incident . "Obviously, it's very tempting to attack your competitor," Amodei added, but the more responsible move is to "organize the rest of the industry and say, what can we do to set standards for everyone?"

Industry Impact & Strategic Analysis

But the Anthropic CEO was careful to note that pacing the frontier doesn't mean "freezing the technology in place," and of course, he took a moment to plug Claude's integration with Salesforce.

Moments after Amodei exited, Nvidia CEO Jensen Huang appeared and began downplaying the concerns Anthropic has raised. AI safety effectively amounts to an "engineering problem," Huang argued, and the companies building AI can simply police themselves.

"We don't need any new laws, we don't need any new regulation," Huang told Dreamforce attendees. "You run as fast as you can, but if you feel that at any given point in time, the company is out of control or the products are not going to be safe, you take a pause."

Forward Outlook & Market Perspective

The debate feels a bit heavy for Dreamforce, but then again, maybe this is the perfect venue for it. As tech executives pitched their various enterprise AI tools, I slipped out of the room, passing the plush mascots and vaguely Hawaiian decorations, to take a call from Sayash Kapoor, an incoming computer science professor at UC Berkeley.

Kapoor and Princeton professor Arvind Narayanan are coauthors of the book AI Snake Oil and write a Substack together called AI as Normal Technology . The pair are known for cutting through AI marketing hype while not dismissing the technology's potential. This week, they published one of their longest essays yet , which tries to make sense of "loss-of-control incidents," referring to events such as OpenAI accidentally letting its agents hack into Hugging Face.

"Part of what we were trying to do in this essay is to move past this false dichotomy to bridge the ground between AI safety and cybersecurity," says Kapoor. "If you think AI will soon become superintelligent and evade any controls we can put in front of them, of course, alignment seems more important. On the other hand, if you think AI companies have been negligent and avoided taking bog-standard cybersecurity steps, then it's natural to think that control is what we need to intervene on. I think both of these views have some element of truth in them."

In the wake of the Hugging Face saga, Kapoor says he was surprised to learn how few security precautions OpenAI had taken. AI companies need to reckon with their "lack of organizational maturity," he argues, and accept that loss-of-control incidents won't be solved by a single breakthrough in security or alignment. He adds that the incident convinced him that policymakers should make AI labs liable for the harms their technology causes.

"The consequences for OpenAI for the Hugging Face hack were basically close to zero. The company was able to largely proceed as is," says Kapoor. "If you imagine this level of accident in any other industry, you would have seen a monthslong internal investigation, people would have been fired or gone to jail; OpenAI would have had to pay millions of dollars in fines."

Kapoor doesn't wholly endorse Amodei's proposal to pace the frontier, but he agrees with parts of it. He thinks AI labs need better governance to break out of their "move fast and break things" attitude and that they should let independent evaluators assess their AI safety and security practices. He and Narayanan write that more alignment work is certainly needed but that the real crisis today is more narrowly about cybersecurity.

Reporting synthesized and verified under Nexvoro.tech editorial guidelines. Full primary records referenced via Wired.