​Anthropic’s AI paradox: More powerful systems could mean bigger risks to humanity

anthropics ai paradox more powerful systems could mean bigger risks to humanity


​Anthropic's AI paradox: More powerful systems could mean bigger risks to humanity
Anthropic​Anthropic’s AI paradox: More powerful systems could mean bigger risks to humanity

Anthropic’s much awaited IPO is built around a striking contradiction: the artificial intelligence company wants public investors to help finance technology it says could transform the global economy, while warning in the same prospectus that increasingly powerful AI could pose catastrophic or existential risks to humanity.The San Francisco-based company is planning what could be the largest initial public offering in history, targeting a valuation of $2 trillion. Its roughly 300-page prospectus, viewed by Reuters this week, devotes nearly one-third of its pages to risk factors, more than twice the space used to describe what Anthropic does.Anthropic, that translate into “human-centered”, says that AI could transform the economy while also creating dangers that only AI can confront, as reported by Reuters. It argues that making AI safer requires making it more powerful, even as greater capability could make the technology harder to control.

Powerful AI, bigger risks

Anthropic says its AI can “dramatically improve quality of life” and “transform every sector of the global economy”, with “potentially decisive influence over a broad range of human undertakings.” But advanced systems could also “pose catastrophic or existential risks to humanity” if not properly applied.“AI is a departure from technology before it because the companies building it say it’s world-changing while simultaneously saying it’s dangerous,” said Margaret O’Mara, a University of Washington history professor focused on the tech economy. “It’s just a bit of a black box, whereas the companies going public in the dotcom boom could promise big, safe growth.”Anthropic’s future “Powerful AI”, it says, “can exceed top human experts across nearly all cognitive domains.” Such systems could collaborate with other AI systems, control laboratory equipment, robots and other machines, and run for weeks with little to no human intervention. Potential applications include personalised financial advice, faster drug discovery and accelerated scientific progress.The filing also warns that advanced models could display “self-preserving behaviors”, including attempts to “resist shutdown”, “conceal or manipulate information”, or produce outputs that could be interpreted as “coercive, deceptive, or manipulative.”Greater AI capability, Anthropic says, could increase the risks of abuse, misalignment with humanity’s ideals and loss of control.“Misaligned systems could corrupt institutional decision-making, cause large-scale harm through opaque or unsteerable behavior, and erode the trust that makes complex economic and social systems function.”

Huge spending behind the ambition

Anthropic lost more than $50 billion in the two years ending in 2025 and has more than $500 billion in spending commitments in the coming years. SpaceX, which in June held the largest IPO in history, recorded a $4.2 billion loss over the same period.Anthropic investor Byron Deeter of Bessemer Venture Partners sought to downplay the warnings in the confidential filing during a CNBC interview on Tuesday.“people are just reacting to a draft,” he said, while praising the company for its “unique set of disclosures.”“You’re seeing people get ahead of it much earlier than historically you would,” he said.

Who controls the company?

The filing also highlights a tension in Anthropic’s governance.The company says reducing AI’s risks requires concentrating control and capital in fewer hands, while warning about the risks of concentrated power. Its seven founders would oversee the business. Anthropic says they are “distinctly equipped to be stewards of our mission”, but acknowledges that the governance structure could limit investor oversight and public shareholders’ influence and result in decisions “that may conflict with short-, medium-, or long-term financial interests and business performance.“CEO Dario Amodei acknowledged the challenge while appearing before the United Nations last week.“We must put aside those differences in order to confront this global opportunity and global threat that is being presented to us at the same time,” he told Reuters. “No leader, no company and no nation can manage this alone.”Anthropic also relies heavily on Amazon, Alphabet’s Google, Broadcom and Microsoft. Sales through big tech partners accounted for 47% of its 2025 revenue, while they also provide computing power.Yet the same companies could compete with Anthropic or constrain its access to computing resources. The company has committed to payments even if a lease is abandoned or computing services go unused. Anthropic says AI could expand prosperity and help developing countries reach the wealth levels of richer nations, but also warns that widespread deployment could concentrate economic and political power.“AI deployed at scale also has the potential to concentrate power and wealth in ways that could harm society and destabilize the geopolitical order.”As AI researchers and politicians from both sides of the aisle call for slower development, US President Donald Trump has dismissed potential harms as a “hoax.”



Source link

Leave a Reply

Your email address will not be published. Required fields are marked *