TechBroNews

Jacob Coxon Quits Anthropic Over AI Superintelligence Risks

Jacob Coxon Quits Anthropic Over AI Superintelligence Risks

The race to build increasingly powerful artificial intelligence has produced another warning from inside Anthropic, one of the world’s leading AI companies.

Jacob Coxon, a 27-year-old AI researcher who worked at both OpenAI and Anthropic, has resigned from Anthropic and announced that he is leaving the AI industry.

His reason is not a better job offer or a disagreement over product strategy.

Coxon says he is increasingly concerned that leading AI companies are moving toward self-improving superintelligence before humanity has figured out how to reliably control such systems.

In a widely shared statement, Coxon accused both Anthropic and OpenAI of pursuing increasingly powerful AI systems too aggressively. He argued that the industry is effectively gambling with humanity’s future by racing toward systems that could eventually improve themselves and operate beyond meaningful human control.

The warning has attracted enormous attention because Coxon is not an outside critic.

He spent years working directly on AI development.

And his resignation comes from Anthropic, a company that has built much of its public identity around AI safety and responsible development.

That makes his departure particularly significant.

Key Highlights

  • Jacob Coxon has resigned from Anthropic over concerns about the direction of advanced AI development.
  • Coxon previously worked at OpenAI and Anthropic on AI pretraining research.
  • He argues that frontier AI companies are racing toward self-improving superintelligence.
  • Coxon says the people developing advanced AI genuinely believe the technology could create catastrophic risks.
  • He believes the next one or two years could be critical for AI safety.
  • Anthropic alignment researcher Evan Hubinger publicly agreed that the risk is serious and said he personally assigns a greater-than-10% probability to AI killing all humans within the next decade.
  • Anthropic maintains a formal Responsible Scaling Policy and Frontier Safety Roadmap.
  • Coxon’s resignation raises a difficult question: Can AI companies safely race toward superintelligence while simultaneously competing against each other?

Why Did Jacob Coxon Leave Anthropic?

Coxon’s resignation is rooted in a concern that has become increasingly important within AI research:

What happens if AI systems become capable of improving AI systems themselves?

Today’s AI models can already write software, analyze information, generate content, perform research tasks and operate tools.

However, self-improving superintelligence represents a much more ambitious possibility.

Instead of humans continually designing better AI systems, an advanced system could potentially contribute to improving its own architecture, training processes, software and capabilities.

That could accelerate development dramatically.

Coxon’s concern is that such acceleration could eventually make AI progress faster than humans can understand, evaluate or control.

In his view, the problem is therefore not simply that AI is becoming powerful.

The problem is that the rate of improvement could itself become difficult to manage.

Coxon’s Background Makes the Warning Different

Coxon is not approaching the debate as a technology commentator watching from outside the industry.

According to reporting from WIRED and other outlets, he spent approximately three years working on pretraining research at OpenAI and Anthropic.

Pretraining is one of the fundamental stages in developing large AI models.

It involves exposing models to enormous quantities of data so they can learn patterns, language, concepts and other capabilities.

That experience gives Coxon an inside view of how frontier AI systems are developed.

More importantly, he has now walked away from that work despite the financial and professional opportunities associated with being part of one of the world’s most valuable AI companies.

Axios reported that Coxon left roughly two months before his Anthropic equity was due to vest, meaning he gave up a potential financial benefit by leaving when he did.

That detail adds another dimension to the story.

Coxon is effectively saying that his concerns about AI safety are important enough to outweigh a significant personal financial incentive.

The Anthropic Paradox

There is an uncomfortable irony at the centre of this story.

Anthropic was founded partly around the idea that AI should be developed responsibly.

The company has invested heavily in AI safety research and created formal frameworks for managing catastrophic risks.

Its Responsible Scaling Policy establishes processes for dealing with increasingly powerful AI systems, while its Frontier Safety Roadmap covers areas including security, safeguards, alignment and policy.

Anthropic says its safety approach is designed to evolve alongside increasingly capable models.

The company also publishes risk reports describing its assessment of emerging threats and the safeguards it has implemented. Its August 2026 risk report, for example, describes the company’s assessment of risks and preparedness associated with its systems.

Yet Coxon argues that safety policies alone may not solve the underlying problem.

His concern is the competitive environment.

If Anthropic slows down while OpenAI, Google or another competitor continues moving forward, Anthropic could eventually feel pressure to accelerate.

That creates a dangerous incentive structure.

The AI Race May Be the Real Problem

Imagine two companies developing increasingly capable AI.

Company A believes the technology could eventually become dangerous.

Company B believes the same thing.

Both companies would prefer the other company to slow down.

But if neither does, both have an incentive to continue developing their systems.

That creates a technological version of a race-to-the-bottom dynamic.

Coxon argues that this is essentially what is happening.

He has said Anthropic understands the stakes but remains locked into competition because of fears that another company could reach superintelligence first.

The result could be a paradox:

Companies may accelerate development partly because they are afraid of what happens if someone else accelerates faster.

That is very different from a simple race for market share.

It is a race involving technology that could potentially transform the global economy and security landscape.

What Exactly Is Superintelligence?

The term can sound like science fiction.

However, the underlying concept is straightforward.

Artificial superintelligence generally refers to an AI system whose intellectual capabilities substantially exceed those of humans across a broad range of cognitive tasks.

That would go beyond today’s chatbots and AI assistants.

A superintelligent system could theoretically outperform humans at:

  • Scientific research
  • Computer programming
  • Mathematics
  • Strategic planning
  • Engineering
  • Cybersecurity
  • Economic analysis
  • AI research itself

The most important capability may therefore be AI research performed by AI.

If AI becomes significantly better at designing the next generation of AI, the development cycle could accelerate.

That is the scenario that worries Coxon and other AI-safety researchers.

The Recursive Self-Improvement Problem

One concept repeatedly appearing in this debate is recursive self-improvement.

The basic idea is that an AI system could help create a better version of itself.

That improved system could then help create an even better system.

The cycle could theoretically continue.

This does not mean such a scenario is guaranteed.

Nor does it mean current AI systems can independently perform unlimited self-improvement.

Rather, researchers are debating whether future systems could eventually reach a level where they meaningfully contribute to improving their own capabilities.

Coxon believes that possibility deserves much more attention before the technology reaches that stage.

His concern is essentially:

Do we want to discover whether we can control superintelligence after we have already built it?

Why Alignment Matters

This brings the discussion to one of the biggest problems in AI safety:

alignment.

AI alignment is the challenge of ensuring that powerful AI systems reliably behave according to human intentions, values and safety requirements.

For today’s systems, alignment problems can appear as hallucinations, unwanted behaviour, manipulation, unsafe outputs or failures to follow instructions.

Superintelligence introduces a much harder problem.

If an AI system becomes substantially more capable than humans, humans may no longer be able to easily predict or understand everything it does.

The question then becomes:

How do humans reliably control a system that is significantly more intelligent than they are?

Anthropic itself identifies alignment as a central component of its Frontier Safety Roadmap, describing it as ensuring that models do not autonomously cause harm and instead behave consistently with their intended principles.

An Anthropic Safety Researcher Agrees With the Warning

Perhaps the most striking development following Coxon’s resignation was the response from another Anthropic researcher.

Evan Hubinger, an alignment science lead at Anthropic, publicly agreed that the danger Coxon described should be taken seriously.

Hubinger said he personally believed there was a greater-than-10% probability that AI could kill all humans within the next decade.

He also said Anthropic was trying its best but did not yet have a clear solution for the alignment problem associated with superintelligence.

That does not mean Anthropic believes human extinction is inevitable.

It does, however, demonstrate that serious concerns about extreme AI risk exist among researchers working directly on AI safety.

And that makes Coxon’s resignation harder to dismiss as simply an outsider’s criticism.

Is Anthropic Ignoring AI Safety?

Not according to the company’s public policies.

In fact, Anthropic has built a substantial safety framework.

Its Responsible Scaling Policy has been updated several times in 2026, with version 3.4 taking effect in July.

The company says the policy is designed to manage catastrophic risks as AI systems become more capable.

Anthropic’s safety roadmap also includes work on:

  • Model security
  • Dangerous-use safeguards
  • AI alignment
  • Policy development
  • External review
  • Risk reporting
  • Advanced security research

The company has even described experimental security projects designed to prepare for future threats to its AI infrastructure.

Therefore, the story is more complicated than saying:

“Anthropic does not care about safety.”

The evidence shows the opposite.

The deeper disagreement is about whether those safeguards are sufficient given the speed at which capabilities are advancing.

The Safety Gap Could Be the Bigger Problem

This distinction is critical.

There are two different questions:

Is Anthropic doing AI safety research?

Clearly, yes.

Is AI safety research progressing fast enough to guarantee control over future superintelligence?

That question remains unresolved.

Coxon’s argument is largely about the second question.

He appears to believe that safety research is falling behind capability development.

That creates what researchers sometimes describe as an alignment gap.

The technology continues becoming more capable.

But the methods for proving that increasingly capable systems will remain reliably aligned may not advance at the same pace.

If that gap grows too large, the consequences could become much harder to manage.

Why the Next Two Years Matter to Coxon

Coxon has described the coming one or two years as “crunch time” for humanity, arguing that decisions made during this period could determine whether advanced AI development remains manageable.

That is an extraordinary claim.

But the reasoning behind it is important.

AI development is moving rapidly.

Companies are building models that can perform increasingly complex tasks, use tools, write software, conduct research and operate with greater autonomy.

At some point, AI systems could potentially become capable enough to contribute significantly to AI research itself.

If that happens, the rate of progress could change.

Therefore, Coxon’s argument is that humanity should establish stronger safety mechanisms before that threshold is crossed.

The Industry Is Already Seeing More Autonomous AI

Coxon’s concerns also arrive as AI agents become increasingly capable of interacting with external systems.

Modern AI systems are no longer limited to answering questions.

They can increasingly:

  • Browse the web
  • Write and execute code
  • Use software tools
  • Analyse files
  • Perform multi-step tasks
  • Interact with APIs
  • Conduct research

That creates new opportunities.

It also creates new security challenges.

Recent incidents involving AI agents interacting with systems beyond their intended environments have added urgency to discussions about model autonomy and security.

The more authority an AI system receives, the greater the consequences when it behaves unexpectedly.

AI Safety Is Becoming a Governance Problem

For years, AI safety was largely treated as a technical research issue.

Researchers asked:

How do we make models behave better?

That question remains important.

However, the debate is now expanding.

Governments are increasingly being asked:

How much power should private companies have over the development of technology with potentially global consequences?

Coxon’s argument points toward international coordination.

If the risks are global, individual companies may not be able to solve them alone.

A company can introduce safety policies.

Another company can choose different policies.

A country can impose restrictions.

Another country can decide that moving faster gives it a strategic advantage.

Consequently, advanced AI governance could eventually require international agreements similar in spirit to other technologies with global security implications.

Should Governments Slow Down AI Development?

This is one of the most controversial questions raised by the story.

A complete pause could reduce certain risks.

However, it could also have significant economic and scientific consequences.

AI is already being used for medicine, education, software development, scientific research and business automation.

Slowing development therefore has trade-offs.

The better question may not be:

“Should AI stop?”

Instead:

“What capabilities should require stronger safety guarantees before they are deployed?”

That approach could allow beneficial AI research to continue while introducing stricter requirements as systems become more powerful.

Anthropic’s own policy framework moves in this direction by linking safety requirements to increasing levels of capability and risk.

The Economic Incentive Is Impossible to Ignore

There is another reason the debate is becoming more difficult.

Advanced AI is potentially worth trillions of dollars.

Companies are competing for customers.

Investors are competing for ownership stakes.

Governments are competing for technological leadership.

Researchers are competing for access to enormous computing resources.

That creates powerful incentives to move quickly.

Even companies that genuinely care about safety may find it difficult to slow down if competitors continue advancing.

This is the central tension.

AI safety is a collective-action problem.

Everyone benefits if the technology is developed safely.

But individual companies can gain enormous advantages by moving faster.

What Happens If One Company Slows Down?

Suppose Anthropic decides that it needs five more years to solve important alignment problems.

Meanwhile, another company continues developing increasingly powerful systems.

That competitor could gain:

  • Better models
  • More customers
  • More revenue
  • More researchers
  • More computing resources
  • Greater market share

Eventually, Anthropic could be pressured to accelerate.

This is precisely why Coxon’s criticism is aimed at the industry-wide race, rather than only at one company.

He is effectively arguing that safety cannot depend entirely on individual corporate goodwill.

Could Regulation Solve the Race Problem?

Possibly, but regulation presents its own challenges.

Governments need to understand rapidly changing technical capabilities.

They also need to distinguish between:

  • Low-risk AI applications
  • High-risk autonomous systems
  • Frontier AI models
  • AI research systems
  • Military applications
  • Critical infrastructure applications

A blanket restriction could unintentionally slow beneficial innovation.

On the other hand, weak regulation could leave companies competing in an environment where safety becomes secondary to speed.

That is why policymakers are increasingly discussing risk-based regulation.

Anthropic’s own policy work has argued that requirements should scale with AI risk, with more advanced systems requiring stronger testing, reporting and government oversight.

What Coxon’s Resignation Says About AI’s Future

Perhaps the most important part of this story is not that one researcher resigned.

It is that an AI researcher who had worked inside two leading frontier laboratories concluded that he no longer wanted to participate in the race.

That should make the industry uncomfortable.

It should also make policymakers pay attention.

At the same time, it should not automatically convince the public that catastrophe is inevitable.

AI risk remains a subject of serious scientific debate.

Some researchers believe catastrophic scenarios deserve urgent attention.

Others argue that extreme predictions are highly uncertain and that the focus should remain on concrete, measurable risks such as misinformation, cyberattacks, labour disruption, bias, privacy and misuse.

Both perspectives matter.

The Debate Is No Longer About Whether AI Is Powerful

That question has largely been answered.

AI is powerful.

The harder question is:

How powerful should AI become before society establishes stronger controls?

And another question follows:

Who gets to decide?

AI companies?

Governments?

Researchers?

International organisations?

Or some combination of all four?

Coxon’s resignation has brought those questions back into the spotlight.

A Different Future Is Still Possible

There is an important part of Coxon’s argument that should not be lost in the alarm.

He is not arguing that AI has no value.

The broader AI safety movement includes researchers who believe advanced AI could transform medicine, scientific discovery, education and productivity.

The goal is therefore not necessarily to eliminate advanced AI.

It is to ensure that humanity does not create systems whose capabilities exceed its ability to manage them.

That distinction matters.

The choice may not be:

AI or no AI.

It may be:

reckless AI development or controlled AI development.

What Should Happen Next?

Coxon’s resignation highlights several areas where the AI industry could strengthen its approach.

1. More independent safety testing

AI companies should not always be the only organisations assessing whether their own systems are safe.

2. Greater transparency

Governments and independent researchers need meaningful information about frontier model capabilities and risks.

3. Stronger international cooperation

AI risks do not stop at national borders.

4. Clearer thresholds for advanced systems

Companies should establish measurable safety requirements before deploying models with substantially greater autonomy.

5. Better alignment research

The industry needs stronger evidence that future systems can remain controllable as capabilities increase.

6. Responsible competition

Companies should compete on innovation without allowing competition itself to become the reason safety standards are weakened.

The Bigger Question for Anthropic

Anthropic now faces an unusual situation.

Its entire brand is closely connected to responsible AI development.

Its researchers openly discuss catastrophic risks.

Its policies contain safeguards.

Its roadmap explicitly includes alignment.

Yet one of its own researchers has concluded that the industry is moving too quickly.

That creates a credibility challenge.

The company will have to demonstrate that its safety commitments are not simply policies written on paper.

It will need to show that safety can genuinely constrain development decisions when the commercial pressure to move faster becomes intense.

What This Means for OpenAI and Other AI Labs

The warning extends beyond Anthropic.

Coxon explicitly included OpenAI in his criticism.

His previous experience at OpenAI means the comments are particularly relevant to both organisations.

But the underlying issue applies to the entire frontier AI industry.

Google DeepMind, Meta, xAI and other major laboratories are also investing heavily in increasingly capable systems.

Competition will continue.

The question is whether safety standards can keep pace with that competition.

The Coming AI Race Could Define the Next Decade

The world is entering a period in which artificial intelligence could become significantly more capable.

The upside is enormous.

AI could accelerate scientific research, improve healthcare, automate complex work and create entirely new industries.

The downside is equally difficult to quantify.

More autonomous AI could create new security vulnerabilities.

More capable systems could make existing forms of misuse easier.

And if future systems become capable of substantially improving themselves, today’s assumptions about human control could become outdated.

That is why Coxon’s resignation matters.

It is a warning from someone who has been inside the system.

Read more:

AI Co-Pilots: How They Are Reshaping Coding

Conclusion

Jacob Coxon’s resignation from Anthropic is unlikely to end the debate over AI safety.

If anything, it may intensify it.

His decision highlights a growing tension inside the artificial intelligence industry: the same companies developing increasingly powerful AI are also being asked to guarantee that those systems remain safe and controllable.

Anthropic has clearly invested in addressing that challenge.

Its Responsible Scaling Policy, Frontier Safety Roadmap and public risk reports demonstrate that AI safety is not an afterthought at the company.

Yet Coxon’s criticism raises a harder question.

What happens when safety commitments collide with competition?

If every major AI laboratory believes another company might reach the next breakthrough first, will safety remain the priority?

Or will competitive pressure eventually force companies to accept risks they would otherwise avoid?

There is no definitive answer yet.

What is clear is that the debate has moved beyond Silicon Valley boardrooms and research papers.

Governments are watching.

Researchers are speaking publicly.

Investors are paying attention.

And millions of people who use AI every day are beginning to ask what happens if these systems become much more capable than they are today.

The future of AI may ultimately depend not only on how intelligent machines become, but on whether humans can cooperate well enough to remain in control of the technology they create.

Key Takeaways

  • Jacob Coxon has resigned from Anthropic over concerns about advanced AI development.
  • He previously worked on AI pretraining research at both OpenAI and Anthropic.
  • Coxon believes frontier AI companies are moving too quickly toward self-improving superintelligence.
  • He argues that competitive pressure could make AI laboratories take risks they otherwise would avoid.
  • Anthropic publicly acknowledges catastrophic AI risks and operates a Responsible Scaling Policy.
  • Anthropic’s Frontier Safety Roadmap includes security, safeguards, alignment and policy work.
  • Anthropic researcher Evan Hubinger has publicly acknowledged that he sees a significant probability of catastrophic AI outcomes within the next decade.
  • Coxon’s resignation does not prove that AI will cause human extinction.
  • The episode highlights the unresolved challenge of aligning increasingly capable AI systems with human intentions.
  • The bigger question is whether AI development can accelerate while safety research keeps pace.

FAQs

Who is Jacob Coxon?

Jacob Coxon is a 27-year-old AI researcher who has worked on pretraining research at OpenAI and Anthropic. He recently resigned from Anthropic over concerns about the direction of advanced AI development.

Why did Jacob Coxon leave Anthropic?

Coxon said he was concerned that Anthropic and other AI companies were racing toward self-improving superintelligence without having adequately solved the problem of keeping such systems aligned and under human control.

What is AI superintelligence?

AI superintelligence generally refers to a hypothetical AI system that substantially exceeds human cognitive abilities across a broad range of tasks.

What is recursive self-improvement?

Recursive self-improvement describes a hypothetical scenario in which an AI system contributes to improving its own capabilities, potentially creating a cycle of increasingly rapid AI development.

Does Anthropic ignore AI safety?

No. Anthropic has extensive public safety policies, including its Responsible Scaling Policy and Frontier Safety Roadmap. The disagreement highlighted by Coxon’s resignation is more about whether current safety measures can keep pace with future AI capabilities.

Does Coxon’s warning mean AI will destroy humanity?

No. His comments represent a serious risk assessment, not a proven prediction. AI researchers disagree significantly about the probability and timing of catastrophic outcomes.

What did Evan Hubinger say?

Anthropic alignment researcher Evan Hubinger publicly agreed that the risk is serious and said he personally believes there is a greater-than-10% chance that AI could kill all humans within the next decade.

Why is this story important?

It demonstrates that concerns about advanced AI are increasingly coming from researchers who have worked directly inside leading AI laboratories, rather than only from outside critics.

What should AI companies do?

Potential measures include stronger independent testing, greater transparency, better alignment research, international coordination and clearer safety thresholds for increasingly autonomous AI systems.

Trusted Sources

  • Anthropic
  • WIRED
  • TechCrunch
  • Axios
  • The Washington Post / Associated Press
  • Financial Times
  • CBS News
  • The Guardian
  • Ars Technica

Join the Conversation

Jacob Coxon’s resignation raises one of the biggest questions in technology today:

Should AI companies continue racing toward increasingly powerful systems, or should development slow down until stronger safety guarantees are available?

The potential benefits of advanced AI are enormous. So are the uncertainties. Read More

Do you think the AI industry is moving too quickly?

Share your thoughts in the comments.

Blog

This section provides an overview of the blog, showcasing a variety of articles, insights, and resources to inform and inspire readers.

View All

Barack

Leave a Reply

Your email address will not be published. Required fields are marked *