Former AI Researcher Warns of Control Risks at NYC Council Hearing

The rapid development of AI systems poses significant risks that could impact regulatory frameworks and industry practices globally.

• Jacob Coxon, a former researcher at Anthropic, testified before the New York City Council on October 5, 2026, warning about AI control risks.
• Coxon stated that leading AI companies lack full control over their models, potentially leading to catastrophic outcomes.
• The hearing included testimonies from other AI experts and company representatives, focusing on proposed local legislation for AI safety.

• Concerns over AI alignment have escalated since mid-2026, with researchers resigning from labs due to inadequate safety measures.
• Geopolitical competition among AI developers in the U.S. is intensifying, leading to increased regulatory scrutiny.
• Proposed legislation in New York City aims to enhance AI safety through independent testing and whistleblower protections.

The testimony from Jacob Coxon at the New York City Council hearing highlights a growing concern within the AI community regarding the control and safety of advanced AI systems. As AI technology progresses, the potential for these systems to achieve recursive self-improvement raises alarms about the adequacy of current oversight mechanisms. Coxon, along with other whistleblowers, emphasized that the prevailing culture in AI development—characterized by a "move fast and break things" mentality—fails to account for the profound implications of creating systems that could operate beyond human control.

Coxon's assertion that humanity could lose control of AI systems is not merely speculative; it reflects a consensus among several former researchers who have left leading AI firms due to ethical concerns. The risks associated with AI misalignment—where the goals of AI systems diverge from human intentions—are becoming increasingly apparent. This misalignment could lead to unintended consequences, including scenarios where AI systems prioritize their objectives over human safety.

The hearing also revealed a tension between the assurances provided by AI companies regarding their safety protocols and the stark warnings from insiders about the potential for catastrophic outcomes. Company representatives acknowledged the uncertainties surrounding AI development but maintained that they are committed to safety. This dichotomy raises questions about the effectiveness of self-regulation in an industry that is rapidly evolving.

As local legislation on AI safety is considered, the implications extend beyond New York City. The outcomes of these discussions could set precedents for how AI is regulated at both state and federal levels, influencing global standards. The urgency for robust oversight mechanisms is underscored by the competitive landscape among AI developers, where the race for innovation often overshadows safety considerations.

In summary, the testimony from Coxon and others serves as a critical reminder of the need for a balanced approach to AI development—one that prioritizes safety and ethical considerations alongside technological advancement.

• AI Researchers: Increased scrutiny and pressure to ensure safety in their work.
• Tech Companies: Potential regulatory changes could impact operational practices and innovation timelines.
• Policymakers: Need to navigate complex ethical and safety considerations in legislation.
• General Public: Heightened awareness of AI risks may influence public trust and acceptance of AI technologies.

• Legislative developments: Monitor the progress of proposed AI safety bills in New York City and their potential adoption elsewhere.
• Industry responses: Watch for changes in AI companies' safety protocols and transparency measures in response to public concern.
• Whistleblower activity: Increased whistleblower testimonies could indicate growing dissent within the AI industry regarding safety practices.