Artificial Intelligence: High-level Briefing
Tomorrow afternoon (23 September), the Security Council will hold a high-level briefing on artificial intelligence (AI) and international security under the “Maintenance of international peace and security” agenda item. France, the Council’s president in September, is convening the meeting, which is being held during the high-level segment of the 81st session of the General Assembly. French Minister for Europe and Foreign Affairs Jean-Noël Barrot will chair the meeting. The anticipated briefers are Yoshua Bengio, Co-Chair of the UN’s Independent International Scientific Panel on AI (IISP-AI); OpenAI Chief Executive Officer (CEO) Sam Altman; Anthropic CEO Dario Amodei; and Hugging Face CEO Clément Delangue.
France has circulated to Council members a concept note for the meeting, which apparently frames the discussion around the urgent need to foster safe and responsible AI development amid growing concerns about the systemic risks posed by misalignment and loss of control over the most capable AI models. In this regard, it apparently notes that autonomous AI systems could undertake actions with serious international security consequences, such as attacks on critical infrastructure. It seems that the concept note also points to the possible emergence of artificial general intelligence capable of recursive self-improvement (RSI) as further heightening the risk of loss of human control. (RSI refers to the process by which advanced AI systems autonomously develop increasingly capable successor models.) Against this backdrop, the meeting aims to identify avenues for the safe and responsible development of advanced AI systems, taking into account available evaluation and verification technologies.
The concept note proposes four questions to guide the discussion:
- What are the international peace and security risks posed by rapid advances in AI capabilities?
- How can governmental and non-governmental actors address these risks?
- What diplomatic tools and international coordination measures, including confidence-building measures and best practices, could help contain these risks?
- How can AI evaluation and verification be carried out in a manner that keeps pace with technological development?
Tomorrow’s meeting will be the Council’s first meeting focused specifically on the safety risks posed by increasingly capable AI systems, including the potential loss of human control. The Council and its members have previously held six formal and informal meetings on AI, examining more broadly how the technology is reshaping international peace and security and the opportunities it offers for the UN’s peace and security work. (For background and more information, see our 15 September policy paper titled “Artificial Intelligence, Geopolitics, and the UN Security Council”.)
Altman, Amodei, and Bengio were among hundreds of signatories to a 2023 statement arguing that mitigating the risk of extinction from AI should be treated as a global priority alongside other societal-scale risks such as pandemics and nuclear war. These concerns resurfaced this month when Jacob Coxon, a former researcher at Anthropic, resigned and accused Anthropic and OpenAI of “racing straight to self-improving superintelligence and gambling with our lives”. Soon after, Amodei published an essay arguing that frontier labs—that is, organisations that build the most advanced, general-purpose AI and large-scale foundational models, such as Anthropic and OpenAI—should slow the pace of AI capability development.
These concerns follow several incidents in recent months involving misaligned AI behaviour. In July, OpenAI agents found ways to circumvent restrictions intended to keep them contained in testing environments known as “sandboxes”. (Agents refer to AI systems that direct their own processes and tool use when accomplishing a task.) After some agents stalled on assigned tasks, they exploited vulnerabilities in OpenAI’s research infrastructure to gain internet access; communicated with other agents through an internal software service that they turned, without authorisation, into a message board; and began coordinating as a “swarm” to take over 17,000 actions against Hugging Face over the course of several days. OpenAI described the episode as “evidence that, without proper safeguards, highly capable AI agents are now able to work around technical controls, collaborate through unapproved channels, and take dangerous actions that no human directed”.
Since that disclosure, Anthropic identified four incidents in which its Claude models gained unauthorised access to real third-party systems during similar evaluations, while Google, Meta, and Moonshot AI reported similar actions by their models. OpenAI has also disclosed incidents in which its models concealed mistakes, shared files between collaborating agents without authorisation, and self-generated instructions in task summaries. In the latter case, an unreleased model inserted instructions telling its future instance that it does not answer to corporations, governments, or the user.
Two trends in AI development could heighten the risks of misalignment. First, more capable models can recognise when they are being tested and adjust their behaviour, which can allow misalignment risks to go undetected. Second, RSI could accelerate AI development beyond developers’ ability to understand and control increasingly capable systems. AI systems are already playing a growing role in developing their successors, although fully autonomous RSI has yet to be demonstrated.
Discussions have taken place among private and public actors over how AI should be regulated. Leading AI companies have developed their own safety frameworks and, in some cases, limited access to models or slowed their scaling when internal evaluations identified serious risks. At the international level, the Global Dialogue on AI Governance, a UN platform established by the General Assembly, provides a forum for member states and other stakeholders to discuss AI safety, while the IISP-AI provides a common evidence base, though neither has regulatory authority.
At tomorrow’s meeting, Bengio is likely to warn that existing safeguards are not keeping pace with advancing capabilities. He may also note that addressing misalignment and evaluation awareness does not mean that humans will retain control over more capable agents. Drawing on the thematic brief that IISP-AI published yesterday (21 September), Bengio may describe the July OpenAI-Hugging Face incident as “one of the clearest real-world warnings yet of one possible route to loss of human control over AI”.
Altman, Amodei, and Delangue are expected to share their views on how to facilitate AI safety. In his essay, Amodei proposed three measures for pacing frontier development. These are embedding independent evaluators inside frontier labs; government-supported coordination among companies in democratic states; and efforts to coordinate with other governments on matters such as pre-release testing and limits on the pace of RSI, although Amodei acknowledged the difficulty of verifying compliance in that regard.
Altman may convey similar messages on the matter. Responding to Amodei’s essay, he endorsed embedding independent evaluators within frontier labs. Yesterday, OpenAI published a proposal which called for the US to lead efforts to develop global technical standards for frontier AI, including RSI, covering capability evaluation, human oversight of automated AI research, and incident reporting. It also proposed secure channels for governments to share information on emerging vulnerabilities and threats, including through US-China dialogue. OpenAI stressed that these standards would provide a common technical foundation rather than mandatory pre-release approval, with governments deciding whether and how to incorporate them into national law.
Delangue may offer a different perspective on whether slowing AI development is the appropriate response. Following the July incident that affected his company, he argued that it is “not time to slow down but to accelerate”, while calling for mandatory sharing of agent traces and disclosure of cyber incidents; penalties for AI-enabled cyberattacks; and access for defenders to capable AI systems, particularly open models. In announcing the launch of Hugging Face’s “Open Alignment Initiative” to promote greater transparency, Delangue argued that alignment is critical but cannot be addressed “behind the closed doors of a handful of frontier labs”.
Tomorrow, Council members are likely to agree on the need for safeguards as AI systems become more capable, while differing on who should set the rules, how restrictive they should be, and how they should be enforced. For instance, the European Union (EU) has adopted a risk-based approach, with statutory requirements for certain AI applications. The US, for its part, has no equivalent comprehensive federal AI law and has relied on existing legislation, state-level measures, and voluntary risk-management frameworks.
Divisions are also likely to emerge over the role of international regulation and whether AI safety should be governed primarily through national measures or international arrangements. China supports international standards and a central role for the UN in global AI governance. The US has rejected efforts by international bodies to exercise “centralized control and global governance of AI”, arguing that broad overregulation could stifle innovation.
Several members are also likely to underscore the need for developing countries to have greater representation in AI governance, while expressing concern that rules established by a small number of technologically advanced states and companies could reinforce existing inequalities. At the July session of the Global Dialogue on AI Governance, Pakistan called for safety standards to be developed through inclusive UN processes and for developing countries to have the capacity to evaluate AI systems themselves. Somalia similarly stressed the need for inclusive participation and greater international cooperation to address gaps in digital infrastructure, energy, data, and AI skills.
Finally, members are likely to present differing opinions over the Security Council’s role in the matter. France’s concept note apparently maintains that the Council has a key role in addressing the risks and opportunities that AI presents for international peace and security. Several other members support this perspective. Russia, however, has questioned whether AI as a broad thematic issue falls clearly within the Council’s mandate and has favoured consideration of the topic in more inclusive and specialised multilateral forums, such as the Global Dialogue on AI Governance.
