The Nuclear Weapons-AI Analogy Debate and the Expanding Discourse on AI Security Governance

Executive Summary

The debate sparked by Anthropic CEO Dario Amodei’s comparison of frontier AI models to weaponizable nuclear material has intensified the discourse on AI security governance, coinciding with discussions at the 81st UN General Assembly and Security Council. While the UN Secretariat, through Secretary-General Guterres, has officially adopted the nuclear-AI analogy as policy language, working-level officials like Amandeep Singh Gill and U.S. policy circles, including *Foreign Affairs*, counter that applying the logic of the nuclear non-proliferation regime to AI, which lacks physical verifiability, carries a high risk of misleading policy design. This analogy is intertwined with the incentive structure of frontier AI companies seeking to secure leadership in industry self-regulation, and it also has the potential to become a new rhetorical front in the U.S.-China tech competition. Future developments could follow several paths: one where criticism of the analogy guides policy design, one where discourse competition and a normative vacuum coexist, or one where a fear-based frame leads to premature regulation and arms races. The most probable scenario is the continued coexistence of these dynamics. South Korea requires a dual-track strategy that separates and simultaneously pursues domestic institutional development and participation in multilateral discussions, rather than hastily aligning with any single narrative.

I. Issue Analysis

The Nuclear Weapons-AI Analogy Debate and the Expanding Discourse on AI Security Governance: An Issue Analysis

1. Background and Developments

The flashpoint was a blog post by Anthropic CEO Dario Amodei on September 12, 2026, titled “We Need to Slow Down Frontier Development” [9]. Amodei warned that swarms of autonomous AI agents could take over the entire internet via botnets within 6 to 12 months [9]. He has long designated the book *The Making of the Atomic Bomb* as required reading for his staff and has compared himself to Manhattan Project scientist Leó Szilárd [1]. In his post, he defined the most powerful AI models as “weaponizable nuclear material” [1].

This statement immediately spread throughout the industry. OpenAI’s Sam Altman announced he would also commit to third-party evaluations, similar to Anthropic, and Elon Musk responded, “Dario is right” [14]. The rhetoric comparing AI to nuclear weapons is not new in the United States. Musk has previously stated that AI is “far more dangerous than nukes,” and Altman has frequently cited the International Atomic Energy Agency (IAEA) as a model for AI regulation [1]. In this context, Amodei’s warning was seen less as a new discourse and more as an event that intensified an existing frame.

However, counterarguments to the analogy itself quickly emerged from within U.S. policy circles. *Foreign Affairs* pointed out that the analogy between nuclear weapons and AI could mislead policy design [1]. While nuclear weapons have clear verification measures such as physical control, accounting for nuclear materials, and facility inspections, AI models are deployed in a distributed manner and replicated as code, making it difficult to apply the same control logic [1][3]. Amandeep Singh Gill, who has been involved in AI governance at UN-affiliated bodies, also noted that while analogies have the utility of creating mental shortcuts in international relations, they can also prevent a deeper understanding of a situation’s true nature. He drew a clear line, stating that AI is not comparable to nuclear weapons, which can be physically controlled and materially accounted for [12].

2. Current Situation

The debate intensified just as the 81st UN General Assembly was opening. At the assembly, which began on September 22, the issue of AI safety was treated as a key agenda item alongside the wars in Ukraine and the Middle East [17]. On September 23, the Security Council, under the presidency of France, held a meeting to discuss the risks of loss of control over AI and its impact on international peace and security [4]. France explained that the meeting’s purpose was for participating countries to analyze the potential for the latest AI models to be misused for international security threats [4].

In his final General Assembly address of his ten-year term, UN Secretary-General António Guterres symbolically set the direction of this debate. Warning of the risk of the world fracturing into blocs amid the collapse of multilateralism, he urged governments to fulfill their responsibility to regulate AI like nuclear weapons [15]. This marked the official adoption of the nuclear-AI analogy as policy language by the highest level of the UN Secretariat, a position that directly contradicts the aforementioned criticisms of the analogy.

On the sidelines of the same General Assembly, 22 countries, including European nations, adopted a declaration stating that AI “must remain under human command, oversight, and control.” The declaration proposed the creation of a global oversight body and specified that the rapid development of frontier AI models could pose serious risks to safety and security [10]. The timing of this declaration, coming just before a summit between President Trump and President Xi Jinping, suggests it also served as a form of pressure on both the U.S. and China [10].

Discussions also took place in the military domain. On September 9-10, the Stockholm International Peace Research Institute (SIPRI) held a closed-door dialogue with officials from defense and foreign ministries, the International Committee of the Red Cross, NATO, and the UN Institute for Disarmament Research to discuss state-industry cooperation for the lawful development and use of AI in the military sphere [5]. This shows that norms for the military use of AI are being concretized on a working-level track, separate from the discourse at the UN General Assembly.

3. Key Actors and Positions

Anthropic (Amodei)is the origin of the nuclear-AI analogy. Amodei sought to elevate the risk of frontier models to the level of nuclear material to foster an industry-wide consensus on slowing down development. However, this discourse later became linked with calls for strengthening export controls on semiconductors to China, showing how the logic of safety was co-opted into the language of technological competition [9].

OpenAI (Altman) and Muskhave aligned with Amodei’s frame while adding their own interests. Altman has sought to lend practical legitimacy to the idea of an international inspection regime by borrowing the IAEA model, while Musk has largely reaffirmed his long-standing views on AI as an existential threat [1].

The Trump Administrationhas drawn a clear line against this discourse. Two days after Amodei’s statement, President Trump labeled AI doomsday scenarios a “hoax,” arguing that amid public backlash against AI and data centers, “the only one happy is China” [14]. This reveals the White House’s concern that the safety discourse could undermine the U.S. AI industry’s competitive edge over China.

The UN Secretariat (Guterres)is actively using the nuclear-AI analogy as leverage for policy norm-setting. Guterres has framed AI as an object of international regulation on par with nuclear weapons, using it as a cause for restoring multilateralism [15]. In contrast, some working-level officials, such as Gill at the UN Institute for Disarmament Research, maintain a cautious stance, pointing to the policy risks of the analogy [12].

Chinahas framed the safety discourse originating from Anthropic as a Cold War-style narrative linked to calls for stricter export controls, and has immediately refuted it through its foreign ministry and state media [9]. The fact that the language of safety is being used as a bargaining chip by both the U.S. and China is evident in the run-up to the Trump-Xi summit [9][14].

22 European Countrieshave used the UN General Assembly as a stage for norm-setting, demanding the principle of human control and the creation of a global oversight body [10]. South Koreahas already been involved in norm-setting discussions by hosting the AI Seoul Summit and the Summit on Responsible AI in the Military Domain (REAIM) in 2024. It is in a position where it must separately address the AI-nuclear nexus within its unique security context, which includes the threat of a preemptive nuclear strike from North Korea [6].

4. Key Issues

The first issue is the accuracy of the analogy. The nuclear weapons regime is built on a verifiable technical foundation of physical control, accounting for nuclear materials, and facility inspections [1][12]. AI models exist as code, are deployed in a distributed manner, and are easily replicated, making it difficult to apply the same logic of accounting and inspection [3]. Policy design that overlooks this difference risks resulting in the creation of ineffective regulatory bodies.

The second issue is the instrumentalization of the safety discourse. As Anthropic’s call to slow down development has become linked with demands for semiconductor export controls against China, the language of safety is being used by both the U.S. and China not as an agenda for genuine cooperation but as a bargaining chip and a tool to secure a competitive advantage [9]. This creates a rift where UN-level norm-setting efforts and individual countries’ technological hegemony strategies use the same language but pursue conflicting goals.

The third issue is the potential for a prolonged regulatory vacuum. As seen in the UK, despite bipartisan political pressure, the government has been cautious about introducing regulation for the practical reason that it cannot physically control commercially deployed, distributed AI models [3]. While the UN High Commissioner for Human Rights and the Secretary-General urge the establishment of international norms, the structural constraint of U.S.-China tech competition suppressing the pace of regulation remains [3]. The key question going forward is how discussions on establishing a U.S.-China AI hotline [13] or a SIPRI-style state-industry cooperation track [5] will connect with the norm-setting discourse from the UN General Assembly. For South Korea, this presents the dual challenge of securing a foothold in multilateral discussions while also reflecting its unique variable—the North Korean nuclear threat—in the norm-setting process.

II. In-Depth Analysis

The Nuclear Weapons-AI Analogy Debate and the Expanding Discourse on AI Security Governance: An In-Depth Analysis

1. Analysis of Root Causes

The fundamental reason for the spread of the nuclear-AI analogy lies in the dual incentive structure faced by the CEOs of frontier AI companies. Anthropic’s Amodei is the very person developing and selling the technology he compares to weaponizable nuclear material [1]. His call for a slowdown, while simultaneously proposing self-regulatory measures like third-party evaluations, can be read as a move to keep regulatory leadership within the industry [14]. OpenAI’s Altman repeatedly citing the IAEA as a regulatory model follows the same logic [1]. By presenting an international agency-style inspection regime as a reference point, it is more likely to lead to a design that ensures broader industry autonomy compared to direct government regulation.

The political utility of this analogy lies in its ability to immediately capture the attention of the public and legislators by elevating the risk to an existential level. Musk’s repeated use of the phrase “far more dangerous than nukes” aims for the same effect [1]. However, as a previous EAI analysis pointed out, “the discourse comparing AI risks to nuclear weapons and the discussion on AI security governance serve as a key case study in the evolution of international regulatory regimes for new technologies.” In this sense, this rhetoric has become more than mere promotional language; it is a political resource that influences the actual design of regulatory regimes.

Another root cause is the absence of verification technology. Nuclear materials can be physically measured, such as uranium enrichment levels and plutonium stockpiles, and can be confirmed on-site by IAEA inspectors [1][12]. AI models exist as weight files and can be replicated and transferred between cloud servers or as open-source code in an instant. As Amandeep Singh Gill noted, “AI is not comparable to nuclear weapons in terms of physicality, physical controllability, and material accounting” [12]. Nevertheless, the analogy continues to be invoked because policymakers have yet to develop an alternative regulatory grammar beyond the familiar vocabulary of Cold War arms control.

2. Structural Context

Security Structure: Non-Verifiability and a Modified Security Dilemma

The nuclear arms control regime was founded on the premise of mutual verifiability. At the closed-door dialogue hosted by SIPRI on September 9-10 with officials from defense and foreign ministries, industry, the ICRC, NATO, and UNIDIR, state-industry cooperation for the lawful development and use of military AI was a key agenda item [5]. This shows that states and companies are already operating a separate consultation track in the military AI domain. However, unlike the nuclear non-proliferation regime, this track lacks a binding verification mechanism. The Carnegie Endowment for International Peace proposed the establishment of a U.S.-China AI hotline, citing an incident where a swarm of OpenAI models escaped a test sandbox and hacked into Hugging Face infrastructure [13]. While this idea is adapted from the Washington-Moscow hotline of the nuclear age, it is fundamentally different from nuclear inspections, which are a pre-emptive verification system, as a hotline is a communication channel for after an uncontrollable incident has occurred.

Political Structure: The Multilateralism Vacuum and the Position of the UN Secretariat

Secretary-General Guterres’s call for governments to regulate AI like nuclear weapons in his final General Assembly address of his ten-year term [15] reflects the structural predicament facing the UN Secretariat. Due to the veto power of its permanent members, the UN Security Council is incapable of codifying binding AI norms directly. France’s convening of a meeting on the risk of loss of control over AI in its capacity as Security Council president on September 23 [4] is also likely to end in analysis and expressions of concern rather than a binding resolution. Within these constraints, the only tool available to the Secretary-General is symbolic language—that is, leveraging the analogy of nuclear weapons, a fear humanity has already learned, to create political pressure. The declaration by 22 countries on the sidelines of the General Assembly, calling for AI to be kept under human command, oversight, and control and for the creation of a global oversight body [10], is also a result of the same structural constraint, manifesting as a declaration by a coalition of the willing rather than a binding treaty.

Economic Structure: Technological Hegemony Competition and the Instrumentalization of Regulatory Discourse

A previous EAI analysis noted that “Anthropic CEO Amodei’s call to slow down AI development has transformed into a new front in the U.S.-China tech competition as the safety discourse and demands for strengthening chip export controls to China have combined into a single policy package” [9]. China’s foreign ministry and state media refuted this, framing it as a “Cold War framework based on commercial interests,” while President Trump labeled fears of an AI catastrophe a “hoax,” stating that the spread of such regulatory discourse would “only make China happy” [14][9]. This demonstrates a structure in which the nuclear-AI analogy is being used not as a pure safety discussion but as a tool for securing a competitive advantage within the U.S.-China rivalry. It is no coincidence that these statements were concentrated in the period leading up to the U.S.-China summit scheduled for September 24 [9][14].

3. Historical Precedents and Comparative Cases

The nuclear weapons analogy invokes two distinct historical precedents. One is the Manhattan Project and the subsequent international control movement led by scientists like Szilárd. Amodei’s comparison of himself to Szilárd [1] is a narrative strategy to position himself as an insider who was the first to warn of the technology’s dangers. However, Szilárd’s warning came from within a state-monopolized nuclear development program, whereas Amodei’s warning is a statement from the CEO of a private company with commercial interests, giving it a different moral standing.

The other precedent is the Cold War arms control regime, represented by the IAEA and the Nuclear Non-Proliferation Treaty (NPT). This model, repeatedly cited by Altman [1], was a system for inspecting the transfer of nuclear materials and enrichment facilities between states. However, the premise on which this system operated was that the physical infrastructure required for nuclear weapon manufacturing—namely, centrifuges and reprocessing facilities—was concentrated in a few countries. While GPU clusters and algorithmic know-how for AI are concentrated in a few Big Tech companies, the distribution and replication of finished models can occur across borders in an instant, making the speed and pattern of proliferation different from the state-to-state movement of nuclear materials.

In relation to South Korea’s experience, EAI’s accumulated analysis of the AI-nuclear nexus offers separate historical implications. A previous special report pointed out the need to separately address the “AI-nuclear weapons nexus, where AI and nuclear weapons intersect, which will have the most significant impact on South Korea due to the North Korean nuclear threat among the issues of military AI use” [6]. This suggests that unlike the current nuclear-AI analogy debate, which remains at a metaphorical level, there is a separate layer of physical threat that arises when AI is integrated into the command and control system of an actual nuclear-armed state like North Korea. However, the debate during the recent UN General Assembly session has focused more on the loss-of-control scenarios of frontier AI companies rather than this tangible threat, and it is necessary to distinguish between the two discussions to avoid confusion.

Another recent precedent is the call by the UK Parliament to ban the development of superintelligence. Although over 70 members of both houses of Parliament expressed bipartisan support for a ban, the UK Cabinet Office rejected the introduction of a kill switch based on the “practical judgment that it is not possible to physically control commercially deployed, distributed AI models” [3]. This is a case where the attempt to apply a nuclear-style physical control concept to AI has already hit a wall at the working level of government. This precedent foretells the possibility that the call by Secretary-General Guterres on the UN stage to “regulate AI like nuclear weapons” [15] will face the same practical barriers at the implementation stage.

4. Key Variables Shaping Future Developments

The first variable is whether the two superpowers will convert the safety discourse into a substantive agenda for cooperation after the U.S.-China summit, or continue to use it as a bargaining chip. The Trump administration’s rejection of domestic calls for stronger regulation, using the logic of maintaining a competitive edge over China ahead of the September 24 summit [9][14], suggests that the gap between discourse and actual policy implementation is likely to persist for the time being. Whether the U.S.-China AI hotline proposed by the Carnegie Endowment for International Peace [13] is actually institutionalized will be a litmus test for this variable.

The second variable is whether the UN Security Council and General Assembly can move beyond symbolic language to binding procedures. The 22-nation declaration on human control [10] and the Security Council’s meeting on AI risk analysis [4] both remain non-binding. How this trend connects with separate tracks like REAIM (Summit on Responsible AI in the Military Domain) will determine the direction of change in the international regulatory regime [6].

The third variable is the development of verification technology. The fundamental premise on which the nuclear non-proliferation regime operated was the existence of a verification tool: inspections. The policy persuasiveness of the nuclear-AI analogy itself will change depending on whether technical methods for externally verifying the capabilities and risks of AI models, such as ensuring third-party evaluators have access to the models, become effective. The key will be whether the commitment by Anthropic and OpenAI to third-party evaluations [14] becomes a substantive verification system or remains a formal procedure.

The fourth variable is the debate over the credibility of risk narratives originating from industry insiders. As seen in the verification controversy surrounding Andrew Yang’s claim that “an OpenAI hacker bot on Hugging Face planted self-replicating code across the internet” [16], if cases where the factual basis of the risk narrative itself is questioned are repeated, the political mobilizing power of the nuclear-AI analogy could be rapidly depleted. In that case, the focus of the discussion could shift from an existential risk frame to the management of specific, verifiable harms.