The Case of an Erroneous Military Order Against China from an AI Hallucination and South Korea’s Task of Building a Verification System

Executive Summary

This spring, the U.S. military prepared an interdiction and seizure operation against a Chinese vessel in the Middle East based on an intelligence report co-authored by an AI chatbot. The operation was aborted at the last minute after fighter jets had already been scrambled. The source of the error was not adversarial deception but the U.S. military’s own intelligence production process. A key flaw was that a poorly substantiated conclusion gained institutional credibility by being formatted as a standard report and was then rapidly disseminated through the chain of command. The currently discussed U.S.-China AI dialogue focuses on notifying the other country of AI-related incidents, a framework that does not structurally cover internal errors like this one. Its effectiveness is also limited by the exclusion of core conflict variables, such as semiconductor export controls, from its agenda. This incident demonstrates that the key to crisis stability lies not in notification channels but in the timing of human verification procedures. South Korea’s priority should be to secure independent verification procedures and access to source data for AI-generated information entering the ROK-U.S. combined intelligence system. Furthermore, South Korea should undertake diplomatic efforts to lead the international community, including the U.S. and China, in addressing this issue.

I. Situational Analysis

Situational Analysis of the Erroneous Military Order Against China Caused by AI Hallucination

1. Background and Timeline

The incident dates back to this spring, during the ongoing war with Iran. The U.S. military received an intelligence report stating that a Chinese cargo ship sailing in Middle Eastern waters was carrying components related to a nuclear weapons program [7][11]. This report was written with the assistance of an AI chatbot [1][4]. The problem was that this conclusion was false. According to four sources cited by CNN, the report was rapidly disseminated within the U.S. military's chain of command after being formatted as a standard intelligence report [11]. A Croatian media outlet also reported that the document was circulated in just a few minutes [11].

Based on this information, the U.S. military command actually drew up a plan for an interdiction and seizure operation against the vessel [11]. Fighter jets were scrambled, and the final decision on whether to halt the operation was reviewed while the aircraft were already in the air [1][9]. The operation was canceled at the last moment [1][4][9]. The Swiss newspaper NZZ described the event as a "fatally precise error," assessing it as a case that demonstrated how misplaced technological trust in inaccurate data could lead to catastrophic consequences [4].

This was not an isolated incident. It came to light alongside an erroneous strike during the same period in which Palantir's Maven AI targeting system incorrectly designated an elementary school in Minab, Iran, as a target, leading to the deaths of over 150 people, including 120 children [3][6][14]. The Austrian newspaper Der Standard reported on both incidents together, raising the possibility that the Palantir AI may have mistaken already destroyed video footage for current data during the Iran airstrike [14]. Both cases revealed a common structural flaw: human verification processes failed to keep pace with the speed at which AI-generated outputs were disseminated.

2. Current Situation

The incident became public when CNN broke the story exclusively on September 18 [1][12]. The timing of the report was, coincidentally, just before then-President Trump announced restrictions on White House access for media outlets including CNN, MS NOW, and Politico [14]. U.S. defense officials have not released an official explanation of the incident's circumstances. The report quickly spread to media outlets worldwide, including in French-speaking West Africa [13], Finland [9], Estonia [16], Croatia [11], India [12], and Brazil [7]. This indicates that the possibility of an accidental AI-driven conflict in U.S.-China relations is perceived not merely as a bilateral issue for Washington and Beijing but as a matter of concern for the entire international community.

China's state-run media outlet, Global Times, used the incident as evidence to support the urgency of U.S.-China AI safety cooperation. Citing a Chinese expert, it assessed that the very fact that "falsely generated U.S. intelligence almost led to a U.S. military plan for the use of force against a Chinese vessel" demonstrates the urgent need for AI safety cooperation between the two countries [15]. This can be interpreted as China seeking to frame the incident as a flaw in the U.S. internal intelligence system and use it as a justification for participating in dialogue, rather than escalating it into a dispute over its own responsibility.

Meanwhile, U.S. Treasury Secretary Besant announced that separate consultations to establish a U.S.-China AI dialogue had already begun [17]. This consultative body aims to create a mechanism for mutual notification in the event of AI incidents that could threaten national security [17]. However, with the incident's disclosure, questions were immediately raised as to whether such a dialogue could actually cover incidents of this type.

3. Key Actors and Stances

The U.S. military chain of commandis the most directly involved party in this incident. The root of the problem was not adversarial deception or hacking but originated within its own intelligence production process [1][4]. The fact that the standard report format failed to filter out the error and instead served as a channel to confer institutional credibility suggests a structural vulnerability in the U.S. military's internal information verification system. This was a problem also pointed out in the earlier erroneous strike by the Palantir Maven system [3][14], leading to assessments that the pace of the U.S. military's AI adoption is outstripping the speed at which it can establish verification systems.

Chinaopted for discourse development through its state-run media rather than lodging a protest through official diplomatic channels. Global Times framed the incident as a technical flaw on the part of the United States, using it as a basis to emphasize the need for U.S.-China cooperation in AI safety [15]. Since the incident did not lead to direct military action against its vessel, China has an incentive to choose a cooperative framework over an escalatory response.

The U.S. Treasury Department and related authoritieshave already been separately pursuing the establishment of a U.S.-China AI dialogue [17]. However, from its design phase, this dialogue has excluded core issues of U.S.-China AI competition, such as semiconductor export controls, from its agenda. Its focus is on mutual notification of national security-level AI incidents, meaning that functions to preemptively detect or prevent errors arising from a country's internal information production process, as seen in this case, are outside its original design scope.

Middle-power actorslike South Korea are not direct parties to this incident, but they have a stake in it due to the potential for similar structural risks within the ROK-U.S. combined intelligence system. It has already been pointed out separately that the very structure of supplying Palantir Maven-series systems to multiple allies in a standardized form carries an inherent risk of error propagation [3].

4. Key Issues

The first issue is the point of origin of the error. This incident was not caused by an adversary's deception tactics or cyber infiltration, but by a false conclusion generated by a domestic AI tool that was circulated without verification after being formatted as a standard report [1][4][11]. This shows that in the context of U.S.-China AI competition, threats do not necessarily come only from the outside.

The second issue is the placement of the verification process. Substantive verification of the error occurred only after the aircraft had already been scrambled, just before the operational phase [1][9]. This case reaffirmed that the key to crisis management lies not in post-hoc notification channels but in the robustness of pre-emptive human verification procedures.

The third issue is the effectiveness of the ongoing U.S.-China AI dialogue. This dialogue focuses on a mutual notification mechanism [17], making it structurally difficult to cover internal information production errors that occur before they become subject to notification, as was the case here. The fact that fundamental variables of AI competition, such as semiconductor export controls, are excluded from the agenda also limits the dialogue's ability to perform a genuine crisis prevention function.

The fourth issue is the risk exposure for allies and middle powers. The potential for errors in the U.S. military's AI-based intelligence and targeting systems to propagate into allied combined operations systems is a problem already identified in the Palantir Maven case [3]. Allies, including South Korea, need to consider securing access to source data and independent verification procedures as a prerequisite when adopting U.S.-led AI-based intelligence and command-and-control standards.

II. In-depth Analysis

In-depth Analysis of the Erroneous Military Order Against China Caused by AI Hallucination

1. Analysis of Root Causes

The source of the error in this incident was not adversarial deception or hacking. It was a hallucination that occurred within the U.S. military's own intelligence production process [1][4]. An intelligence report supported by an AI chatbot misidentified the cargo of a Chinese vessel as nuclear weapons program components [7][11]. This conclusion was poorly substantiated. Nevertheless, it passed through the standard intelligence report format [11]. The fact that it went through a formal procedure did not guarantee the credibility of its content.

Here, the core problem emerges. AI-generated text takes the form of a standardized report. This format itself acts as a signal of credibility to reviewers. The Croatian newspaper Jutarnji list reported that the report triggered the highest level of alert within the U.S. military chain of command in just a few minutes [11]. The speed of dissemination outpaced the verification of the information's authenticity. The Swiss NZZ described this as a "fatally precise error." It was an assessment that this case demonstrated how misplaced technological trust in inaccurate data could lead to catastrophic consequences [4].

The same structural flaw was identified in the erroneous strike on the Minab elementary school in Iran. It has been suggested that the Palantir Maven targeting system, in the process of identifying approximately 13,000 targets over 38 days, may have mistaken already destroyed video footage for current data [3][14]. The Austrian newspaper Der Standard reported on both incidents on the same day, highlighting their common denominator [14]. The processing speed of AI targeting and intelligence analysis compressed the time available for human verification. This should be seen not as a flaw in an individual system but as a problem spanning the entire military decision-making structure following the introduction of AI.

2. Structural Context

The high-intensity crisis situation of the Iran war served as the backdrop for the error. At a time when it was engaged in conflict with Iran, the U.S. military was simultaneously conducting a mission to control shipping in Middle Eastern waters [7][13]. In a crisis, the cycle of intelligence production and consumption shortens. Commanders are required to make swift judgments. In such an environment, the rapid information synthesis capability offered by AI is an attractive tool. At the same time, it also acts as an incentive to bypass verification procedures. The fact that the error was discovered only after fighter jets had already been scrambled [1][9] means that verification took place not before the start of the operation, but just before the critical moment.

Viewed through the lens of U.S.-China relations, this incident illustrates a new pathway to accidental military conflict. Previous U.S.-China military tensions have been discussed mainly in the context of close encounters or gray-zone provocations in the Taiwan Strait and the South China Sea. This incident showed that it is possible to reach the brink of using force purely from an error in the intelligence production stage, without any actual physical contact or provocative act. The Brookings Institution, referencing the 2023 reconnaissance balloon incident, had previously pointed out that without safety mechanisms like a hotline, AI-based unmanned aerial vehicles and unmanned surface vessels are likely to trigger more accidental escalations in the future [2]. This incident confirmed that the same risk exists at the intelligence analysis stage, not just with unmanned platforms.

The AI dialogue that the U.S. and China are separately pursuing does not adequately cover this structural risk. The U.S.-China AI dialogue mentioned by Treasury Secretary Besant focuses on a mechanism for notifying the other country of AI incidents that could threaten national security [17]. This is a system for the two countries to inform each other of incidents after they have occurred. However, in cases like this incident, where the hallucination occurred within a country's own intelligence production process, there is little incentive to notify the other country in the first place. The very definition of an 'incident' covered by the dialogue is designed in a way that makes it structurally difficult to capture this type of error. Moreover, key variables of U.S.-China AI competition, such as semiconductor export controls, are excluded from this dialogue's agenda. This means the effectiveness of the notification channel is limited from the outset.

3. Historical Precedents and Comparative Cases

The closest comparison is the erroneous strike on the Minab elementary school that occurred in the same year. In that case, an error by the Palantir AI targeting system led to an actual strike, killing more than 150 people [3][6][14]. A UN fact-finding mission determined that this could constitute a war crime. In contrast, the White House immediately denied the investigation's findings [3]. The crucial difference between this case involving the operation against China and the Minab incident is whether the consequences materialized. In the Minab case, the verification failure resulted in actual casualties. In the operation against China, last-minute intervention prevented physical damage. When the two incidents are placed side by side, it becomes clear that whether an AI error leads to an actual disaster was a matter of near-accidental timing. It was not because human verification was structurally guaranteed in the system's design.

An older precedent is the 2023 reconnaissance balloon incident mentioned by the Brookings Institution [2]. That incident was a case of conventional misidentification without AI involvement. Nevertheless, it left a structural lesson in that the absence of a crisis communication channel between the U.S. and China amplified tensions. At that time, the need to establish a hotline was raised afterward, but substantive progress was slow. The re-emergence of discussions about a U.S.-China AI dialogue in the wake of this AI hallucination incident follows a similar pattern. The mode of triggering discussions on communication mechanisms only after a crisis has occurred is being repeated.

A comparison with Cold War-era false alarm cases is also valid. In the 1983 incident where the Soviet early warning system falsely detected a missile launch, the personal judgment of the duty officer prevented escalation. This recent incident is different in that the error was caught not by an individual's judgment, but during a final review stage after the aircraft had already been scrambled [1][9]. In that human intervention was closer to an accidental discovery than a procedure guaranteed by the system's design, this case can be seen as revealing a crisis management structure that is actually more fragile than that of the Cold War era.

4. Key Variables Shaping Future Developments

The first variable is whether the U.S. military's internal intelligence verification procedures will be reformed. The fact that the error was circulated after gaining credibility by passing through the standard report format [11] suggests that there was no separate labeling or classification system for AI-generated information. The key will be whether the U.S. military introduces procedures to distinguish between AI-generated outputs and human-verified information in the future, and whether it moves that verification to a stage prior to the commencement of operations.

The second variable is whether the agenda of the U.S.-China AI dialogue will be expanded. The currently discussed framework is limited to mutual notification [17]. Whether this framework will be expanded to include demands for transparency regarding internal errors, or whether it will remain a formalistic channel, is a variable that will determine crisis stability. The fact that the Chinese side is using this incident as a justification for U.S.-China AI safety cooperation [15] shows that Beijing has an incentive to engage actively in this discussion. However, as long as substantive competitive variables like semiconductor export controls are excluded from the agenda, the gap between the dialogue's political symbolism and its practical effectiveness is likely to persist.

The third variable is public opinion and political fallout. The fact that CNN's exclusive report came out just before then-President Trump's announcement of restrictions on White House access for the media outlet [14] implies that this incident could become entangled with the domestic political debate over media control in the United States. With the U.S. Department of Defense yet to issue an official explanation, whether further revelations or congressional demands for an investigation follow will determine the incident's impact.

The fourth variable is the ripple effect on allies. The U.S. military's AI-based intelligence system is linked with various allied networks, including the ROK-U.S. combined intelligence system. If the verification gap revealed by this incident spreads to become a problem of credibility for information shared with allies, there will be a greater incentive for allied nations to demand independent verification rights in the process of adopting U.S.-led AI-C2 standards.