The AI Memory Wall and Power Bottleneck: Structural Constraints in the U.S.-China Infrastructure Competition and Their Policy Implications

Executive Summary

The scaling paradigm that has underpinned the expansion of AI models is hitting physical limits, namely memory bandwidth and power supply. Nvidia has scaled back its Rubin Ultra design from a 12-stack to an 8-stack High Bandwidth Memory (HBM) configuration due to supply constraints, while Chinese semiconductor firms are sharply increasing accelerator prices because of the same bottleneck. Huawei is attempting to bypass this gap with optical interconnect technology, challenging the Nvidia- and Broadcom-centric standards with its own proprietary 7.2 Tbps Near-Packaged Optics (NPO) module. This development is shifting the front line of the U.S.-China technology competition from raw GPU performance to a contest over interconnect and packaging standards. The bottleneck is also affecting the U.S. Department of Defense's deployment speed of AI for missile and space threat detection. In a baseline scenario where power and memory constraints persist, competition for resources between the defense sector and the commercial AI ecosystem is inevitable. South Korea faces both collateral benefits and risks in the HBM supply chain, and any decision on participating in Huawei's standard must be preceded by a government-level review of U.S. export control risks, rather than being left to private-sector judgment.

I. Situational Analysis

The 'Memory Wall' and Power Bottleneck in AI Development: Structural Constraints in the U.S.-China AI Infrastructure Competition

1. Background and Developments

The physical limits of scaling AI models are coming to the forefront of the industry. This is the so-called “memory wall” problem: the gap between the speed at which GPUs perform computations and the speed at which memory can supply data. The Israeli media outlet Globes pointed out that this issue “forces unsustainable energy consumption, where the need to deploy so many GPUs grows exponentially” [12]. When a processor hits this memory ceiling, it “waits idly, burning energy for half the time, just waiting for data to flow in” [12].

This bottleneck first became visible in specific segments of the semiconductor supply chain. Taiwan's DigiTimes Asia reported that Nvidia's next-generation Rubin Ultra accelerator has been scaled back from its original 12-stack High Bandwidth Memory (HBM) design to an 8-stack version. The change was reportedly driven by rising HBM costs, leading to a design that prioritizes bandwidth efficiency [17]. The same outlet also reported that Chinese semiconductor companies challenging Nvidia are sharply increasing the prices of their current and next-generation accelerators due to HBM shortages. This adds another cost constraint to China's push for homegrown computing hardware [14].

China has opted to bypass this bottleneck with optical technology. At the China International Optoelectronic Exposition (CIOE) in Shenzhen, Huawei unveiled what it claims is the industry's first 7.2 terabits per second (Tbps) Near-Packaged Optics (NPO) module. This approach replaces copper wiring with optical signals to increase data transfer speeds between AI processors [1]. The Nihon Keizai Shimbun framed this as a “challenge to Nvidia and Broadcom,” analyzing it as an attempt by Huawei to secure a greater say in an emerging area of chip manufacturing that could shape the future of next-generation AI computing [9]. Similarly, Taiwan's DigiTimes Asia assessed that Huawei is not merely launching a new product but is also attempting to “create a new optical interconnect standard as AI clusters hit their limits” [5].

2. Current Situation

Huawei's announcement is highly symbolic, as it comes amid U.S. export controls on semiconductors to China. Huawei's move to challenge the existing optical and interconnect standards, which are centered around Nvidia and Broadcom, with its own proprietary specifications is an extension of China's strategy to close the technology gap using bypass technologies, given its restricted access to advanced GPUs. However, as confirmed by DigiTimes Asia, Huawei and other Chinese firms are not immune to the shared constraint of HBM shortages [14].

The power bottleneck carries broader geographical implications. The Brussels-based think tank Bruegel has assessed that the EU's domestic AI computing capacity stands at just 2 gigawatts, or 5% of the global total, and that planned projects will do little to close this gap. This points to Europe's structural disadvantage, as indigenous computing infrastructure is a prerequisite for AI sovereignty [6]. Similarly, the German consulting firm Roland Berger analyzed that Europe is falling further behind in the winner-and-loser dynamic created by the data center boom [4]. The Carnegie Endowment for International Peace has argued that the physical limits of power supply and training data cast doubt on the sustainability of the scaling paradigm itself, presenting this constraint as a problem inherent to the technological paradigm [13].

Power infrastructure problems are also escalating into security vulnerabilities. The Brookings Institution reported that in March, three Amazon Web Services (AWS) data centers in the Middle East were damaged by a drone attack that Iran dubbed ‘Operation Epic Fury.’ The attack reportedly cut off power, triggered fire suppression systems, and resulted in water damage. Iranian state media claimed that the facility in Bahrain was targeted because it provided support functions for the U.S. military [2]. This incident demonstrates that frontier AI data centers built overseas can become targets for physical attacks, suggesting that the securitization of computing and power infrastructure is already underway.

3. Key Actors and Positions

Huawei (China)is leveraging its optical module technology to offset the gap in advanced GPUs, a result of U.S. export controls, by seeking to establish a dominant position in interconnect standards. As the Nihon Keizai Shimbun notes, this is not merely a product competition but a contest for standards leadership that “could shape the future of next-generation AI computing” [9]. Meanwhile, Chinese semiconductor firms more broadly are facing HBM supply shortages and have resorted to the defensive measure of price hikes [14].

Nvidia (United States)is responding to rising HBM costs by modifying its product designs. The decision to scale back the Rubin Ultra design from a 12-stack to an 8-stack HBM configuration reflects a prioritization of bandwidth efficiency and system economics [17]. This shows that even U.S. Big Tech firms are not immune to the memory bottleneck.

U.S. Department of Defenseis pushing for the operational deployment of AI despite these infrastructure constraints. Defense News reports that the DoD is pursuing AI adoption to identify missile and space threats, aiming to overcome the challenges of missile defense—such as high-speed objects, short response times, and ambiguous sensor data—with artificial intelligence [16]. Defense startups are also introducing AI systems designed to maximize the effectiveness of each round of ammunition [19]. Ultimately, however, the pace of these deployments will be constrained by infrastructure bottlenecks in GPU availability and power supply.

China's People's Liberation Army (PLA)has itself assessed that its military AI applications remain confined to auxiliary roles. According to an EAI report, researchers from the Shandong Corps of the Chinese People's Armed Police Force acknowledged that the PLA's military AI is concentrated in “auxiliary areas such as intelligent Q&A and content generation,” while its offensive combat applications are still in an exploratory phase [11]. This gap is analyzed as a structural problem stemming not from a lack of technology but from political caution surrounding Party-military relations and the delegation of command authority [11]. The Brookings Institution has also pointed out that the PLA's Strategic Support Force (SSF) has been pursuing an aggressive innovation strategy with the goal of becoming the world's leading AI power by 2030 [10].

Europe (EU member states and companies)is seeking ways to secure its autonomy amid a structural deficit in computing infrastructure. The Bruegel think tank has recommended that the EU maximize its domestic computing capacity, regardless of the technology's country of origin [6].

4. Key Issues

The memory wall and power bottleneck have emerged as the next front that could determine the outcome of the U.S.-China AI competition. Huawei's optical module represents an attempt to establish a dominant standard not in GPU performance itself, but in the alternative pathway of inter-GPU data transmission. This shows that there are domains that U.S. semiconductor export controls cannot effectively contain.

At the same time, this bottleneck affects the U.S. and its allies symmetrically. Nvidia's design scale-back and the EU's shortage of computing capacity confirm that the bottleneck is not a one-sided problem. As the Carnegie Endowment for International Peace suggests, if the physical constraints of power and data expose the limits of the scaling paradigm itself, the focus of competition could shift from ‘who can secure more GPUs’ to ‘who can first build an architecture that bypasses the bottleneck’ [13].

In the defense sector, these infrastructure constraints are a key variable determining the pace of operational deployment. Both the U.S. Department of Defense's AI for missile and space threat detection [16] and the PLA's pursuit of integrated command and control [11] are ultimately dependent on securing sufficient computing and power resources. Moreover, Iran's attack on AWS data centers [2] demonstrates that AI infrastructure itself can become a physical target, highlighting infrastructure security as an emerging vulnerability in the military AI race.

II. In-Depth Analysis

The 'Memory Wall' and Power Bottleneck in AI Development: Structural Constraints in the U.S.-China AI Infrastructure Competition

1. Analysis of Root Causes

The memory wall problem originates from a long-standing asymmetry in the semiconductor industry. While computational performance has steadily improved due to Moore's Law and advances in parallelization, the data transfer bandwidth between memory and processors has not kept pace. As an industry source cited by Globes explains, deploying GPUs on a large scale causes “energy consumption [to] increase exponentially,” yet the processors themselves “wait idly for half the time, just waiting for data to flow in” [12]. This is the result of a structural divergence between computational power and data supply capacity.

This gap has recently manifested as a shortage in HBM supply. Even Nvidia has downgraded the HBM configuration for its Rubin Ultra accelerator from the planned 12-stack to an 8-stack design, a change that prioritizes bandwidth efficiency and system economics [17]. The situation is even more precarious for Chinese semiconductor firms challenging Nvidia. As HBM becomes scarcer and more expensive, they are sharply increasing the prices of their current and next-generation accelerators [14]. For China, which is committed to promoting its domestic computing hardware, this presents a double bind: a cost constraint layered on top of the existing technology gap.

The root cause of the power bottleneck is different. The demand for electricity to train and run AI models is outpacing the expansion of the global power grid for data centers. The Carnegie Endowment for International Peace has pointed out that “it is unclear how much longer developers can continue scaling due to the limits of power supply and training data” [13]. This suggests that the fundamental approach of growing models by adding more computing power is itself hitting a physical wall. This is not a semiconductor design issue but a constraint on an entirely different level: energy infrastructure, specifically power generation and transmission capacity.

2. Structural Context

Technological Competition Structure: Huawei's unveiling of its 7.2 Tbps NPO module is a response to the institutional constraint of U.S. semiconductor export controls. It represents an attempt by China, which is blocked from acquiring advanced GPUs, to find a workaround using interconnect technology that replaces copper wiring with optical signals [1]. The Nihon Keizai Shimbun described this as a “challenge to Nvidia and Broadcom,” a move by Huawei to secure a greater say in a chip manufacturing domain that could dictate future AI computing standards [9]. DigiTimes Asia similarly assessed that Huawei is not just launching a new product but is also attempting to “create a new optical interconnect standard as AI clusters hit their limits” [5]. This competition over standard-setting represents a new front in the U.S.-China technological rivalry. The focus is shifting from the performance of individual semiconductors to the question of who will set the rules for interconnect and packaging specifications.

Military and Security Structure: This bottleneck is directly linked to the pace of operational deployment for defense AI. A key example is the U.S. Department of Defense's initiative to adopt AI for detecting missile and space threats. The goal is to use AI to alleviate the burden on human operators who must handle “high-speed objects, short time constraints, and unclear sensor data” in combat scenarios [16]. However, such real-time command and control AI requires massive computing resources and a stable power supply. If the memory wall and power bottleneck persist, the pace of defense AI expansion will be inherently limited. The situation is similar for China. The Brookings Institution has analyzed that the PLA is pursuing an aggressive innovation strategy to become a world-leading AI power by 2030 [10]. Yet, according to an EAI analysis, even researchers within the PLA have assessed that their “military AI applications remain confined to auxiliary areas such as intelligent Q&A and content generation,” with offensive combat use still in an exploratory phase [11]. This gap is analyzed as a “structural problem stemming not from a lack of technology but from political caution surrounding Party-military relations and the delegation of command authority” [11]. When combined with the computing bottleneck, this political caution could doubly delay the PLA's progress in AI militarization.

Supply Chain and Economic Security Structure: The HBM shortage is a common challenge affecting both the United States and China. With HBM supply so tight that even Nvidia has had to scale back its designs, the position of Chinese firms—whose access to advanced memory is already restricted by export controls—is even more difficult [14][17]. This gives China a strong incentive to stake its success in the race for memory bandwidth on an alternative path: optical interconnects. In this context, Huawei's unveiling of its NPO module functions as both an industrial strategy and a supply chain bypass strategy.

Geopolitical Vulnerability Structure: The physical concentration of computing infrastructure creates new security vulnerabilities. The Brookings Institution highlighted the March incident in which three AWS data centers in the Middle East suffered structural damage from an Iranian drone attack. The attack cut off power, activated fire suppression systems, and caused additional water damage [2]. Iranian state media claimed that the Islamic Revolutionary Guard Corps targeted the facility in Bahrain because of its role in supporting the U.S. military [2]. This incident demonstrates that as more AI infrastructure is built overseas, the surface area exposed to such physical attacks grows. This creates a paradoxical situation where the very expansion of infrastructure aimed at resolving power and memory bottlenecks introduces new security risks.

Structure of Regional Disparities: The competition over computing infrastructure extends beyond the U.S.-China duopoly. The Bruegel think tank assessed that the EU's domestic AI computing capacity is only around 2 gigawatts, a mere 5% of the global total, and that planned projects will do little to close this gap [6]. This highlights that while Europe considers indigenous computing infrastructure a prerequisite for AI sovereignty, it is starting from a position of structural disadvantage. Roland Berger also analyzed that Europe is falling further behind in the winner-take-all dynamic of the data center boom [4]. This suggests a risk that a structure may become entrenched in which a third pole lags behind in the computing infrastructure race led by the United States and China.

3. Historical Precedents and Comparative Cases

It is not uncommon in the semiconductor industry for a bottleneck to become an arena for standards competition. The emergence of HBM itself followed a similar pattern, as it was an attempt to bypass the bandwidth limitations of conventional DDR memory using 3D stacking technology. Huawei's current push for optical interconnects is an attempt to bypass the next bottleneck: the physical wiring limits between processors and memory that even HBM cannot fully resolve. A sequential pattern is repeating itself, where the resolution of one bottleneck reveals the next.

From the perspective of standards competition, this situation parallels how Huawei secured influence within international standardization bodies in the telecommunications industry through its 5G standard-essential patents. At that time, Huawei solidified its position across the industrial ecosystem less through raw hardware performance and more through its influence over the standard-setting process. Similarly, the recent unveiling of its NPO module is not merely about product competition but, as the Nihon Keizai Shimbun notes, is aimed at securing “a greater say in an emerging area of chip manufacturing that could shape the future of next-generation AI computing” [9]. The arena of competition has effectively shifted from telecommunications standards to those for semiconductor and AI infrastructure.

The power bottleneck can be compared to historical location decisions in energy-intensive industries. Just as aluminum smelting and data storage industries migrated to regions with lower electricity costs, a similar trend is emerging with AI data centers relocating to areas with readily available power. However, the location of AI data centers is not determined by cost logic alone, as they are often directly linked to national security data. The vulnerability of overseas data centers to physical attack, highlighted by the Brookings Institution, illustrates this distinction [2]. This implies that the economic logic for offshoring energy-intensive industries does not apply straightforwardly to AI infrastructure.

4. Key Variables Shaping Future Developments

The first variable is the pace of recovery in the HBM supply. The relative appeal of China's optical bypass strategy will depend on how long the current tight supply situation—severe enough to force even Nvidia to alter its designs—continues [14][17]. If the HBM supply normalizes, the perceived need for Huawei's optical interconnect solution could decrease.

The second variable is any adjustment to the scope of U.S. export controls on China. Whether optical modules and interconnect technologies are added to the control list will determine the potential for Huawei's standard to proliferate internationally. Tighter controls would likely confine it to a closed ecosystem centered on China's domestic market, whereas looser controls could create an opening for it to spread to non-Western markets.

The third variable is the pace of power infrastructure expansion and its political acceptability. The surge in electricity demand from data centers could conflict with local community opposition, challenges in financing grid investments, and renewable energy transition targets. How these tensions are resolved will determine the respective pace of computing expansion in the United States, China, and Europe.

The fourth variable is the actual progress of defense AI adoption. A key factor will be the extent to which the U.S. Department of Defense's AI project for missile and space threat detection [16] can be operationally deployed despite computing and power constraints. Concurrently, the timing of any easing of the “political caution” [11] within the PLA is also a variable. The future direction of the U.S.-China military AI gap will depend on which side can first resolve its bottlenecks and accelerate its pace of deployment.

The fifth variable is the expansion of computing infrastructure by third powers, including Europe. How the low 5% market share identified by Bruegel [6] evolves over the next five years will determine whether the AI infrastructure competition solidifies into a pure duopoly or is reshaped into a multipolar contest. For middle powers like South Korea, the interplay of these five variables will define the strategic parameters for securing their own AI sovereignty.