More than 40 years ago, I was sitting in the office of one of the science and technology assessment directorates of a well-known three-letter agency.
The conversation then was about how what we now call Artificial Intelligence (AI) could be a force multiplier in both weapons design and intelligence analysis.
J-36 Fighter Artist Rendition. X Screenshot.
J-36 Fighter from China. Image Credit: Creative Commons.
But even then, with computer processing power at a minuscule, laughable level compared to today, there were still realistic views of its limitations.
Placing an inordinate level of dependence on this technology was not only unrealistic, said one analyst present, but it was also being used as an unrealistic catch-all to bypass the day’s technological bottlenecks.
Speaking of one particular weapons program, the same analyst told me that “every time we come up against a shortcoming in the design requirement due to limitations in computer processing power, the military guys pull out the same wild card,” he said. “What they always say is ‘don’t worry, AI will take care of that problem’ as if it is a cure for anything.”
Decades and endless iterations of computer processing technology later, weapon system designers in the People’s Republic of China (PRC) are – albeit for different reasons – coming to the same conclusions about the pitfalls of being overly reliant on AI in their own work.
Military AI Hallucinations
A report recently in the Hong Kong English-language South China Morning Post quotes the design team for the Chengdu 6th-generation J-36 fighter aircraft as saying that, when applied to weapon systems modeling and design analysis, there is a distinct possibility that the analyst could suffer from “AI Hallucinations.”
Large language models, the article begins, are “experts in intelligence work, able to quickly mine key data and analyze weapons performance – tasks that once took analysts huge amounts of time and effort.”
Engineers at the Chengdu Aircraft Research and Design Institute (CADI), the design bureau for the Chengdu Aircraft Industry Group (CAGI), are urging caution about relying too heavily on AI as a design or analysis tool. Both entities are component enterprises under AVIC, the Aviation Industry Corporation of China conglomerate.
This design team was responsible for developing the People’s Liberation Army Air Force’s (PLAAF) workhorse 5th-generation J-20 fighter, as well as the next-generation J-36 stealth fighter, and is cautioning its peers to be aware of AI’s drawbacks.
In a paper originally published on 20 June, engineer Zhang Xianzhe said that “AI models can invent aircraft specifications – length, payload, weapons, speed, combat radius – and calculate radar scan ranges and frequency bands wrong” in the process.
“The so-called hallucination effect, where models produce plausible but false outputs, could have severe consequences in the high-stakes, low-error world of defense intelligence, possibly even causing strategic miscalculations,” he said.
Zhang’s article was published in the Chinese journal Information Studies: Theory & Application. The publication itself is owned and published by the PRC’s state-owned China North Industries Corporation (NORINCO), which is one of the PRC’s largest defense industrial conglomerates.
Design Process and Assessment of Requirements
According to the paper, “every fighter jet begins with a deep understanding of the enemy and the battlefield. Designers must answer critical questions such as: what is the opponent’s radar coverage and which frequencies do they use?”
They must also take into account the boundaries of the “no-escape zone” of their air-to-air missiles, the frequency bands on which their electronic warfare systems operate effectively, the coverage of the battlespace by their AEW&C aircraft, and how ground-based air defense networks are configured.
“These factors determine stealth design, radar power, electronic warfare architecture, maneuverability, range and combat radius. In short, intelligence gathering was the very first step in building a fighter,” Zhang wrote.
But he warned that “since large language models are rarely trained on authoritative military sources, they often generate content that defies basic physics, design principles, or operational constraints.”
This could include “specifying materials that exceed fatigue limits, proposing flight maneuvers that violate aerodynamics, or drawing up mission plans beyond an aircraft’s actual range.”
Zhang said that when asked to assess defense technology, AI could generate answers that looked accurate but were factually incorrect – not just aircraft parameters, but also radar scan ranges, frequency bands, and even it could even generate entirely fictitious military bases, units, or exercise movements.
“The most direct harm comes in intelligence analysis and simulation, where acting on such hallucinations can lead to false assessments of combat capability,” he added.
Real-world incidents have already occurred, he said. In March 2026, a US strike on Iran hit a school, killing more than 175 children. This tragedy, he claims, was the result of databases not being kept up to date. According to some reports and expert analysis, blame was eventually assigned to the AI system that listed the building as a high-priority military target.
According to US officials and various reports, human error drove the final decision, but the speed and volume of AI-processed intelligence can be impossible to monitor.
This raises serious questions about whether frontline analysts can thoroughly vet every AI-recommended target in time.
Zhang’s suggested safeguards to reduce AI hallucinations are: feed models with validated defense data, build searchable databases for reference, write clear and specific prompts, and cross-check results by having AI bots debate the findings to weed out inaccurate assessments.
About the Author: Reuben F. Johnson
Reuben F. Johnson has thirty-six years of experience analyzing and reporting on foreign weapons systems, defense technologies, and international arms export policy. Johnson is the Director of Research at the Casimir Pulaski Foundation. He is also a survivor of the Russian invasion of Ukraine in February 2022. He worked for years in the American defense industry as a foreign technology analyst and later as a consultant for the U.S. Department of Defense, the Departments of the Navy and Air Force, and the governments of the United Kingdom and Australia. In 2022-2023, he won two awards in a row for his defense reporting. He holds a bachelor’s degree from DePauw University and a master’s degree from Miami University in Ohio, specializing in Soviet and Russian studies. He lives in Warsaw.
