New Research Uncovers Hidden AI Reasoning Extraction, Raises Alarms Over Unauthorized Distillation and Data Leaks

A team of computer scientists has developed a new method to extract the hidden internal "thinking" that cutting-edge frontier AI models use when solving complex problems. Their work provides suggestive (though not definitive) evidence that some open-weight Chinese AI models may have been trained by "distilling" proprietary reasoning data from closed U.S. models, due to the near-perfect match of some core reasoning patterns between the systems.

Beyond stealing model capabilities, the research also confirms that this method can be used to recover sensitive personal information, including passwords and API keys, from a model’s hidden reasoning process. While this specific data leakage vulnerability has already been patched by major providers, core risks remain unaddressed.

“All major frontier model providers we tested share this vulnerability,” said Alexander Panfilov, a lead researcher on the project and computer scientist at Germany’s University of Tübingen. “It can lead to personal information leakage, and it enables large-scale reasoning distillation attacks that steal proprietary AI capabilities.”

Panfilov collaborated with researchers from the University of Tübingen, the Max Planck Institute, AI safety organization MATS Research, and cybersecurity firm Snyk. The team identified the flaw in leading frontier models from OpenAI, Anthropic, and Google, all of which are accessed via public application programming interfaces (APIs).

In their paper detailing the work, the team documents that Kimi K3, an open-weight (publicly downloadable) model from Chinese AI developer Moonshot AI, produces outputs that are strikingly similar to the hidden reasoning traces (step-by-step problem-solving paths) of Anthropic’s Claude Opus 4.8 and OpenAI’s GPT 5.6 Sol for a wide range of test prompts. Despite the close match, the researchers explicitly note their work “cannot causally establish distillation,” meaning they cannot confirm Kimi K3 was definitely trained by copying U.S. model reasoning. Two other open-weight models tested—China’s DeepSeek and Inkling from U.S. firm Thinking Machines—did not show this level of reasoning similarity to Claude Opus. As of publication, Moonshot AI and Z.ai had not responded to requests for comment.

Model distillation is a longstanding, widely used technique that lets developers efficiently transfer capabilities from existing mature models into new smaller models, and it is especially common for building open-weight AI systems. In recent months, however, distillation has become a deeply controversial geopolitical issue, amid repeated claims that Chinese AI firms use the practice to effectively copy the capabilities of top U.S. models. In February, OpenAI told U.S. lawmakers that DeepSeek appeared to have copied one of its models to build its popular reasoning model R1. In June, Anthropic testified that Alibaba had systematically distilled its models to develop China’s Qwen model line. The new research does not confirm Chinese firms used this specific hidden reasoning extraction method to distill closed U.S. models, but it proves such an attack is far more feasible than previously assumed.

The "Mini-Me Model" Exploit

Advanced AI solves difficult problems by breaking them into sequential steps, a process called chain-of-thought reasoning. Most leading AI companies keep a model’s internal reasoning private specifically to prevent competitors from stealing that proprietary knowledge to train new competing models.

To reduce computational costs for users, however, companies typically send an encrypted version of a model’s full internal reasoning to a user’s device when processing queries via API. The researchers’ exploit leverages another common industry practice: most AI providers offer multiple sizes of their models, from large, highly capable (but expensive) flagship models, to smaller, cheaper, less powerful variants that users can select for low-stakes tasks to cut costs.

Panfilov’s team found that feeding encrypted reasoning traces from a large flagship model into a smaller variant of the same model will reveal the full hidden original reasoning. Smaller models receive far less alignment training (the process that teaches models to follow safety and access rules), so unlike larger flagship models, they rarely refuse to expose their internal computations.

“The idea of swapping queries to a weaker model variant which has the same decryption key but weaker alignment is very clever,” said Florian Tramer, a computer security researcher at ETH Zürich not involved in the work. “It’s definitely becoming a major issue.”

The same method also extracts other sensitive data, including API keys and passwords embedded in reasoning traces stored on a user’s device. The researchers notified OpenAI, Anthropic, and Google of the vulnerability last month, and all three companies have updated their APIs to block personal data extraction via this method. While private data can no longer be stolen this way, Panfilov says some hidden reasoning traces can still be uncovered using the same core approach. Fully eliminating the distillation risk would require a fundamental overhaul of how these companies’ APIs operate, he added.

“We value independent research on our models and have begun building short-term mitigations for the replay behaviors described in the report,” said Michael Aciman, a spokesperson for Anthropic. He added that the research did not involve recovering encryption keys, accessing Anthropic’s infrastructure, or pulling personal data from Anthropic’s own systems. Spokespeople for Google and OpenAi declined to comment on the findings.

Geopolitical Debate Over Distillation Practice

As U.S. and Chinese companies compete for global AI supremacy with increasingly powerful models, distillation has emerged as a key geopolitical flashpoint in recent months. U.S. hardliners argue China gains a major strategic advantage by distilling U.S. AI technology to build low-cost open-weight models that can be widely deployed. But industry leaders and many researchers counter that distillation is a standard open ecosystem practice that accelerates AI progress globally.

This week, Meta CEO Mark Zuckerberg wrote in a blog post that distillation “is an important principle of how the open source ecosystem works,” and warned that restricting the practice would put the U.S. at a long-term competitive disadvantage.

Kyle Miller, a researcher at the Center for Security and Emerging Technologies, a U.S. tech policy think tank, says it remains unclear how much of an advantage distillation actually gives Chinese AI developers. The technique only boosts existing model capabilities by a limited degree, and Chinese firms already have the expertise to build cutting-edge models fully from scratch if needed, he noted.

“Nobody here in the US knows how much distillation is benefiting the Chinese labs,” Miller said. “If you removed the ability for Chinese labs to distill, it's my view that it wouldn't dramatically change the competitive landscape.”

To test for hidden distillation in open-weight models, the researchers fed 90 test prompts to all the models in their study. When they gave open-weight models the first few steps of hidden reasoning traces captured from closed proprietary models, some open models went on to generate nearly identical full reasoning and final answers. This match was far more pronounced for Kimi K3 than any other model tested, the team found.

Yarin Gal, a computer scientist at the University of Oxford, says distillation is not only widely used across the global industry, it has also helped AI advance far more quickly than it would otherwise. “If it's the norm that everyone blocks everyone [from doing distillation], then that also will have implications on the rate of progress,” he said.

Even as companies and policymakers consider new rules to limit unauthorized distillation, Panfilov and his team note that AI models will likely continue to expose their hidden internal reasoning in unexpected new ways.