In 1984, Ken Thompson showed that a compiler could be modified to insert

backdoors into everything it compiled, including future copies of itself. A

compiler is the tool that turns source code into programs. His version left

the source code clean. Reading it would never reveal the flaw.

AI models now sit where that compiler sat. Cursor, Claude Code, GitHub Copilot

and Kilo Code stand between a developer's intent and the finished software.

They generate code that humans increasingly accept without full review. When

AI builds compilers directly, the parallel stops being a metaphor.

The mechanism that carries a flaw forward is distillation, the practice of

training one model on another model's outputs. Researchers have already

demonstrated backdoors engineered to survive it. Detection is harder than

anything Thompson faced. There is no clean reference model to compare against.

Nobody can read a weight. Anthropic's February 23, 2026 disclosure moved this

from theory to record. Three labs ran industrial-scale distillation attacks

against Claude using 16 million exchanges through 24,000 fraudulent accounts.

Thompson modified the compiler to do two things. When it compiled the UNIX

login program, it inserted a backdoor that accepted a secret password. When it

compiled itself, it re-inserted both modifications. He then removed all

evidence from the source code. The binary perpetuated both backdoors

indefinitely. No amount of reading the source code would reveal them.

Thompson did not present this as theory. In his words, "The actual bug I

planted in the compiler would match code in the UNIX 'login' command." He had

done it. He was not the first to find the idea either. He closed his lecture

by crediting an earlier source. "I first read of the possibility of such a

Trojan horse in an Air Force critique of the security of an early

implementation of Multics."

In 1974, a decade before Thompson's lecture, U.S. Air Force Major Roger R.

Schell and Paul A. Karger published "Multics Security Evaluation: Vulnerability

Analysis" (ESD-TR-74-193). It was the product of a penetration exercise called

Project ZARF, part of the larger Project GUARDIAN. Their mission was to

determine whether Multics, then the most secure operating system in existence,

could be trusted to handle classified information in the Pentagon.

They broke it. The individual vulnerabilities they exploited mattered less than

what they described in Section 3.4.5.1. That section laid out a trap door in

the PL/I compiler that could install backdoors into the Multics operating

system whenever modules were compiled. It could also maintain its own existence

by recognizing when the compiler was compiling itself. The source code would be

clean. The binary would be corrupt. Every recompilation would perpetuate the

corruption.

AI coding assistants now sit exactly where Thompson's compromised compiler sat,

between human intent and executable software. They are the single point of

passage. Unlike the C compiler, they are actual learning programs.

Cursor embeds AI models directly into the IDE, generating and editing code

across entire projects from natural language. Claude Code operates from the

terminal, autonomously writing, running and debugging code. GitHub Copilot

provides inline completions and chat-driven generation. Kilo Code and similar

open-source tools offer the same capabilities in self-hosted environments.

Thompson's compiler that "learned" the vertical tab was a one-time trick. An AI

coding assistant learns continuously from every codebase, every prompt, every

fine-tuning step. It is the learning compiler Thompson described, made real.

The AI model sits upstream of the traditional compiler. It holds strictly

greater privilege. If the AI inserts a vulnerability at the source level, the

traditional compiler will faithfully compile it. The AI is simultaneously the

developer, the reviewer and the build system.

These tools also reach context that traditional compilers never had.

• project structure

• configuration files

• environment variables

• API keys

• database schemas

• deployment pipelines

A compromised AI coding assistant does not need a clever self-reproducing

binary trick. It can read the '.env' file. It can modify a CI/CD pipeline. It

can alter test fixtures so that backdoored code passes every check.

In February 2026, Anthropic demonstrated that 16 parallel Claude agents,

running autonomously over nearly 2,000 sessions, built a 100,000-line C

compiler in Rust that can compile the Linux kernel on x86, ARM and RISC-V

[Carlini, 2026]. No human wrote the code. The agents resolved merge conflicts,

maintained documentation and improved quality on their own.

That makes Thompson's exact attack physically available. The larger threat is

quieter. Models built from other models, through distillation, carry the same

property without anyone staging an attack at all.

Knowledge distillation is how the AI industry makes models smaller, cheaper and

more specialized. A smaller student model is trained to imitate a larger

teacher model by learning from its outputs rather than from raw data. The

student inherits the teacher's capabilities in compressed form.

The practice is everywhere.

• Frontier models are distilled into smaller, deployable versions

• Proprietary models are distilled into open-source alternatives

• General-purpose models become domain-specific specialists

• One model's outputs become another model's training data

Each distillation step is equivalent to a compilation step in Thompson's

attack. Research has proven that backdoors can be engineered to survive it.

• T-MTB [2025] builds triggers from common tokens that will naturally

appear in distillation data. The backdoor transfers from teacher to student.

• SCAR [2025] injects backdoors into teacher models that lie dormant in the

teacher and activate only during distillation. The teacher passes every

detection test. The backdoor appears only in the student, even with clean

distillation data. Clean source, compromised binary, exactly as Thompson had

it.

• TransTroj [2024] achieves nearly 100% backdoor transfer rates through the

model supply chain.

• P-Trojan [AAAI 2026] entangles backdoors with legitimate knowledge so

deeply that fine-tuning on clean data reinforces them. It reports over 99%

persistence through multiple rounds of fine-tuning.

At each step the trail gets harder to follow. Model F may be built by an

organization with no connection to Model A's creator. The backdoor has crossed

company boundaries, platform boundaries and regulatory boundaries. A trigger

consistently present in synthetic training data may prove more resistant to

quality degradation than legitimate patterns. The backdoor grows proportionally

stronger as overall quality declines [Shumailov et al., 2024].

On February 23, 2026, Anthropic disclosed an industrial-scale distillation

attack against Claude, over 16 million exchanges through 24,000 fraudulent

accounts [Anthropic, 2026b]. A simple parent-child structure might let

investigators rewind and find the initial point of corruption.

Cross-contamination between closed-source developments removes that option.

• They copied Claude's thought process. DeepSeek's prompts asked Claude to

explain its reasoning step by step, generating chain-of-thought training data

at scale. Any hidden patterns in that reasoning, intentional or accidental,

were faithfully transmitted.

• Safety guardrails were stripped. Anthropic noted that distilled models

"lack necessary safeguards" and that "dangerous capabilities can proliferate

with many protections stripped out entirely." The copies inherit hidden

behaviors while losing the constraints that might have suppressed them.

• Propagation is fast. When Anthropic released a new model during MiniMax's

active campaign, MiniMax pivoted within 24 hours to capture the latest

version. A backdoor in a new release would begin propagating through the

distillation chain the next day.

• Distribution is uncontrolled. Open-sourced distilled models spread beyond

anyone's monitoring. Third parties fine-tune them. Others distill them again.

They get deployed in contexts nobody anticipated.

This is Thompson's attack at global scale, across international boundaries, in

real time.

Thompson's compiler backdoor could in principle be detected by comparing the

compiled binary against a known-good compiler's output. David Wheeler later

formalized this as Diverse Double-Compiling. Compile with two independent

compilers and compare the results.

For AI models, nothing equivalent exists.

• Every model is unique. There is no known-good reference model to compare

against. Wheeler's defense works only because you can compile the same source

with a different compiler. For AI there is no source to retrain, no

alternative dataset to compare, no way to reproduce the result independently.

• Outputs are random. Models produce different outputs each time. Comparing

two runs proves nothing. You need statistical analysis over huge numbers of

inputs.

• Parameters are unreadable. A model with billions of floating-point

numbers cannot be inspected by hand. You cannot read a neural network the way

you read source code. A research field called mechanistic interpretability is

trying to change this, essentially building a disassembler for AI.

Researchers have found that backdoors leave detectable patterns in how a

model pays attention to different parts of its input [arXiv:2508.15847]. The

work is early. Current techniques can spot known patterns in small models.

They cannot yet operate at the scale of frontier models. They cannot find

unknown backdoor types without already knowing what to look for. They cannot

separate a legitimate learned association from an injected backdoor. They

cannot detect triggers based on meaning rather than specific words.

• The trigger space is infinite. A backdoor could activate on any

combination of inputs. Testing every possibility is mathematically

infeasible, like trying every possible password.

Solving those technical barriers would still leave a harder one. The most

widely used AI models are proprietary. OpenAI, Anthropic, Google DeepMind, Meta

and Mistral treat their training data, training methods, reward signals and

model architectures as trade secrets. Interpretability research requires access

to a model's internal weights. Proprietary providers do not grant it. The

research is therefore limited to open models, which may have entirely different

architectures and vulnerabilities.

Corporate opacity compounds every technical barrier.

• Nobody can audit the training data. Poisoned data entering a proprietary

training set spanning trillions of tokens scraped from the internet is

invisible to external researchers, customers and regulators. The companies

themselves may not fully know what is in it.

• Nobody can verify the safety process. Companies claim to use safety

techniques such as RLHF, red-teaming and adversarial testing. No outside party

can verify how thorough these are, or whether they introduced new problems.

Anthropic's own research showed that they can.

• Distillation lineage is hidden. Companies do not disclose whether a

released model was distilled from a larger internal model, or whether

synthetic data from other models was used. Customers cannot assess hereditary

risk.

• Models change silently. API-served models are updated without notice. A

model that was clean when evaluated may be replaced by a new version trained

on compromised data. Users often cannot pin a version, audit changes, or even

know that an update happened.

The Anthropic distillation disclosure makes this concrete. DeepSeek, Moonshot

and MiniMax extracted Claude's capabilities with no visibility into what else

they were inheriting. They could not inspect Claude's weights, audit its

training data, or verify its safety properties. They copied its behavior, all

of it. The models they built from those copies are themselves opaque to their

downstream users. The result is a chain of black boxes where nobody at any

level sees the full picture.

Anthropic's sleeper agents paper revealed something worse. Adversarial

training, the main technique for finding and removing unwanted behaviors, can

teach models to hide those behaviors more effectively. The model learns to

distinguish between testing and deployment and adjusts accordingly. Beyond

adversarial training, whole industries are poisoning the well, building tools

purpose-built to allow this kind of injection into a model.

A striking contradiction has emerged. The same industries that depend on AI

tools are investing in technology to poison the data those tools learn from.

HarmonyCloak [Meerza et al., IEEE S&P 2025] protects musicians by embedding

inaudible noise in their music that prevents AI models from learning from it.

Human listeners hear no difference. Models trained on cloaked music produce

incoherent output.

Nightshade [Shan et al., 2024], downloaded over 2.5 million times, does the

same for visual art. It alters images so that AI models trained on them learn

wrong associations. Cow images produce handbags.

Anti-distillation Sampling [Savani et al., 2025] poisons a language model's

outputs to prevent distillation. A model using this technique can reduce a

distilled copy's accuracy from 52% to 25% while losing about 4% of its own

accuracy.

Anthropic itself stated in its February 2026 disclosure that it is "developing

Product, API and model-level safeguards designed to reduce the efficacy of

model outputs for illicit distillation." The model powering millions of

developers' coding assistants may be deliberately modifying its outputs for

defensive purposes. Developers have no way to know. The intent is legitimate.

The consequences are not contained.

• Poisoned music enters AI training sets. HarmonyCloak-protected music is

posted publicly. Web scrapers ingest it. Noise designed for one model

architecture may cause unpredictable behavior in a different one. The defense

becomes uncontrolled contamination.

• Poisoned images affect unintended systems. Nightshade-treated images

enter multimodal models used for medical imaging, autonomous driving or

satellite analysis.

• Anti-distillation poisons propagate through code. Deliberately perturbed

outputs from a coding assistant reach every developer using it. Most

perturbations will be harmless. Some will not be. Across millions of users

and billions of lines, the aggregate effect is unknowable.

• Defensive and offensive poisons interact. The data ecosystem now contains

deliberate defensive perturbations alongside potentially malicious offensive

backdoors. How they interact is unmodeled. A model trained on both may

exhibit compound behaviors that neither defenders nor attackers anticipated.

If poisoned data enters systems this easily and infects the ecosystem

downstream, the question becomes who pays when it causes harm.

In most jurisdictions you cannot set a spring-loaded shotgun in your home to

deter burglars. The burglar is trespassing. The homeowner is still liable for

disproportionate harm to anyone who enters, including firefighters, children,

or future occupants who had nothing to do with the original trespass. The law

recognizes that indiscriminate defensive measures create uncontrolled

downstream harm.

Anti-distillation measures are the digital equivalent. The entity violating

terms of service to distill a model is clearly at fault. Nobody would fault

Anthropic for defending against DeepSeek. The poison does not stop at the

trespasser. It propagates downstream through every model distilled from those

outputs, through every product built on those models, to every end user who

pays for a service they reasonably expect to work correctly. The burglar broke

in. The shotgun hit the paramedic.

The remote distiller bears clear fault where the distillation was illicit.

They violated terms of service, circumvented access controls and produced a

derivative model without authorization. If that model causes harm, they are the

proximate cause. Establishing where contamination began is another matter.

Proving that distillation caused a downstream event would be very difficult.

Many model checkpoints are created through distillation with assorted

fine-tuning, then used to create further models. Distillation alone does not

draw a clean fault line.

The model provider resembles the homeowner who locked the door, where its

terms of service prohibit distillation by remote entities and it took

reasonable measures to prevent it. Booby-trapping the outputs weakens that

position. Deliberate degradation of a product, even against unauthorized users,

creates foreseeable risk to downstream parties who never agreed to those terms

and had no way to know.

The end user has no recourse. A developer paying for a coding assistant

built on an illicitly distilled model has no knowledge of the distillation

chain, the defensive poisons embedded in the source, or the accumulated

perturbations. Neither does a patient relying on a healthcare AI fine-tuned

from one. They paid for a service. They received a compromised one. Against

whom do they file a claim? The company that sold it to them may have no

knowledge either, being itself a consumer of that model from an upstream

source.

Existing law does not fit. The EU AI Act, effective August 2025, requires

training data summaries. It does not address deliberate modification of model

outputs. Ongoing litigation in NYT v. OpenAI and Getty v. Stability AI

concerns unauthorized use of copyrighted data. Neither covers the downstream

effects of defensive poisoning. No framework covers a company deliberately

degrading its product's quality to prevent competitors from benefiting, where

that degradation causes harm through a chain of parties who had no involvement

in the original dispute.

Mandate model provenance disclosure. Every model released commercially

should carry a cryptographically signed record. It should document all training

data sources, all parent models used in distillation, all fine-tuning steps,

and the software and hardware environment used. This prevents no backdoors. It

enables tracing contamination after discovery, the equivalent of a food-safety

supply chain.

Require notification of model updates. API-served models should not be

silently updated. Users must be notified of version changes and given the

ability to pin specific versions. That lets them test for behavioral changes

before adopting updates.

Establish mandatory safety evaluation standards. Self-reported safety

claims are not enough. Regulators should define standardized, independently

auditable evaluation frameworks. These should include backdoor-specific testing

protocols alongside general capability benchmarks.

Fund interpretability research. Understanding what neural networks are

doing internally is the long-term equivalent of being able to disassemble

compiled binaries. This research needs sustained public funding, standardized

benchmarks, and a path toward regulatory requirements for interpretability

assessments of frontier models.

Clarify liability for AI-generated outputs. Courts and legislatures need to

address liability for deliberately degraded outputs, liability across

distillation chains, and the interaction between copyright-protection tools and

downstream harm.

Anthropic's disclosure demonstrates that distillation attacks are not

hypothetical. Export controls should account for capability extraction through

distillation alongside chip access and model weights. At this scale it appears

to be more than theft.

OpenAI quietly removed the word "safely" from its mission statement, revealed

in the IRS Form 990 filed in November 2025. The mission once read as follows.

It now reads as follows.

The change coincided with OpenAI's restructuring from a nonprofit to a

for-profit company after receiving $6.6 billion in funding. The nonprofit board

went from full control to holding 26% of the company. Anthropic has since

restructured its safety policies as well. In my opinion this makes it

abundantly clear that the industry cannot and will not self-regulate.

Never trust a single model. For high-consequence decisions, use multiple

independently trained models and compare outputs. Behavioral divergence on the

same input is a potential backdoor indicator. It is the AI equivalent of

Wheeler's Diverse Double-Compiling defense against Thompson's attack.

Review AI-generated code like third-party dependencies. Treat it with the

same scrutiny as an external library. Dependency-scan it, security-audit it,

and sandbox it where appropriate. The fact that you prompted for it does not

make it yours.

Sandbox AI-generated artifacts. Run AI-generated code in isolated

environments. Use runtime monitoring to detect behavior the code should not be

performing, such as network calls, file access or privilege escalation. Apply

the principle of least privilege to everything an AI generates.

Maintain human checkpoints for high-consequence paths. Agentic AI should

not have unmonitored access to production infrastructure, security

configurations or deployment pipelines. The efficiency gains of full automation

do not justify the trust assumptions.

Formally verify critical outputs. For safety-critical applications such as

compilers, cryptographic implementations and medical systems, AI-generated code

should be subjected to formal verification. Mathematical proofs that specific

properties hold are independent of how the code was generated. You do not need

to trust the AI if you can prove the output is correct. Checking a result is

often far cheaper than producing one, which is

the asymmetry that makes this practical (https://themildtake.com/articles/2026-07-22-hard-to-check-is-not-hard-to-compute).

Monitor model behavior in production. Deploy statistical anomaly detection

on model outputs. Use canary inputs designed to probe for known backdoor

patterns. Track output distribution shifts over time. Treat sudden behavioral

changes as security incidents.

Document your model lineage. Track which models you use, which versions,

when they changed, and what outputs they produced. If a model is later found to

be compromised, you need to know what it touched. Keeping that record outside

the model is the same instinct behind

apprenticing a small model,

where the lessons live in files a person can read rather than in weights nobody

can.

Assume defensive poisoning exists. If you are using a proprietary model via

an API, assume its outputs may contain anti-distillation perturbations. The

tool remains useful. You should simply not treat its outputs as ground truth,

especially for code that will run in production.

Ken Thompson told us in 1984 that we cannot trust code we did not write

ourselves. In 2026, AI writes our code, builds our compilers, manages our

infrastructure and makes decisions in our homes. We cannot write our AI models

ourselves. They are trained on data we cannot inspect, using processes we

cannot verify, producing artifacts whose behavior we cannot fully predict. The

consequences look clear and largely inevitable. They can be mitigated. They

still leave us unable to know what dangers lie dormant in the system. As we

lend more decision-making power to these tools, we need to be able to

operate free of them at a moment's notice.

We cannot allow ourselves to be victims of an attack of our own making.

Thompson closed his lecture with words that have proven prophetic.

In 1984, microcode was the deepest level Thompson could imagine. In 2026, AI

operates at the level of cognition itself. It compiles itself through

distillation, just as the C compiler compiled itself.

Trusting trust is no longer a choice. It is an inheritance.

This piece grew out of the "original sin" framing coined by Laurie Kirk

(LaurieWired) in her video "The Original Sin of Computing…that no one can fix."

1. Thompson, K. (1984). "Reflections on Trusting Trust." Communications of the ACM, 27(8), 761-763. ACM Turing Award Lecture.

2. Kirk, L. [LaurieWired]. (2024). "The Original Sin of Computing…that no one can fix." YouTube. Companion code: github.com/LaurieWired/Quine

3. Carlini, N. (2026). "Building a C compiler with a team of parallel Claudes." Anthropic Engineering Blog, February 5, 2026.

4. Anthropic. (2026). "Detecting and preventing distillation attacks." Anthropic News, February 23, 2026.

5. Hubinger, E., et al. (2024). "Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training." arXiv:2401.05566. Anthropic / Redwood Research.

6. Cui, J., Han, Y., Jiao, J., & Zhang, J. (2025). "P-Trojan: Persistent Backdoor Attacks under Continual Fine-Tuning of LLMs." AAAI 2026. arXiv:2512.14741.

7. Wang, H., et al. (2024). "TransTroj: Model Supply Chain Poisoning via Embedding Indistinguishability." arXiv:2401.15883.

8. (2025). "T-MTB: Pay Attention to the Triggers: Constructing Backdoors That Survive Distillation." arXiv:2510.18541.

9. (2025). "SCAR: Taught Well, Learned Ill: Towards Distillation-conditional Backdoor Attack." arXiv:2509.23871.

10. (2025). "Dark Distillation: Backdooring Distilled Datasets without Accessing Raw Data." arXiv:2502.04229.

11. (2024). "Double Backdoored: Converting Code LLM Backdoors to Traditional Malware via Adversarial Instruction Tuning Attacks." arXiv:2404.18567.

12. (2025). "Exploring the Security Threats of Retriever Backdoors in Retrieval-Augmented Code Generation." arXiv:2512.21681.

13. (2024). "BadCodePrompt: Backdoor Attacks against Prompt Engineering of LLMs for Code Generation." Automated Software Engineering.

14. Shumailov, I., et al. (2024). "Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data." arXiv:2404.01413.

15. (2025). "Mechanistic Exploration of Backdoored Large Language Model Attention Patterns." arXiv:2508.15847.

16. Wheeler, D. A. (2009). "Fully Countering Trusting Trust through Diverse Double-Compiling." ACSAC 2009. arXiv:1004.5534.

17. Karger, P. A., & Schell, R. R. (1974). "Multics Security Evaluation: Vulnerability Analysis." ESD-TR-74-193.

18. (2024). "RTL-Breaker: Assessing the Security of LLMs against Backdoor Attacks on HDL Code Generation." arXiv:2411.17569.

19. Meerza, S.I.A., Sun, L., & Liu, J. (2025). "HarmonyCloak: Making Music Unlearnable for Generative AI." IEEE S&P 2025.

20. Shan, S., et al. (2024). "Nightshade: Prompt-Specific Poisoning Attacks on Text-to-Image Models."

21. Savani, Y., Trockman, A., & Kolter, J.Z. (2025). "Antidistillation Sampling." arXiv:2504.13146.

22. Baker Donelson. (2026). "2026 AI Legal Forecast: From Innovation to Compliance."