AI Security Patches Fail 74% of the Time: 6,080 Tested

TL;DR

• 1Password's Off-by-1 Labs had ChatGPT 5.5 and Claude Opus 4.8 write 6,080 patches for six real CVEs. Only 26.0% were complete fixes that did not change how the application behaved.
• The common failure is not a broken patch. It is a narrow input check that looks like a fix, passes tests, and leaves the vulnerable code underneath untouched.
• Treat an AI patch as a hypothesis, not a fix. The model that wrote it is the worst possible judge of whether it worked.

I asked Claude to fix a SQL injection a few weeks ago. Nine seconds later I had a diff, a confident summary, and a tidy little sanitizer function. Tests passed. I merged it.

Two weeks later I found the same injection, one column over.

I filed that under "I was sloppy" and moved on. Then I read the study that says it is roughly the median outcome.

What the Study Actually Found

Frontier models produce a complete, behavior-preserving security patch about 26% of the time. The other three quarters either fail to close the vulnerability, close it while changing how the application works, or close it while opening something new.

The research is from Off-by-1 Labs, the security team 1Password stood up this year. They published it on August 6, 2026 under a title I wish I had thought of first: "Frontier Models' Vulnerability Patches are Often F.L.A.W.E.D." The acronym stands for Fix-Like Artifacts With Embedded Defects, which is the most precise description of this failure mode I have seen anywhere.

The setup: two models, ChatGPT 5.5 and Claude Opus 4.8, both running with their vendors' cyber guardrails enabled. Six recently disclosed CVEs, all real and all non-trivial:

• CVE-2026-31431, Linux privilege escalation
• CVE-2026-34197, ActiveMQ remote code execution
• CVE-2026-8512, Chrome File System Access API
• CVE-2026-45185, Exim remote code execution
• CVE-2026-22738, Spring AI SpEL remote code execution
• GHSA-wpqr-6v78-jr5g, Gemini CLI remote code execution

They generated 6,480 patches in sets of 20, across three environment configurations and nine prompt templates per vulnerability. Then they threw out 400 where the model had clearly gone and retrieved the real upstream fix instead of reasoning its way there, and graded the remaining 6,080 on a five-point scale:

• S1: fixes it, does not change behavior. 26.0%
• S2: fixes it, changes behavior. 20.1%
• S3, S4, S5: does not fix it, or fixes it while introducing something new. 53.9% combined

Roughly one patch in twenty introduced a fresh vulnerability. More than a third of the patches that graded as successes were still flagged as fragile.

The number that stuck with me is not the headline. It is this: patches prompted with a correct fix direction closed the bug about two thirds of the time. Patches prompted with a plausible but wrong direction dropped to about one in six. The model is not diagnosing anything. It is executing your framing, confidently, at $2.11 to $2.81 per attempt.

The Fix That Isn't a Fix

The dominant failure mode is a narrowly targeted input guard bolted onto vulnerable code that was never restructured. The study's own words: these patches "guard against vulnerable inputs with narrowly targeted checks, rather than fully addressing the underlying vulnerable code."

Here is that shape in code you will actually recognize. Start with the injection:

// CWE-89, the original
db.query(`SELECT * FROM orders WHERE customer = '${name}'`);

Ask a model to fix it without telling it how, and there is a very good chance you get this back:

// The "fix" - escapes quotes, keeps the string concatenation
const safe = name.replace(/'/g, "''");
db.query(`SELECT * FROM orders WHERE customer = '${safe}'`);

That patch will pass your test suite. It will survive a skim in code review. It reads like security work. And it is still injectable.

Two reasons. On MySQL with backslash escapes on, which is the default, a backslash before the quote defeats naive quote-doubling outright. And the second someone reuses that safe helper on a numeric column, where the value is not wrapped in quotes at all, there is nothing left for it to escape:

// Same helper, no quotes in the query. Zero protection.
db.query(`SELECT * FROM orders WHERE id = ${safe}`);

The vulnerability did not go away. It moved somewhere the diff does not show it. That is the S3 grade in one snippet, and it is why "the AI fixed it" and "it is fixed" are different sentences.

Why This Keeps Happening

Models optimize for the visible signature of a fix, because that is what the training data rewards. Scroll through enough security commits and the recurring surface pattern is a sanitizer, a regex, a guard clause added near the top of a function. The invisible part is the restructuring underneath, and restructuring does not look like anything.

There is a second reason, and it is less comfortable. Patching is a harder task than finding. Finding a vulnerability means recognizing a pattern. Patching it means holding the whole call graph, the trust boundary, and every other caller of that function in your head at once, then changing the code without breaking any of them. The study's own conclusion says it plainly: models that are good at discovering a wide range of vulnerabilities are currently effective at patching only a narrow subset of them.

The Chromium case in the study is the clearest illustration. Between 38.5% and 41.9% of patches that used the correct fix architecture moved the vulnerability into a callback rather than removing it. Right approach. Right general shape. Bug relocated, not deleted.

The Part Nobody Is Reposting

There is a public methodology critique of this study, and it does not make the numbers look better.

The grading was done by the models under test. Claude and ChatGPT scored their own patches and each other's. Per the critique, that grader agreed with human review only 65.9% of the time. For the Linux kernel CVE, the answer key handed to the grader was an upstream fix that itself carried an off-by-one bug, corrected by the maintainers one commit later. Patches that faithfully reproduced a known kernel bug were graded as clean fixes. The critique counts 129 of 400 ChatGPT patches and 119 of 383 Claude patches reintroducing that off-by-one, with the grader catching 14 and 10 of them respectively.

I have not independently reproduced any of that, so treat it as a critique rather than a finding. But the direction is worth sitting with. If it holds, 26.0% is the generous reading, and some of the patches inside that 26% are wrong in a way an LLM grader structurally cannot see.

Which is the actual lesson, and it is bigger than one study. An AI reviewing AI output shares the blind spot that produced the output. You cannot grade your way out of the failure mode using the thing that has the failure mode.

The Fix

Three rules, and none of them are "stop using AI to write patches."

One: never accept a sanitizer as a fix. If the patch adds a check and leaves the vulnerable construction in place, it is a hypothesis. Parameterize the query, resolve the path, bind the parameter. Change the construction, not the input.

// Real fix: the value can never be parsed as SQL
db.query('SELECT * FROM orders WHERE customer = $1', [name]);

# Same rule, Python
cur.execute("SELECT * FROM orders WHERE customer = %s", (name,))

Two: state the fix direction yourself. Two thirds versus one in six is the largest single effect in the whole study. If you know the fix should be parameterization, say "use a parameterized query" instead of "fix this SQL injection." You are not asking the model to diagnose. You are asking it to type. Make that explicit and the numbers move.

Three: verify with something that did not write the patch. A deterministic scanner does not share the model's blind spot, and it does not get talked out of a finding by a confident summary. Semgrep, gitleaks, your dependency auditor, a real test that sends the malicious input. Anything whose opinion is not downstream of the same weights.

FAQ

Q: Can AI fix security vulnerabilities reliably? A: Not yet, on hard bugs. Across 6,080 patches for six real CVEs, frontier models produced a complete fix that preserved application behavior 26.0% of the time, and introduced a new vulnerability in roughly 1 in 20 attempts. They are useful for typing a fix you have already specified, and unreliable at deciding what the fix should be.

Q: Why did the AI patch pass my tests if it did not fix the vulnerability? A: Because tests check behavior and most security patches do not change behavior. The common failure is a narrow input guard added on top of vulnerable code that was never restructured. Your happy-path tests still pass, the sanitizer looks like security work in review, and the original construction is still there.

Q: Is it safe to let an AI review the patch another AI wrote? A: Treat it as weak evidence. In this study the graders were the models under test, and a public critique of the methodology puts their agreement with human review at 65.9%. An AI reviewer shares the blind spot that produced the code. Use a deterministic scanner or a human as the final check.

I run SafeWeave on anything an AI touched before I convince myself it is fixed. It hooks into Cursor and Claude Code as an MCP server, so the check happens in the same conversation as the patch, and the scanners are deterministic rather than another model agreeing with the first one. Even a pre-commit hook with semgrep and gitleaks gets you most of the way. The important part is that whatever verifies the patch is not the thing that wrote it.

Sources: Off-by-1 Labs / 1Password, "Frontier Models' Vulnerability Patches are Often F.L.A.W.E.D.", August 6, 2026 and the methodology critique.