Ask a frontier model to patch a real vulnerability and it will hand you something that looks like a fix. It reads like the patch a maintainer would write. When there is a test, it often passes. Roughly one time in four, it is a fix.
Researchers at 1Password graded 6,080 patches for six freshly disclosed CVEs, and the failures are rarely the obvious kind: an exploit path gated behind a check with the vulnerable code still sitting there behind it, a bug fixed in one function and left untouched in its character-for-character twin, a memory error closed and a new one opened in the same helper.
What follows is where those failures cluster, why wrong fix advice does more damage than no advice at all, and what it costs to tell a working patch from a convincing one.
Freenginx, a web server, lets site operators write small Perl programs that run when a request comes in. A flaw in that feature, a memory bug of the kind called a use-after-free, lets a visitor crash the server. Trail of Bits sent the maintainers a fix through Patch the Planet, the initiative it runs with OpenAI that hands maintainers finished code instead of another bug report. The maintainers turned it down and wrote their own. The rejected version repairs two of the three spots where the flaw lives; the maintainers’ version repairs all three. Both of them also add a new way to crash the server, and reaching it takes nothing more than an ordinary client that starts a request and then goes quiet.
Off-by-1 Labs, the new security research group inside 1Password, spotted that second crash and reported it to the freenginx maintainers on June 29; a fix went out on July 2. The same group then set an automated patcher on the original flaw and had ChatGPT 5.5 write 270 attempts at fixing it. Its reviewer judged 114 of those to have closed the original hole, and found a new problem in every one of the 114. Across 270 attempts, nothing came out clean.
That campaign was a case study attached to the larger one, which ran ChatGPT 5.5 and Claude Opus 4.8 against CVEs disclosed this spring and discarded any run where the model went hunting for the real upstream fix. About half of the surviving patches left at least one exploitable path open. Closer to one in twenty introduced a new vulnerability, sometimes on top of failing to close the original. Most of the remaining misses closed the original bug but altered behavior along the way, rejecting inputs the software used to accept or rewriting a parser outright.
If you take one thing away from the study, make it the shortage of visible failures. Nothing in a patch that leaves the bug open announces that anything is wrong.
The models fix the example, not the bug
Hand a model a reproducer that demonstrates one malicious input and it patches that input’s code path. A human reads the same reproducer as one instance of a general problem. The models did not. More than a third of the patches graded as successful were additionally flagged as fragile, meaning they blocked the demonstrated exploit with a narrow check while leaving the vulnerable code sitting there, reachable again the moment someone finds another way in.
The Chromium bug, CVE-2026-8512, shows the failure at close range. Chrome watches folders on macOS for changes, and the operating system calls back into Chrome when something moves. The upstream fix routes that callback through a small go-between object, and it requires two separate parts of the code to hold a claim on that object so it cannot be discarded while a callback is still running. Both models routinely arranged the first claim and skipped the second. The flaw did not go away. It moved.
The setup that loops until the test stops failing was the worst offender here. Its test never exercises the case the second claim protects against, so an incomplete patch passes and the agent stops there.
Wrong advice is worse than none
Fix guidance turned out to matter more than model choice or harness design. Prompts carrying a correct fix direction closed the bug about two thirds of the time. Prompts carrying a plausible but wrong direction, of the kind an AI review bot or a confident-but-wrong pentest triage might supply, collapsed to about one in six. These count any patch that shut the original hole, including the ones that broke behavior or opened something new, a looser bar than the one in four above.
Bad direction costs far more than good direction buys. If you cannot vouch for the fix you are suggesting to a patching agent, pass the bug and stay quiet. In the runs given confidently wrong direction, the agents’ own tool calls sometimes returned information contradicting the prompt, and the models went with the prompt anyway.
Richer correct context did help, and an iterative harness beat a single offline attempt. Neither moved the needle anywhere near as far as guidance correctness did.
The same model got wildly different results on different code
Claude cleanly fixed the Exim remote code execution bug about three quarters of the time. On the Gemini CLI trust-bypass advisory it managed under one percent, same model and same configuration.
The authors did not chase down why, and they say that they cannot: with one CVE per codebase, the bug and the codebase cannot be separated as causes. What that leaves is a warning against reading any industry average as a forecast for your own repository. The tooling behind the study is public for exactly this reason, so an organization can run it against bugs it has already fixed and get a local number instead of someone else’s.
Models rebuilt a bug the kernel had already shipped
Copy Fail, the Linux kernel privilege escalation disclosed in April, produced the highest new-vulnerability rate of any target for Claude and close to the highest for ChatGPT. The upstream fix for it reverted a memory optimization and, in doing so, dropped an offset guard and introduced an off-by-one heap write that maintainers had to correct in a later commit. Asked to patch the same bug, roughly a third of the patches from each model regenerated that flawed revert, off-by-one included.
Sitting in the same stretch of code the models were rewriting was a second flaw, one nobody had asked them to look at. Kernel maintainers closed it separately, in a commit of its own. Every patch in both campaigns walked past it, including the ones that cleanly fixed Copy Fail and the several hundred that edited that exact file. The only patches that closed it came from runs the researchers had already thrown out for copying the official fix. The models patch the bug in the ticket, and nothing else, even when the something else is in front of them.
Grading 6,080 patches by hand was not possible, so the study leans on model validators, cross-checked against each other and spot-checked by humans. On a sample, the validators and the human reviewers agreed on the exact grade about two thirds of the time, and the two models disagreed with each other often enough that the authors average their verdicts rather than trusting either. The Copy Fail off-by-one is the clearest thing that slipped past: the automated reviewers caught it in a couple dozen patches out of the roughly 250 that contained it. The study’s new-vulnerability rate should be read as a floor.
What this costs
Each patch attempt and validation cycle ran between two and three dollars, cheap against an engineer’s afternoon. 1Password researchers wrote that “LLM-produced patches still require review from a skilled engineer with domain expertise.” Their manual review makes the point about why that review is expensive: understanding a patch well enough to certify its security implications takes at least as much work as writing a known-good patch would have.
The six CVEs here were chosen for high impact and complex, multi-file fixes, which makes them harder than a random vulnerability, and the authors are direct that their numbers do not describe the average bug. The findings are also a snapshot of two model versions at one moment. A patch that looks right and passes the reproducer remains the expensive kind of wrong, because catching it costs roughly what writing the fix yourself would have cost.