University of Edinburgh · Mitacs Globalink · 2026
Does a Working Exploit Help an Agent Patch?
A controlled test of whether a working exploit helps an agent write a better patch. It did not.
- Evaluation
- Ablation
- Null result
Overview
The common assumption is that showing a repair agent a working exploit produces a better patch. I tested it with everything else held fixed — same agent, same model, same prompt wording — so the exploit evidence was the only variable.
Findings
- No significant difference: +3.0 points on a 0–100 scale across 12 matched CVE pairs, win/loss/tie 6–5–1, sign test p ≈ 1.0 — smaller than the scoring harness’s own measured noise — at ~3× the cost per run.
- Built the evaluation harness: a reference-anchored LLM-as-judge scoring patches against the real upstream fix across 8 dimensions, with a noise floor measured before comparing anything.
- Diagnosis: a single proof-of-vulnerability is a narrowing signal, so a gate defined on that one path rewards patches that close it rather than the weakness behind it.