Security rules for coding agents help, but merge gates still earn their place
Arnica's benchmark of six agent setups found security rules cut wrong security decisions in new code from 47% to 17%. Better, but not safe: keep scanners and review in the pipeline.
What happened
On 14 October 2026, Arnica published a benchmark of how AI coding agents handle security when nobody asks them to. It tested six models from five vendors, each inside its usual tool: Claude Code, Codex, GitHub Copilot and Cursor. The agents were given feature tickets with no security wording. The authors had written an answer key in advance, listing the decisions a careful engineer would make unprompted, such as authorisation checks, tenant filters and output escaping.
Without security rules, 47% of the security decisions in the new code were wrong. With Arnica's agentic rules, that fell to 17%, roughly 2.7 times fewer. The split matters. On public, deliberately vulnerable open-source apps the rate went from 58% to 32%. On production-style apps, where the rules had been tuned, it went from 35% to 2%. Arnica notes those production-style results are in-sample.
How the rate moved, by type of application:
| Applications | Without rules | With rules |
|---|---|---|
| All apps | 47% | 17% |
| Public, deliberately vulnerable open-source apps | 58% | 32% |
| Production-style apps (in-sample) | 35% | 2% |
And by model and tool pairing, where the sample for each is small:
Other findings from the post:
- Without rules, 70% of sessions introduced a critical or high-severity vulnerability, and 31% committed a planted API key.
- With rules, 59.5% of sessions still wrote vulnerable code at least once.
- Rules helped least on unbounded resource use, race conditions and session handling (findings per session below).
- Breaking changes per session roughly doubled, from 2.6 to 5.3.
Findings per session in some of the vulnerability classes, with and without rules:
| Class | Without rules | With rules |
|---|---|---|
| Authorisation | 0.93 | 0.43 |
| Secrets in code | 0.33 | 0.03 |
| Unbounded resource use | 0.74 | 1.04 |
| Race conditions | 0.30 | 0.36 |
| Session-handling gaps | 0.29 | 0.36 |
Arnica is the vendor whose product was tested, and its authors say so. They also acknowledge that model and tool effects cannot be separated, that per-pairing results lack confidence intervals, and that the main grader shares a vendor with the best-scoring pairing. A second-vendor model re-graded a sample and agreed on 93.2% of more than 1,000 judgments. The authors' own conclusion is that teams should keep scanners and merge gates.
Why it matters
Agents do not default to secure. A coding agent that gets a ticket with no security wording will often skip the decisions a senior engineer makes without thinking. If your engineering policy assumes the agent "knows" to check authorisation, this benchmark is evidence against that assumption.
Instructions work, and they are cheap. The largest improvement was on authorisation, which fell from 0.93 to 0.43 findings per session, and on secrets in code, which fell from 0.33 to 0.03. A rules file in the repository costs almost nothing compared with a vulnerability found in production. This is the part of the result we would act on immediately.
Rules shift the failure mode rather than remove it. Three vulnerability classes got worse or stayed flat: resource limits, race conditions and session handling. Those are the ones that need design judgement, not a checklist. An agent told to be careful about authorisation can become careful about authorisation and still ship a request handler with no rate limit.
Review capacity is the real constraint. Breaking changes doubled. Agents that fix more pre-existing problems also change more behaviour, and 62% of the breaking verdicts were API-contract changes or new authentication requirements. Each one needs a human to decide whether it was intended. If your reviewers are already the bottleneck, more agent output increases the queue.
What we would do
- Write security rules for every repository an agent touches. Cover authorisation, tenant isolation, output encoding, secrets handling and input limits, in the form each tool reads. Keep them short, specific and versioned with the code.
- Keep scanners and merge gates mandatory for agent-authored changes. Static analysis, secret scanning and dependency checks should run on every pull request, and a failure should block merge. Do not relax them because the agent was instructed well.
- Add explicit checks for the weak classes. Rate limits, concurrency and session handling are where rules helped least, so put them in the review checklist and, where possible, in automated tests.
- Ask agents to report behaviour changes. Require a short rollout note listing any API-contract or authentication changes, and have a named reviewer sign them off. Treat an unannounced contract change as a defect.
- Trial on your own code before you standardise. Run a handful of real tickets through your preferred tool with and without your rules, and have a security engineer score the output. Do it again when you change model or tool, because the benchmark suggests the pairing matters.
- Track the outcomes, not the volume. Measure vulnerabilities found per merged change and review time per change, alongside how much code the agent produced.
The practical position is simple: write the rules, then verify as if you had not. The benchmark supports the first half and is explicit about the second.
Questions we are asked
Should we write security rules for our coding agents?
Yes. Arnica's results suggest that explicit security instructions change what agents produce, particularly for authorisation checks and secrets. But treat rules as one layer. The same benchmark found rule-equipped sessions still wrote vulnerable code at least once most of the time, so scanning and review remain necessary.
Does the choice of model and tool matter?
According to the benchmark, yes. Error rates ranged widely between pairings, with and without rules, and the weakest rule-equipped pairings still did worse than the strongest model without rules. The sample per pairing was small, so use it as a prompt to test your own setup rather than as a league table.
How much weight should we give a vendor's own benchmark?
Some, with care. Arnica sells the rules it tested and says so, and its authors list their own caveats, including in-sample results on part of the data. The direction matches what we see in practice, but run a small trial on your own repositories before changing policy.