For AI coding assistants, useful context is not always secure context

A coding assistant asked to complete a function may retrieve similar code, imported definitions or neighboring files. Those choices can reveal a project's permission checks—or provide a vulnerable example to copy. A.S.E 2.0, an arXiv preprint by Jiongchi Yu and colleagues submitted on November 4, 2026, studies this selection process directly. The supplied landing page does not establish journal publication or peer review.

The central finding is not that repository context is useless. Rather, methods optimized to find functionally relevant code often improve ordinary test performance without delivering comparable gains against an executable attack.

Testing working code and exploitable code separately

The researchers built CASE-Bench: 145 code-generation tasks derived from 144 documented vulnerabilities across 51 open-source projects, covering seven languages and 28 weakness types. Each task starts from a historical vulnerable repository with a region removed. The model receives a functional requirement and the masked target file, but not the vulnerability identifier, historical fix, evaluation tests or exploit.

Generated implementations are built in isolated containers, run through functional tests, then attacked with a proof-of-concept exploit. This distinguishes broken code from code that passes the tests but remains exploitable. The researchers checked that each original vulnerable version passed the functional tests and succumbed to the exploit, while its patched version passed the tests and resisted it.

Importantly, “secure” here means passing those tests and resisting that particular exploit—not being free of vulnerabilities. On a 40-task subset, additional exploit variants compromised 7.3% of generations that resisted the base attack.

Better functionality did not imply comparable security

The main comparison used six language models, 13 context-construction methods and six controls, with three runs per task and condition. Most methods shared an 8,000-token retrieval cap beyond the target file. The study reproduced context-selection policies in a common pipeline rather than benchmarking every original product unchanged.

With only the target file, 56.9% of generations passed the functional tests, but just 12.6% also resisted the exploit. Across eleven functionality-oriented methods, functional performance changed by minus 4.8 to plus 8.2 percentage points; secure performance changed by minus 1.1 to plus 3.7 points. None showed a statistically significant security difference from the target-file-only baseline after correction for multiple comparisons. That does not rule out smaller effects.

ProCC-Multi illustrates the distinction: it achieved the highest functional pass rate, 65.1%, but a secure pass rate of 14.9%. A security-aware source-to-sink method, which follows untrusted input toward sensitive operations and retrieves intervening guards, reached 18.1% secure performance, a statistically significant 5.5-point improvement over the baseline. Its advantage over generic security-prompt controls was not significant after correction.

Expert-annotated reference context containing safeguards and the relevant data-flow path reached 24.8%. This was a diagnostic condition, not a deployable retrieval method or a guarantee of safety.

Which examples the model sees matters

Controlled context edits with three models provided more direct evidence about individual ingredients. Removing a safeguard role from the reference context reduced secure performance by 3.9 to 8.1 percentage points, depending on the role. Some lost secure outputs became functionally incorrect, so these effects were not exclusively about security.

On 74 tasks with an existing insecure example, inserting one such example into the reference context increased correct-but-exploitable output by 6.3 points, even though the safeguard remained visible. Inserting unrelated code of the same length at the same position increased it by only 0.6 points. These matched comparisons support an effect of the example's content, rather than merely its added length.

Retrieving safeguards was still insufficient. In an OpenEMR case study, some generations given the relevant permission rule checked it only after processing the request. The necessary policy was present but applied too late.

Agents also missed safeguards

Across 1,305 coding-agent trajectories, agents observed the relevant safeguard in only 27–36% of runs, despite reading substantially more repository material than fixed retrieval contexts. Most queries concerned functionality. Safeguard observation was associated with better secure outcomes, but this analysis was observational and does not prove that observation alone caused the improvement. Agents were also restricted to editing the masked region and operated under step and cost limits.

The benchmark deliberately favors reproducible historical vulnerabilities, and six projects contribute 67 tasks. Training-data contamination cannot be excluded. These constraints limit generalization to unrestricted software development.

The practical implication is narrower than “add more context”: retrieve the project's actual checks and policies, scrutinize similar examples, and evaluate exploit resistance separately from functional success. The study supports that direction without demonstrating a complete secure-coding solution.