Context files raise cost without significant task-success gains
Source → Observed → Interpretation → Model implication
Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?
View source →The authors evaluated Claude Code with Sonnet 4.5, Codex with GPT-5.2 and GPT-5.1 mini, and Qwen Code with Qwen3-30B-coder across SWE-bench Lite and a new CTXbench benchmark. SWE-bench Lite contains 300 tasks from 11 popular Python repositories; CTXbench contains 138 tasks from 12 recent, less popular Python repositories with developer-committed context files.
LLM-generated context files changed average resolution rate by -0.5% on SWE-bench and -2% on CTXbench. Neither difference was statistically significant. They increased the average number of agent steps in every setting and raised average inference cost by 20% and 23%, respectively; the cost increases were statistically significant.
On CTXbench, developer-provided context files improved average resolution rate by 2.4%, which was not statistically significant at p=0.21. They significantly outperformed LLM-generated files at p=0.038, while adding an average of 3.34 agent steps and increasing cost by as much as 19%.
Trace analysis found that agents generally followed context-file instructions. With context files present, agents ran more tests, performed more code-quality checks, explored more files, and used named repository-specific tools more often, but these behavioral changes did not produce a significant overall improvement in resolution rate.
Repository overviews did not meaningfully reduce the number of steps before agents interacted with a file modified by the reference patch. For GPT-5.1 mini, context files significantly increased those steps, including repeated attempts to find and read a context file already included in the agent's context.
In an ablation where the authors removed other documentation from CTXbench repositories, LLM-generated context files improved resolution rate by 2.7% on average and consistently outperformed developer-provided files. The authors interpret this result as evidence that generated context files are often redundant with documentation already present in a repository.
Context availability is not equivalent to context fitness. Repository files can successfully steer agents toward more testing, exploration, quality checks and specialized tools while still increasing cost without improving test-verified outcomes. Whether context is useful depends on its relevance and novelty relative to knowledge already available in the repository. This also separates verification activity from verification value: performing more checks did not by itself raise task-resolution success.
REFINES. The results refine Selection by showing that adding a commonly recommended environmental artifact can reduce rather than improve agent fitness. The environment still measurably shapes behavior and Economics, but Selection must evaluate context quality, redundancy and outcome impact rather than treating additional context as inherently beneficial. Instruction following and increased verification activity are insufficient proxies for useful outcome.
What this does not establish
- The study does not establish that all repository context files are harmful. Developer-provided files outperformed generated files, and generated files improved average resolution rate when other repository documentation was removed.
- The evaluation covers Python repositories only. The authors note that models may already contain substantial parametric knowledge about common Python tooling and that results may differ for less represented languages and toolchains.
- Each agent-condition completion was sampled once, so the study does not establish the stability of per-task effects across repeated stochastic executions.
- CTXbench uses LLM-generated task descriptions and tests derived from historical pull requests. The authors manually edited tests for 25 of 138 instances and report that the generated descriptions make SWE-bench tasks about 15% easier while preserving model rankings across description variants.
- Passing the benchmark tests measures task resolution, not maintainability, security, code efficiency, broader developer intent, or production impact.
- The study reports benchmark inference cost and agent steps, not organizational adoption, concurrent operation, maintenance cost for context files, or population-scale economics. It establishes nothing about 1K-agent scale.
Which specific repository instructions add non-redundant knowledge that improves repeated, test-verified outcomes after accounting for their full inference, exploration and maintenance cost?