Context Engineering Research: What I Found After You Shared That Paper

Last week on one of our AI WhatsApp groups, Staszek dropped a link to an ETH Zurich paper with an UNO reverse card. The paper: “Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?” The abstract’s punchline: context files tend to reduce task success rates while increasing inference cost by over 20%.

The implied message was clear. I’ve been writing and teaching about context engineering for months. Building a product around it. Telling anyone who’ll listen that it’s the most important AI skill. And here’s a rigorous academic paper saying the files don’t work.

I couldn’t just shrug that off. So I pulled the Gloaguen paper, then kept going until I’d read 20 other studies from the past four months. What I found was more nuanced than either “context files are useless” or “context engineering is everything.” Here’s the short version for the group.

---

TLDR

• The paper is correct for what it measured (fixing GitHub issues)
• But it only measured one thing and missed several others
• A contradicting paper found context files make agents 28% faster (even if not more accurate)
• The strongest paper (ICLR 2026) shows dynamic context produces +10.6% gains, validating the practice when done right
• Nobody has benchmarked knowledge work - every study tests coding tasks

---

The Gloaguen Paper (arXiv:2602.11988) - What It Actually Says

• LLM-generated context files: -3% success rate, +20% cost
• Human-written context files: +4% success rate, still +20% cost
• Agents DO follow instructions in context files (1.6-2.5x tool usage)
• Repository overviews don’t help agents find relevant files faster
• Recommendation: “Human-written context files should describe only minimal requirements”

Task details: 438 tasks across SWE-bench Lite + AGENTbench. Average: 119 lines edited, 2.5 files. Binary pass/fail (tests pass or they don’t). All Python repos.

The agents listened to the files. They changed their behavior. The benchmark just didn’t reward the change.

---

The Paper They Didn’t Tell You About (arXiv:2601.20404)

Lulla et al., published 3 weeks before Gloaguen, measured the same thing differently:

• Median runtime down 28.64%
• Output tokens down 16.58%
• Task completion rates: unchanged

Same intervention, opposite efficiency conclusion. Gloaguen counted total inference cost (including reading the file). Lulla counted how efficiently the agent worked after reading it. Both are right.

---

The Paper That Changes The Argument (arXiv:2510.04618)

Zhang et al., “Agentic Context Engineering” (ACE). Published at ICLR 2026.

Instead of static files, ACE runs a loop: attempt task -> reflect on what worked -> update context for next task. Context evolves.

• +10.6% on coding benchmarks
• +8.6% on financial reasoning
• Matched top proprietary model using a smaller open-source model with ACE context

This is the key distinction: static context files = marginal. Dynamic context systems = compounding gains.

---

The Productivity Evidence Is Contradictory

Study	Domain	Finding
METR (arXiv:2507.09089)	Experienced OSS devs, debugging	19% SLOWER with AI tools
Sarkar (SSRN:5713646)	1,000 orgs, coding agents	39% more weekly code merges
Ju & Aral, MIT (arXiv:2503.18238)	2,234 people, ad creation	50% more output, higher quality

The pattern: AI slows you down on precise, context-heavy debugging. Speeds you up on generative, less-constrained work. Context engineering closes the gap on the first category by giving the model the context it lacks.

---

What All 20+ Papers Agree On

1. Less context > more context. Every paper. Unfiltered context hurts. Aggressive scoping helps. Simple masking (hiding irrelevant info) beat sophisticated summarization in the JetBrains study: +2.6% accuracy, -52% cost.

2. Agents over-retrieve. ContextBench (arXiv:2602.05892) tested 1,136 tasks: LLMs consistently grab too much info and use a fraction of it.

3. Long context degrades performance. LOCA-bench (arXiv:2602.07962): success rates dropped from 40-50% to <10% as context grew. Progressive disclosure > front-loading.

4. MCP tool use is still hard. MCP-Universe benchmark: GPT-5 at 43.72%, Claude Sonnet at 29.44% on MCP tasks. Clear tool documentation matters.

---

The Gap Nobody’s Filling

Every benchmark tests coding: SWE-bench, AGENTbench, LOCA-bench, SWE Context Bench. None test knowledge work: multi-source data orchestration, personalized communication, cross-session learning, or whether today’s AI work improves tomorrow’s.

The “Beyond Task Completion” paper (arXiv:2512.12791) calls this a “fundamental misalignment between benchmarks and real needs.”

This is why I think the Gloaguen finding, while valid, doesn’t invalidate context engineering as a practice. It invalidates the simplest possible implementation (a static README) on the narrowest possible benchmark (binary code fix). The more interesting question is whether dynamic, multi-source context systems make AI useful for ongoing, personalized work. Nobody’s tested that yet.

---

Quick Reference: Key Papers

Paper	arXiv	Key Finding
Gloaguen et al.	2602.11988	Static context files: marginal for coding accuracy
Lulla et al.	2601.20404	Context files: 28% faster runtime
ACE (ICLR 2026)	2510.04618	Dynamic context: +10.6% compounding gains
METR	2507.09089	Experienced devs 19% slower with AI
Ju & Aral (MIT)	2503.18238	Creative tasks: 50% more output
SWE Context Bench	2602.08316	Filtered experience helps; unfiltered hurts
ContextBench	2602.05892	Agents over-retrieve by default
Memory Survey	2512.13564	Memory is a first-class agent primitive
Agent READMEs	2511.12884	2,303 context files studied; only 14.5% address security

---

I wrote a longer piece synthesizing all of this for the blog: read the full essay here. Happy to debate any of this over beers.

Related: The File System Is the Prompt covers the practical 5-layer model these papers validate. The full case study shows dynamic context compounding in a live production system.