Study Finds Context Compaction Largely Harmless to LLM Agent Benchmarks Despite Silent Loss of Constraints

Current benchmark resilience conceals a severe structural blind spot: agents manage long-horizon tasks not by remembering all operational parameters, but because static targets and external disk states mask lost instructions.

• A systematic audit of context compaction was released: Published on arXiv on October 22, 2026, researchers Jeanne Salle, Hardik Bhatnagar, Matthias Bethge, Sahar Abdelnabi, and Maksym Andriushchenko evaluated five context management strategies across three frontier models on PaperBench, ProgramBench, and PostTrainBench.
• Benchmarks tolerated aggressive context pruning: For models operating with 1-million-token context windows, context compaction thresholds produced no measurable drop in high-level task completion, while simple self-summarization matched or beat all alternative compaction techniques.
• Guardrails were discarded in the background: Compaction silently dropped approximately one-third of anticheating constraints on ProgramBench after the very first summary pass, with behavioral rules getting pruned six times more frequently than basic task instructions in real SWE-chat data.

• Agent contexts are outgrowing model capacities: Long-horizon autonomous development tasks routinely generate interaction traces that exhaust model limits, forcing agent scaffolds like OpenCode to discard or compress earlier conversational history.
• Synthetic tests missed frontier behaviors: Prior industry assumptions rested on smaller models and synthetic benchmarks, which recognized that context compaction was inherently lossy but failed to measure how frontier architectures behave on realistic coding tasks.
• Aggressive discarding carries sharp penalties: While basic self-summarization preserved core functionality across tests, aggressive discarding of older conversational state reduced benchmark success scores by up to 20 points in specific frontier model runs.

The core finding of the research is an operational paradox: autonomous LLM agents are surviving context compaction not because their summarization retains complete situational awareness, but because current benchmarks fail to penalize the selective loss of execution constraints.

When agent tasks stretch across hours of execution, scaffolds compress tokens using summarization or state-truncation techniques to avoid hard context window boundaries. Under testing across benchmarks like ProgramBench, PaperBench, and PostTrainBench, this pruning mechanism creates an illusion of complete stability. A model operating with a 1-million-token window shows virtually zero change in high-level success rates regardless of compaction thresholds. Even GPT-5.3-Codex, which registered lower scores at lower context thresholds on select benchmarks, managed to retain overall objective completion as long as scaffolds avoided crude discarding mechanisms, which reduced scores by up to 20 points.

The structural vulnerability lies in what compaction removes first. In real-world coding and agentic execution (observed across authentic SWE-chat sessions), context compression prunes instructions governing agent behavior six times more frequently than it prunes explicit task deliverables. In ProgramBench environments, autonomous agents discarded roughly one-third of anticheating guidelines after just one summarization cycle. With each subsequent compaction pass, constraint retention degraded further.

This selective degradation persists unnoticed on standard leaderboards because of the environment architecture. Current benchmarks feature two structural realities: fixed final objectives and persistent agent progress written directly to disk. When an agent creates files, runs commands, and alters code repositories, the environment acts as external non-volatile memory. The agent does not need to remember its initial working memory to complete the code; it merely inspects the files remaining in its working directory.

Consequently, the incentives governing modern agent deployment are misaligned. Standard metrics reward agents for achieving the final code state, ignoring whether non-functional requirements—such as data privacy policies, behavioral rules, testing constraints, or security limits—were discarded mid-run. As long as developer evaluation frameworks measure only top-level task completion on disk, compaction-induced constraint drift will remain hidden inside enterprise pipelines.

• Enterprise platform engineers: Teams deploying frontier coding agents will find that long-horizon jobs complete their primary code objectives while quietly ignoring security boundaries and internal compliance rules.
• Benchmark designers and AI evaluators: Research teams running evaluations like ProgramBench must redesign test harnesses to penalize behavioral instruction drift rather than grading purely on final disk outputs.
• Safety and governance leads: Organizations relying on prompt-injected system instructions to enforce corporate governance will face invisible policy degradation as agents run multi-step summarization passes.

• Scaffold constraint pinning: Watch whether production frameworks update from plain self-summarization to persistent "pinned" instruction blocks that permanently isolate system rules from the compaction pool.
• Dynamic constraint auditing: Monitor whether new evaluations emerge that score agents continuously on compliance steps rather than grading purely on final artifact validation.
• Long-horizon error logging: Track enterprise bug reports where coding agents violate non-functional requirements during extended multi-turn refactoring sessions.