Large language models (LLMs) are increasingly used in real-world environments, where relevance and accuracy of information are critical. But for agentic tasks, existing LLMs struggle to accumulate long histories of actions, observations, and account data to accommodate the growing context length.
This is because:
Fixed-token windows in LLMs force information to be truncated, summarized, or discarded as conversations grow.
Context decay: Gradual loss of older information, which often competes for attention with newer inputs.
This results in contradictions, repeated information, forgotten goals, and poor performance.

Sure, strategies like retrieval-augmented generation (RAG) improve the factual accuracy by extending the model’s knowledge boundaries through external content. But this often results in irrelevant retrieved content, increased inference costs, and may exceed LLMs’ context window limits, particularly in long-running interactions.
Context compression addresses these issues by eliminating irrelevant information from context before LLM generation. But there is still a challenge: Existing context compression methods struggle to adaptively adjust compression rates across contexts, maintain low latency, and integrate information across multiple documents.
This article proposes an adaptive context compression technique intended for long-running LLM agents.
Our Proposed Adaptive Context Compression Framework for LLMs
The framework has mainly three components, each discussed below one by one:
Context segmentation
This involves separating conversation history into explicit units:
Task objectives: driving planning loops in LLM-based AI agents
User preferences: refer to personalization requirements
Constraints: strict requirements that apply even when the conversation shifts
Episodic events: store decisions made, tools run, and outcomes, and later retrieve them when relevant.
Intermediate reasoning: temporary thoughts that can help in the moment but usually don’t need to be stored long-term. For long-term interaction, this can be safely compressed or discarded later.
Adaptive compression policies
Each proposed policy below is inspired by prior mechanisms, but adapted for dynamic context reduction:
Importance-weighted summarization: Generative-agent architectures select memories based on factors such as relevance, recency, and importance. In our framework, we use the same approach for context compression, allocating token budget to crucial information, such as core objectives, constraints, and high-impact events. Low-importance content is compressed using extractive summarization or structured key-value distillation.
Hierarchical memory tiers: Our framework uses a tiered memory system to separate prompt (“fast memory”) from external storage (“slow memory”). Three tiers of the framework are:
a short rolling window of raw turns
a structured compressed state (typed units)
an archival retrieval store (vector DB / document store) for episodic details
This design ensures the prompt does not become overloaded while still retrieving the required historical details.
State-aware compression: Compression should adapt to the agent’s current task phase: Planning, execution, and debugging. Our framework triggers aggressive consolidation after key milestones, such as task completion or ingestion of tool results. When contradiction signals appear, the system performs a targeted coherence check and updates the structured state accordingly.
Coherence anchoring mechanism
Compression should reduce size, but it must not remove critical information that defines the task. To prevent this, we introduce three coherence anchors that remain stable across compression cycles:
Task-state object: Current goal stack and sub-goals
Constraints registry: a persistent record of hard requirements.
Commitments log: List of decisions made during the interaction
These anchors are updated incrementally, before generating a response, rather than rewritten during summarization.
The system reconstructs the active task state from these structured records. This aligns with long-term memory frameworks that continuously update and reuse structured memory rather than treating summarization as a one-off rewrite.
Experimental Design
To evaluate adaptive compression properly, test long-horizon behavior rather than just short document question answering. We recommend a mix of benchmarks:
LongBench / LongBench v2 for multi-task long-context understanding
RULER (and optionally L-Eval) for controlled stress tests at increasing context lengths
MSC (Multi-Session Chat) for testing memory across extended conversations
Some research shows that benchmark scores don’t always match human judgment. So the evaluation should combine automatic scoring, LLM-based judging, and human evaluation.
Results and analysis
First, compared to retrieval-only and hierarchical baselines, token efficiency should increase as context windows of a large memory history are condensed into structured memory. This is consistent with memory systems that maintain small “memos” or summaries, allowing long, coordinated conversations to take place over time without a linear increase in prompt size.
Second, coherence is likely to improve compared with sliding windows because anchors (objectives, constraints, commitments) are explicitly retained, whereas they are otherwise lost when sliding out of the input buffer. Research on long-term conversation reports that models without attention mechanisms fail, and that methods which retrieve or summarize past turns achieve at least some performance benefit over standard architectures.
Third, in contrast to static summarization, adaptive policies should mitigate the problem of “summary drift” by selectively updating memory and downgrading transient reasoning traces. It is motivated by iterative memo and memory-update frameworks and systems designed to move information between memory tiers rather than whole rewrites.
Possible Risks
Risk of miscompression: if summarization introduces errors or deletes a “binding” constraint, the agent can become confidently inconsistent. Agent simulations explicitly report common breakdowns when retrieval fails or when the model fabricates embellishments in memory-conditioned answers.
Failure of retrieval memory: The system may retrieve semantically similar but pragmatically wrong episodes. It’s a challenge acknowledged across retrieval-based dialogue memory work.
Growing privacy and safety risks: Any adaptive compression system deployed in user-facing settings should therefore include retention limits, auditability, and user controls for deletion.
Scalability considerations
Scalability depends on constant-factor engineering: embedding generation, vector indexing, and retrieval are not free. That’s why toolkits like FAISS exist because embedding stores can get large, and careful trade-offs have to be made between indexing and compression. Hierarchical memory managers can reduce prompt growth, but at the cost of orchestration complexity and increased movement of models/tools.
Future Scope
The next logical step is to learn compression policies instead of hand-tuning them. Memory-R1 shows that reinforcement learning can train agents to do structured memory operations (ADD/UPDATE/DELETE/NOOP) for outcome-driven rewards, which directly translates to adaptive compression. Another avenue is benchmark-driven development: long-context suites (LongBench v2, RULER, L-Eval) and memory-specific benchmarks can help assess not only recall but also long-horizon coherence under budget constraints.



