Picture an analyst who keeps one agent conversation open for a week. On Monday they pull quarterly numbers for five companies. On Tuesday they compare margins and build a chart. On Thursday they come back and ask, “what did we decide about the second one?”
The naive design keeps the last N turns and forgets everything else. It’s predictable, it’s cheap, and it forgets exactly the things people come back for. The other naive design keeps everything until the context window overflows, then fails. Compaction lives between those two, and I think the most useful way to design it is to be honest about what you’re willing to lose.
Trigger early, on a budget you can measure
Compaction should run before the window is full, not when a request is rejected. A threshold like “80% of the budget, minus a reserve for the reply” leaves room for the turn that’s about to happen: tool results, a long answer, context injected at the end.
Measure that budget in whatever unit you can compute everywhere. If you can’t load a tokeniser for every model you support, characters with a conservative ratio work fine, as long as the same estimate is used for admission, for packing history, and for the final pre-dispatch check. A request that passes one estimate and fails another is worse than a slightly pessimistic estimate used consistently.
The projection should include everything that goes into the call, not just the conversation. Injected context, a status block, and skill text all count, and they’re easy to forget because they aren’t “history”.
Cut on turn boundaries
Once you’ve decided to compact, the order matters:
- Keep a recent window raw. Walk backwards from the newest turn and keep whole turns — the user message, the assistant reply, and that turn’s tool evidence together — until a budget is used. Never cut inside a turn. A tool result without the call that produced it, or a reply without its question, confuses models in ways that are hard to debug.
- Clear old tool output first. Raw tool results are usually the largest and least reusable part of old turns. Replace them with a short placeholder that tells the model they were cleared and how to get them back.
- Summarise what’s left of the middle. Pass the previous summary in and ask for a merged one, rather than re-summarising from scratch each time. Summaries of summaries drift. Merges drift more slowly.
def compact(turns, prior_summary, budget):
recent, older = split_recent(turns, budget.keep_recent) # whole turns only
older = [strip_tool_output(t) for t in older]
summary = summarise(prior_summary, older)
return summary, recentConstrain what the summary may say
A free-form summary is a second model generating claims about the first model’s conversation. I’d give it fixed sections — symbols, facts, numbers, URLs, artefacts, blockers, open threads — and a few hard rules: list only facts present in the source, never invent prices or identifiers, never carry credentials or one-time codes forward. Then run a redaction pass over the output anyway, because “never include secrets” is an instruction, not a guarantee.
Low temperature helps. So does capping the summariser’s own input, so a pathological conversation can’t turn compaction into its own context overflow.
Store the summary next to the conversation, along with a marker of the first turn it doesn’t cover. And leave the original events alone. Compaction should change what the model sees, never what the system remembers.
The summariser is subject to the same policy as the chat
It’s tempting to send compaction to the cheapest model you have access to. A good default order is: an explicitly configured summariser, then any allowed model that’s small, then the chat model itself.
The important word there is allowed. If a workspace keeps certain content on a self-hosted model, compaction can’t become the side door that sends a summary of that content to a cheaper cloud model. A summary of confidential material is confidential material. Run the summariser request through the same inference policy check as any other call, and fall back to a less ideal model rather than bypass the check.
Availability or completeness — pick one on purpose
The textbook rule is that a failed or empty summary must never advance history. If you couldn’t summarise the older turns, you shouldn’t drop them.
I’ve come round to thinking that’s the wrong default for an interactive product. Consider what it means in practice. The summariser times out after 45 seconds. Under the textbook rule, the user’s turn now either fails or runs with a context that’s still too large, and every following turn hits the same wall. The conversation is effectively dead until someone intervenes.
The alternative is to keep going. If the summariser returns nothing but there’s a previous summary, keep the old summary and still move the recent window forward. If it throws, do the same. If there’s no previous summary at all, drop the oldest turns until things fit, and record that you did. In every case some turns disappear from the model’s view, and the conversation stays usable.
That’s a loss of completeness in exchange for availability, and I think it’s the right trade only if the loss is recoverable. Which brings me to the half of compaction people tend to skip.
Recall makes lossy summaries survivable
If everything that leaves the model’s view is still in an append-only event log, and the agent has a way to search it, then compaction is a cache eviction rather than a deletion.
Two tools are enough: search over the conversation’s events (full-text search with SQLite FTS5 or Postgres works well; you can fuse in an embedding retriever with reciprocal rank fusion if you need it), and expand, which returns a small window around a hit. Keep both lean by default. Search should return a handful of short hit cards, not windows of context. Expand should have a hard radius and character cap. Tell the agent to expand at most a couple of times per turn, and to re-fetch the original source if what it found is still truncated.
The details that made search useful in my experiments were boring ones. Exclude lifecycle noise (“response completed”, “run started”) from the index, or the top hits will be markers rather than evidence. Rank hits from the current turn lower. Add stop words. Strip secrets before indexing. Scope search to the current conversation or one explicitly named conversation the user owns, and make zero hits say “not found here”, not “doesn’t exist”.
There’s one more rule I’d keep: facts from a summary are weak evidence. If a number from the summary is going into today’s answer, the agent should expand the original or fetch it again. A summary tells you where to look. It shouldn’t be quoted as a source.
Some runs shouldn’t see history at all
Compaction quality isn’t the only way history goes wrong. A scheduled analysis that inherits an earlier chat can blend what someone discussed with what the data says. A recurring report about one company can pick up a comparison with a different company that someone ran by hand the week before.
For scheduled and triggered runs, I’d start with zero prior turns and let the run read what it needs from current, saved records. For “inspect this saved thing” runs, pass only the last message or two, labelled as being for resolving references like “it” or “that alert”. Context isolation is often a bigger win than better summarisation, in both correctness and cost.
Structured records need care too. If a saved workflow has a schedule, a calendar of eligible days, and a large targeting payload, compacting it as one blob can drop the schedule to save the payload. Keep the small, decision-relevant fields separate from the bulky ones so they survive trimming.
What I’d still worry about
Overlap: if the stored summary is prepended to the recent window, and the window has moved since the last compaction, the two can cover some of the same turns. That wastes a few hundred characters, not correctness, but it’s worth tracking. Character-based budgets are fine for one provider and loose for others. And an agent that never searches its history will behave exactly like one that has no history, so it’s worth checking in evaluations that it actually reaches for search when a question refers to something old.
The part I’m confident about is the split: an event log that keeps everything, a model view that keeps little, and a search tool between them. With that in place, the summariser is allowed to be imperfect.