Block 05 / 07
The context window is the interface: its contents determine the next step.
Classical software has a request and a response. Language-model systems have a window. Everything the model can use must be tokens in that window: contract, tool schemas, retrieved evidence, dialogue, last observation. If it is not in the window, it does not exist for that hop. Context engineering is packing that interface on purpose.
Order is policy. Pin the contract first. Cache the stable prefix. Retrieve evidence per turn. Compact completed traces. Isolate untrusted text so a RAG chunk stays data. Stuffing is how you page out the schema.
Budget 8,000 tokens · moderate pressure, scratch is evicted and history is compacted
Drag the slider to change the token budget. Watch what gets compacted first, and what never does.
Packing the window on purpose is a set of concrete techniques, not one trick. The three below are the ones that show up in nearly every production agent.
Structural
Providers cache a request’s shared prefix so a repeated call skips reprocessing it. That only works if the prefix is byte-for-byte stable: a timestamp, a random ID, or a reordered field anywhere in the system prompt or tool list invalidates the cache on every call.
Windowing
Once history exceeds its share of the budget, the oldest turns are dropped first, the same first-in-first-out rule behind a circular buffer, and the same windowing idea used for attention itself (block 03). It is cheap and deterministic, at the cost of losing whatever only appeared in the turns that fell off the front.
Allocation
When demand exceeds the budget, a fixed priority order decides what gets cut first: pin what cannot move, then trim the rest greedily until the total fits. It is the same greedy logic behind bin-packing and cache eviction, applied here to token budgets instead of bytes or cache lines.
Paper
Position in the window is not uniform. U-shaped attention to evidence is a measured constraint on how long context should be packed.
Article
A current account of packing production agents: layers, compaction, isolation.
Course
CS224N supplies the token and attention machinery. The lab on this site makes overflow visible.
The context window is a finite interface. Whatever the model uses on this hop must already be tokens in that window.
The contract, tool schemas, retrieved evidence, and the last observation compete for the same budget. Order is a policy: pin the contract, cache the prefix, compact completed traces.
Chunks that do not fit are absent for that hop. Overflow is a paging problem. Isolation keeps untrusted HTML as data so it cannot overwrite the contract.
A stable prefix (system plus tools) can be cached. A prefix that mutates every request, timestamps included, cannot.
Allocate system, tools, evidence, history, and scratch against 8K, 32K, and 128K caps. Implement a compaction order that never evicts the schema. Isolate one untrusted chunk. Cache a stable prefix (even if you only simulate the cache with a hash of the prefix). Overflow must be visible.