Context is the hidden lever in most LLM systems. It drives cost and latency, but it also drives quality. When context grows without discipline, behaviour becomes unstable: evidence is truncated, policies are lost, and the model guesses.
Context budgeting is the practice of allocating a fixed token budget across layers and enforcing truncation rules that are observable and safe.
Define a budget per layer
A practical layering model looks like:
- Instructions. System prompt and policy prompts.
- Evidence. Retrieved sources and citations (see grounding).
- Memory. User preferences and session summaries (see assistant memory).
- Tools. Tool outputs and structured results (see tool reliability).
Then allocate a budget to each layer so no single layer consumes the whole context window.
Use deterministic truncation rules
Truncation should be deliberate:
- Evidence-first. Keep top sources and drop duplicates.
- Policy-first. Never truncate safety policies; drop optional memory instead.
- Bound tool output. Tools should return compact, structured results rather than verbose text.
This is also where caching and retrieval hygiene reduce pressure on the budget (see caching and deduplication).
Summarise intentionally
Summaries are useful but can become a source of drift. Treat summarisation as a controlled mechanism:
- Use structured summaries with stable fields.
- Keep references to source IDs so you can trace what was summarised.
- Regenerate summaries when policies or workflows change significantly.
Make budget decisions observable
During incidents, teams need to know whether context was truncated, what was dropped, and why. Capture budget decisions as telemetry and decision logs with reason codes (see telemetry schema and decision logging).
Use budgets as operational guardrails
Context budgets are also cost controls. When spend spikes, one of the fastest stabilisation levers is to reduce token budgets or disable expensive layers temporarily (see cost anomaly detection and change freeze).
Stable quality often comes from simple discipline: fixed budgets, clear truncation rules, and telemetry that makes those choices visible.