LLM systems rarely fail cleanly. They degrade: latency climbs, rate limits trigger, tool dependencies time out, and quality drifts. If you do not design explicit degradation modes, the system will invent one for you, usually the worst one: slow, expensive, and unpredictable.
A fallback strategy is a set of predefined modes that keep the service usable and safe when dependencies are stressed.
Design fallbacks around user intent
Not every request needs the same quality bar. A common pattern is tiered service:
- Critical workflows. Use the best route and strict guardrails.
- Standard requests. Use a balanced route with cost controls.
- Non-critical requests. Use cheaper models or cached responses.
Tiering aligns with feature flags and canary strategies (see feature flags and canary rollouts).
Fallback type 1: route switching and model fallback
Route switching is the default fallback. Examples:
- Switch to a backup provider or region on failure.
- Switch to a smaller model for high-load periods.
- Disable optional tools or long context for a "basic mode".
To avoid regressions, tie routing changes to versioned configuration and audit logs (see configuration drift).
Fallback type 2: reduced-context and safer defaults
When tokens are expensive or slow, reduce context. Degradation modes can include:
- Smaller context window. Keep only the latest turns.
- Retrieval cap. Retrieve fewer documents and require citations.
- Abstain more. Ask clarifying questions or abstain when evidence is weak (see answerability gates and refusal calibration).
In degraded mode, it is usually better to be more conservative than to guess.
Fallback type 3: caching and partial automation
Caching can keep systems responsive when providers are slow. Use caching carefully:
- Cache only when it is safe for the data classification and tenant model.
- Include entitlement context in cache keys to prevent leakage.
- Prefer caching deterministic intermediate results (retrieval results, tool lookups) over full answers.
See safe caching strategies for guardrails.
Fallback type 4: human handoff and approvals
When the system cannot complete safely, hand off cleanly:
- Provide a structured summary of what was attempted and what is missing.
- Escalate high-risk actions to approvals (see approvals).
- Route to human review queues for policy-sensitive work (see human review ops).
Operationalise degradation with SLOs and runbooks
Fallbacks work only when operators know when to use them. Define:
- Trigger conditions (p95 latency, 429 rate, tool timeout rate).
- Which mode is active and how it changes behaviour.
- Rollback steps and communications templates.
This is standard service operations applied to AI systems (see SLOs and alerting and runbooks).
Fallbacks are not a compromise. They are a design feature that makes LLM systems predictable under stress. Users prefer a clear, safe degraded mode over a slow and erratic system that sometimes fails silently.