AI features change faster than traditional software. Model updates, prompt improvements, and new data sources can all shift outcomes. Feature flags provide controlled change, safety switches, and measurable rollouts.
What to put behind a flag
In AI systems, feature flags should cover more than UI toggles:
- Prompt templates and policies. Ship prompt updates as versioned releases (see prompt registry).
- Model/provider selection. Switch models for specific tenants or workflows.
- Tool access. Enable or disable tool calling per role or environment (see tool authorisation).
- Retrieval and ranking. Turn on reranking or larger retrieval sets only for high-value intents.
- Fallback behaviour. Route to a safer mode when confidence or policy checks fail.
Rollout patterns that work for AI
Use rollout patterns that control both quality and risk:
- Canary by workflow. Start with a low-risk workflow and expand (see canary rollouts).
- Rings by tenant or role. Pilot with internal users, then trusted customers, then broad rollout.
- Auto-rollback triggers. Roll back on spikes in refusal rate, tool errors, or incident signals.
Safety switches and degradations
Flags also support fast recovery. Maintain a kill switch that disables high-risk capabilities (tool use, autonomous actions) and degrades to a safer mode. For some workflows, a human fallback is the right safety net (see human-in-the-loop). For provider outages or latency spikes, route to alternative models or a minimal response mode (see routing and failover).
Measure experiments, not vibes
A rollout without measurement is a gamble. Define metrics and guardrails before enabling a flag: task completion, escalation, groundedness, cost per task, and incident rate (see usage analytics and value metrics). For quality changes, use evaluation rubrics and replay harnesses (see evaluation rubrics).
Govern flags like production controls
Feature flags are operational controls. Treat them as auditable: record who changed what, when, and why. For higher-risk changes, require approvals and keep stabilisation playbooks ready (see approvals and change freeze).
With the right discipline, feature flags turn AI delivery into an observable, reversible process instead of a series of high-stakes releases.