AI features expand your secret surface area. A single LLM product can touch model providers, vector databases, ticketing systems, payments, CRM, and internal APIs. Every integration introduces credentials, and “just put the key in an env var” turns into secrets sprawl fast.
Start by inventorying secret types: provider API keys, database credentials, OAuth client secrets, service-to-service tokens, webhook signing keys, and encryption keys. Then decide which secrets should ever exist in an application runtime, and which should be exchanged dynamically via identity.
The safest posture is simple: treat the model as an untrusted external service and keep credentials out of the model context entirely. Combine this with data residency and integration security so secrets never leak across boundaries.
Prefer workload identity over shared keys
Replace static API keys with workload identity and short-lived tokens. Issue credentials per service, per environment and, where possible, per action. A token that can “read customer profile” should not also be able to “issue refund”. Keep the blast radius small and revoke quickly when something looks wrong.
Where user data access depends on the end-user, propagate user identity separately from service identity. Do not “borrow” a privileged service token to perform user-scoped actions; that pattern collapses auditability and makes least privilege impossible.
Centralise storage, automate rotation
Store secrets in a managed secrets service backed by KMS. Encrypt at rest and in transit, enforce least-privilege access, and log every read with service identity and reason. Rotation should be automatic: if a secret cannot rotate without a maintenance window, it is a reliability risk as well as a security risk.
- Separate planes. Keep prompt templates, embeddings and logs in the app plane; keep secrets and keys in the control plane. Do not replicate secrets cross-region unless explicitly required.
- Use egress controls. Route outbound calls through an API gateway or service mesh that can inject identity, enforce allowlists, and redact sensitive headers from logs.
- Harden the tool layer. Tools should run with scoped credentials, validate parameters, and refuse unsafe operations. Treat tool outputs as untrusted input back into the model.
- Redact and monitor. Add log scrubbing for tokens and PII, and alert on unusual secret access patterns (new caller, high volume, off-hours).
Include build-time controls too: secret scanning in repositories, pre-commit hooks, and separate dev/test credentials that cannot reach production data. Avoid “developer convenience” exceptions that quietly become permanent and expand blast radius.
Apply least privilege to observability as well. Separate logging sinks per environment carefully, avoid storing prompts where not necessary, and ensure debugging views are access-controlled and audited.
Common pitfalls to avoid
Two mistakes show up repeatedly: putting sensitive URLs or keys into prompts “so the agent can use them”, and logging raw prompts/responses without redaction. Both create durable leakage paths. Prefer a tool proxy that holds credentials and exposes only safe operations to the model.
- Environment drift. Accidentally using production secrets in staging, or letting developers copy keys into local scripts.
- Header leakage. Logging request headers that contain bearer tokens or signed URLs.
- Over-broad tokens. One token that can do everything, making revocation and audit almost meaningless.
- Non-rotating webhooks. Long-lived webhook secrets that never rotate because “nothing depends on them”.
Prove rotation works by running game days: expire tokens, revoke credentials and ensure systems fail safely (graceful degradation, clear errors, no silent retries). For compliance-heavy environments, tie this to change control and regular access reviews—see Compliance by Design.
Secrets management is not separate from AI engineering; it is foundational. The more tools you connect, the more important it becomes to make identity, rotation and auditability part of your platform, not an afterthought.