Tool use is where AI systems become operationally powerful—and where they become dangerous. A chatbot that hallucinates is annoying. An agent that hallucinates while calling tools can create incidents: emails sent to the wrong recipients, refunds issued incorrectly, records changed, or data accessed without authorisation.
The solution is to treat tool calls as proposals, not actions. Prompts can influence behaviour, but they cannot enforce permissions. Authorisation must live outside the model.
Separate “decide” from “do”
A safe architecture has two distinct stages:
- Decision stage. The model proposes a tool call and arguments (ideally as validated structured output).
- Policy stage. A deterministic layer authorises or rejects the action, then executes through a controlled integration.
This is the same security principle used in other systems: untrusted input is evaluated by a trusted policy engine.
Design a least-privilege permission model
Most “agent permissions” problems come from over-broad access. Define permissions explicitly:
- Tool scopes. Which tools can be used for which workflows (read-only vs write).
- Resource boundaries. Which records or domains the tool can access (tenant, region, business unit).
- Actor mapping. The agent should act on behalf of a real user or service identity with clear roles.
- Argument constraints. Allow-lists for destinations, amounts, and other high-risk parameters.
If you do not have clear identity, you cannot have meaningful authorisation. For sensitive environments, pair this with strong data boundaries and residency constraints (see data residency and integration security).
Approvals and step-up controls
Some actions should never be fully autonomous. Use step-up controls:
- Human approval. Require a human to approve high-impact actions (see human-in-the-loop).
- Two-person rules. For sensitive changes (payments, account closures), require two distinct approvals.
- Justification capture. Require a short explanation and supporting evidence (citations, tool output) before approval.
- Rate limits. Limit actions per time window to reduce blast radius if something goes wrong.
Make tools safe by construction
Tool design is security design. Safer tools have properties like:
- Idempotency. Support idempotency keys so retries don’t duplicate actions.
- Dry-run mode. Return what would happen without executing.
- Scoped tokens. Use time-limited capability tokens per action rather than a single long-lived credential.
- Structured outputs. Require schema-validated arguments (see structured outputs).
For more patterns, see safe tooling for LLM agents and defenses against prompt injection, which frequently target tool layers (prompt injection defence).
Audit trails and replayability
For every tool call, log: the actor identity, intent classification (if used), tool name, arguments, authorisation decision, and policy version. This is essential for incident response, regulatory assurance, and debugging (see AI incident response).
Finally, treat changes to tool permissions and policies as controlled releases. If you cannot explain “what changed” between two behaviours, you cannot operate agentic systems at scale.