23 tools, 4 agents, zero surprises

For ninety days we ran one production MCP server — 23 tools curated from a 214-endpoint catalog — for four agents doing real work, and nothing surprised us. Not zero incidents; zero surprises. Every anomaly in that window was attributed to a specific agent and key within minutes, every latency regression was caught by a budget alert before an agent timed out on it, and the one breaking schema change that could have hurt was blocked at publish. This post is the accounting of what that took, with the numbers.

The setup

The server fronts a payments and identity catalog and speaks MCP to four callers: a customer-support copilot, an ops runbook agent, a finance reconciliation agent that runs nightly, and whichever engineer has Claude pointed at staging that week. Each agent authenticates with its own scoped key — support can read orders but never move money, finance can reconcile but not refund. Traffic averaged 38,000 tool calls a week, spiking to 9,000 a day at month-end when reconciliation runs widen. The 23 tools are the survivors of an admission test: nothing ships to the server below a B agent-readiness grade, whatever the catalog holds. The mix is deliberately lopsided — 14 read tools, 6 writes gated behind scoped keys, and 3 search tools with tight output budgets — because reads are where agents live and writes are where incidents do. Every tool name states its object and verb (get_payout, search_orders), a convention that sounds cosmetic until you watch a model choose between twenty-three of them.

Ninety days in numbers

The quarter's totals, for calibration: 489,000 tool calls, 99.2% of them successful on first attempt. p95 latency across all tools held at 410ms after the pagination fix described below, p99 at 1.4 seconds. Retries were 5.8% of traffic at the start of the window and 1.9% at the end, almost entirely from error-message improvements rather than reliability changes. The support copilot drove 61% of call volume but only 9% of token cost — short reads — while the finance agent inverted that: 7% of calls, 38% of tokens. Peak sustained load was 41 calls a second during a month-end reconciliation run. No number here is remarkable, which is the point. Remarkable numbers in agent traffic are usually a bug wearing a costume.

Attribution before anything else

The first dashboard panel is requests per agent, and it earned that position in week one. A Tuesday spike tripled baseline traffic; the per-agent view showed the ops agent alone climbing, the per-tool view narrowed it to get_incident, and the argument log showed the same incident ID on repeat — a retry loop triggered by a malformed timestamp in one record. Four minutes from alert to root cause, no grepping, and the fix was a data correction rather than a rollback. Without attribution that morning is an hour of guessing which of four callers went feral. The general rule held all quarter: one agent doing 80% of traffic is either your best integration or a bug, and the dashboard's job is to tell you which before you have to ask.

Keys are the unit of trust

Attribution only works because identity is enforced at the key, and keys are boring on purpose: one key per agent, scoped to the minimum tool set, rotated on a 30-day schedule, revocable in one click. The interesting case is the fourth "agent" — whichever engineer has Claude pointed at staging that week. Those sessions use short-lived keys minted for 24 hours, so experimentation stays cheap and cleanup is automatic; an experiment that would have become permanent shadow traffic under a shared key simply expires instead. The rule that made governance real: no shared keys, ever, including for "just testing." Every incident story that begins with an unattributable call ends with someone admitting a shared key existed.

Latency is a retry multiplier

We budget every tool at 800ms p95, and the reason is not user experience — it is arithmetic. Agents cancel slow calls and retry them, so a slow tool does not just serve requests late, it manufactures new ones. Our worst offender was list_transactions, which shipped unpaginated: 2.9s p95, 5,800 output tokens per call, and agents retrying it enough to make it 31% of server traffic. Pagination plus a field projection took it to 340ms and about 900 tokens, and total server traffic dropped 22% with no behavior change from any agent — the retries simply stopped. The token column matters as much as the milliseconds: verbose outputs are a tax on every conversation turn, and the bill lands on your callers.

Watching the schema gap

The near-miss that justified the whole setup: an upstream refactor renamed settledAt to settlementTime in the transaction payload. The finance agent branches on that field nightly. The contract diff caught the rename at publish time, classified it as breaking against the agent contract, and blocked it pending approval — so what actually happened was a rename PR plus a deprecation window, and the agent never saw an inconsistent payload. The counterfactual is the ugly kind of failure: no error, no alert, just a reconciliation agent silently treating every transaction as unsettled and reporting success. Schema drift in agent systems fails quiet, not loud. The only defense we trust is the gate, plus tool-level error-rate panels watching for the drift that sneaks in anyway — a validation-error rate that ticks up on one tool after a deploy is drift's tell, and it has paged us once this quarter, correctly. The gate is not bureaucracy; the rename shipped four days later with a migration note. What it bought was the difference between a scheduled change and a silent one.

What the dashboard shows

Four panels, in the order we look at them. Requests per agent, for attribution. A tool latency table — p95 and p99 per tool with budget markers — because averages hide the one slow tool an agent hammers. Error rate per tool, split by error class, since a validation-error spike means a schema problem while a 5xx spike means an upstream one, and the fixes have different owners. Token cost per call, per tool, because that number is invisible in conventional API monitoring and it is the one your callers' bills are made of. Everything is filterable by key, so "what did the support copilot do between 2 and 3am" is a query, not an investigation.

The panel we deleted

The first version of the dashboard had an aggregate requests-per-minute chart, inherited from web-monitoring habit, and it survived three weeks. Aggregate RPM answers a question nobody was asking — total load was never the risk — while smearing the signal that mattered across four callers until it disappeared. A 3× spike from one agent reads as a 40% bump in aggregate, which alerts nobody. Its replacement, per-agent small multiples, is what caught the retry loop in week one. The lesson generalized: every panel on the final dashboard answers "which caller, which tool," and anything that cannot be sliced that way is decoration. Observability for agent traffic is attribution first, volume second.

The boring checklist

Zero surprises came from unexciting rules, applied without exception:

• Every tool has a named owner; no owner, no deploy
• Every key is scoped to the least it needs, and rotated on a schedule
• Every latency budget has an alert, and alerts page the tool's owner, not a channel
• Every schema change diffs against the agent contract before publish, no fast path
• Every month, the toolbox is reviewed — we removed three tools this quarter that no agent had called in 30 days

Twenty-three tools, four agents, ninety days, and the most eventful page in the runbook was the one about the retry loop. That is the goal. Infrastructure is supposed to be boring, and now the tooling exists to make MCP servers boring too.