Scoring agent readiness: what the A–E grade actually measures

The A–E grade answers one question: if you handed this API to a capable agent with no human in the loop, how often would calls succeed for the right reasons? It is computed from four weighted components measured directly on the API catalog — description coverage at 35%, schema completeness at 30%, auth clarity at 20%, and operational metadata at 15% — with hard caps so a strong score in one area cannot paper over a hole in another. An A is 90 or above after caps. An E means an agent calling this API is mostly guessing.

What the grade is for

The grade exists to make one decision cheap: does this endpoint go in front of an agent or not. Before scoring, that decision was a vibe — someone eyeballed the spec, decided it looked fine, and the agent discovered the gaps in production. After scoring, it is a threshold. Most teams we host gate their MCP servers at B: endpoints below it stay in the catalog, visible and improvable, but are not exposed as tools until they clear the bar. The second job is comparability. "Payments is an A, billing is a D" is a sentence an engineering manager can act on without opening either spec, and trend lines per service turn API quality from a one-time cleanup into something a team can actually hold.

Description coverage: 35%

Coverage counts three levels — the operation, every parameter, and every response field — and then checks quality, not just presence. A description that restates the field name (id: the id) scores zero. So does one that is duplicated across six fields, because a description that does not discriminate does not help a model choose. Length, specificity, and distinctiveness all feed the check. The median repo starts at 41% meaningful coverage, and it is almost never uniform: request sides are documented, response fields are deserts. The weight is the largest because descriptions are the only part of your API a model reads the way a human reads docs — over MCP, the description field is effectively the tool's user interface.

Schema completeness: 30%

Completeness asks whether the schema constrains reality or merely gestures at it. Typed fields instead of bare string; enums where values are finite; required arrays that match the handler's behavior; additionalProperties decided rather than defaulted; response schemas declared per status code, not just for 200. Each miss widens the space of payloads an agent can construct that your API will reject — or worse, accept and misinterpret. The scoring here is mechanical against the OpenAPI 3.1 document, which is deliberate: it means the same fix that raises the grade also tightens generated tests and MCP tool schemas, because all three read the same source.

Auth clarity: 20%

Auth is scored on three questions. Is a security scheme declared on every operation that enforces one? Are scopes named and documented? Is the token acquisition path — OAuth 2.0 flow, API key issuance, whatever it is — described somewhere a consumer can find? Undeclared auth is the most punishing gap in agent traffic. A human meeting an undocumented 401 reads the error page; an agent meets it, retries, tries a different tool, retries again, and burns a conversation. When Elva's repo analysis finds endpoints whose middleware enforces auth the spec never mentions — four per repo is typical — closing that gap is usually the single cheapest grade improvement available.

Operational metadata: 15%

The smallest weight, but capped: without a machine-readable error shape the grade cannot exceed C, whatever the rest scores. This component checks the unglamorous fields agents branch on. Error responses structured along the lines of RFC 9457, so failures carry a type and a pointer to the offending field. Pagination declared where lists can grow. Rate limits stated rather than discovered. Idempotency documented consistently with RFC 9110's method semantics, so a retry after a timeout is a safe decision instead of a gamble. None of it is exciting. All of it is what an unattended caller actually needs at 3am.

The caps, precisely

Weighted sums hide holes, so four caps override the arithmetic:

• No machine-readable error shape anywhere → the grade tops out at C
• Any state-changing endpoint with undeclared auth → tops out at C
• Meaningful description coverage below 20% → tops out at D
• Any exposed endpoint whose response schema is entirely undeclared → tops out at D

A service can score 88 on the components and still wear a C, and that is the point. The capped areas are the ones where a single gap produces silent wrong behavior rather than a visible error, and silent wrong behavior is the failure class agents amplify. The dashboard always shows both numbers — the weighted score and the cap that bound it — so the fastest route to the next letter is never a mystery.

Service grades weight the worst

A service grade is not the average of its endpoint grades. Averages flatter: a service with forty A endpoints and six E endpoints averages comfortably into a B while remaining a trap, because an agent plans a workflow across whatever endpoints the task requires, and one E in the chain sinks the run. So the service roll-up weights the worst quartile at half the total, which drags the six Es into the light. Teams occasionally object that this feels punitive — then a workflow eval fails on exactly the endpoint the roll-up was pointing at, and the objection retires. If you want the service letter to move, fix the floor, not the ceiling. It is usually cheaper anyway.

Why these weights

The weights come from failure attribution, not intuition. We run an evaluation harness — around 600 tasks across 40 APIs in varied states of documentation — where an agent must complete multi-step workflows unassisted, and we tag every failed run with its proximate cause. Bad or missing descriptions account for just over half of failures: the agent picks the wrong tool or fills a parameter with a plausible wrong value. Schema gaps account for roughly a quarter. Auth confusion and missing operational metadata split the rest. Reweighting the same measurements to match observed failure share is the whole methodology, re-run as models improve. The honest footnote: weights shift a few points between model generations, so we version the rubric and label every grade with it.

What the eval harness looks like

Attribution needs ground truth, so the harness runs workflows, not trivia. Each task hands the agent a goal — reconcile these two invoices, find and cancel the duplicate order — plus an MCP server generated from the API under test, and success is judged on end state, not on transcript vibes. Every failure gets a taxonomy label: wrong tool chosen, right tool with wrong arguments, auth dead-end, retry loop, premature success report. The model is held fixed within a rubric version, and the full suite reruns when either changes. It is the least glamorous system we maintain and the only reason the weights are numbers instead of opinions; when someone proposes changing the rubric, the required argument is a harness delta, not a slide.

What the grade refuses to measure

A grade can only see the catalog, so there are things it deliberately does not claim. It does not know whether your API is semantically correct — a perfectly described endpoint that returns wrong numbers grades an A and fails your users. It does not measure uptime or latency; observability owns that. And it cannot detect a schema that is precise but lies, which is why grades pair with generated tests that call the endpoints and check the answers. Treat the grade as necessary, not sufficient: it measures whether an agent can understand your API, and only the test suite measures whether it should believe it.

How scores move in practice

The improvement curve is front-loaded. Week one is auth declarations and error shapes: config-level changes, typically worth 10 to 15 points. Weeks two and three are the description pass — AI-drafted, human-reviewed at roughly 200 fields an hour — which is where D services cross into B territory. The long tail is schema tightening, because bare strings become enums only when someone confirms the value set with the owning team, and that is a conversation, not a commit. Plateaus are diagnostic too. A service stuck at B usually has structural debt — polymorphic responses, undocumented pagination — that scoring can name but only refactoring can fix. Teams that hold a weekly half-hour review of score deltas keep climbing; teams that check quarterly regress between checks.

The grade travels with the contract

Because a contract is a filtered view of the catalog, it gets its own grade, computed only over what that audience can see. The same service can be a B internally and an A through the partner contract, if the partner view exposes only the curated, well-described subset — which is exactly the outcome curation is supposed to buy. This is also how MCP gating works in practice: the agent contract's grade is the one that matters for tool export, so teams polish the twenty endpoints agents actually touch without first paying down documentation debt on two hundred internal ones. Scope discipline turns "make the API agent-ready" from a quarter-long program into a sprint with a visible finish line.

Reading your first report

First reports almost always show high variance — an A service next to a D built by the same team two years earlier. Fix order matters less than teams expect, but the cheap wins are consistent. Declare the auth that middleware already enforces: hours of work, immediate points. Generate description drafts with the AI agent and human-review them, which typically moves coverage from 40% to 85% in a week. Leave schema tightening for last, since it can require code changes. The pattern to expect is the one in our own demo: billing rescored from D 61% to B 83% in a single sprint, and nothing about the API's behavior changed — only how honestly it was described.