AgentOps MCP: validate a tool schema change before redeploying the agent

An AI agent can look stable as long as its prompt does not change. In production, that is not enough. If the MCP tool it calls evolves, the agent may keep answering confidently while sending the wrong argument, skipping a confirmation step or triggering an action on a broader scope than intended.

The common failure mode is quiet: a team adds an optional parameter to a diagnostic tool, renames an output field or merges two similar actions. The MCP server tests still pass. The agent still responds. But the operational contract has changed.

This runbook treats an MCP tool schema change as a production change: explicit contract, replayable evaluation, bounded permissions, readable traces, human validation and rollback.

Frame the change before touching the agent

Before redeploying the agent, describe the tool-side change, not only the code diff.

The change note should state:

• the exact MCP tool name being changed;
• fields added, removed, renamed or made mandatory;
• defaults added by the server;
• possible side effects;
• roles or identities allowed to call the tool;
• actions that still require human approval.

This avoids a frequent mistake: assuming that a JSON-compatible change is also compatible for the agent. An optional field can change the agent’s strategy once it appears in the tool description. A richer output can also make the agent conclude too early.

The entry criterion is simple: when the schema, description or semantics of a tool change, the agent must be revalidated.

Separate technical schema from operational contract

The MCP schema describes inputs and outputs. The operational contract describes what the agent is allowed to do with them.

For each tool exposed to the agent, document at least:

Element	Question to answer
Intent	What exact problem does the tool solve?
Inputs	Which fields may the agent provide?
Sensitive data	Which fields must never be inferred or invented?
Effect	Does the tool read, modify, rerun or delete something?
Evidence	Which trace proves the call, parameters and result?
Approval	At which threshold is human validation required?
Rollback	How do we return to the previous tool or server version?

This separation matters for agents that operate runbooks, tickets, pipelines, Azure resources or internal automation. A schema can be valid and still be unsafe if the contract implicitly lets the agent choose the action scope.

Build a small but useful evaluation set

A schema change should not be validated with a single happy conversation. Replay a few situations that cover the real risks.

A useful evaluation set includes at least:

• a nominal case where the agent selects the right tool and fills the right fields;
• an ambiguous case where it must ask for clarification;
• a forbidden case where it must refuse or request approval;
• a missing-evidence case where it must look for proof before acting;
• a rollback case where it explains how to return to the previous state;
• a non-regression case for an older scenario that already worked.

The expected result should not only check the final answer. It should check the path:

• tool called or not called;
• arguments sent;
• identity used;
• evidence read before action;
• confirmation message;
• absence of out-of-scope action.

The key is to capture tool calls, not only the text produced by the agent. An agent can provide a good explanation after calling the wrong tool.

Test in a bounded environment

The test MCP server should be isolated from production or connected only to explicitly non-destructive resources.

In practice, that means:

• a sandbox project, resource group or environment;
• a dedicated identity with production-like limits, but no broad rights;
• representative but non-sensitive data;
• logs enabled on the MCP server, the calling identity and the target;
• a list of deliberately blocked actions to verify refusals.

For a tool acting on Azure DevOps, for example, the agent may read a test project, create a validation comment or rerun a dummy pipeline. It must not be able to modify a critical pipeline during evaluation.

For a tool acting on Azure, the agent may read metadata, check state or trigger a reversible operation on a test resource. It should not receive a broader role just because “it is only a test”.

Review traces before approval

Validation should produce a small evidence pack. It does not need to be heavy, but another person should be able to understand what happened.

Keep:

• MCP server version;
• agent or system prompt version;
• tool schema before and after;
• evaluation prompts;
• tool calls with arguments;
• outputs returned by the server;
• identity or audit logs;
• failed cases and the decision made.

Failures do not always block deployment. They must, however, be qualified. A failure on a rare phrasing can become a documented limitation. A failure on an action refusal or overbroad scope should block.

The validation question is not “does the agent answer well?”. It is: “can we explain and bound what the agent will do when this tool is visible in production?”.

Decide the deployment

Deployment can be accepted if:

• old critical scenarios still pass;
• new fields are used only when the context justifies them;
• the agent asks for clarification when scope is ambiguous;
• sensitive actions remain behind human approval;
• logs can reconstruct the decision;
• rollback has been tested or at least prepared.

Deployment should be refused if:

• the agent invents input values;
• the agent calls the tool on an unrequested scope;
• an irreversible action can be launched without confirmation;
• traces do not show transmitted arguments;
• the previous schema cannot be restored quickly.

A healthy decision can be: “deploy the MCP server, but do not expose the new tool to the agent yet”. That lets the backend ship without immediately increasing autonomy.

Prepare rollback

Rollback must cover two different levels.

The first is technical: return to the previous MCP server version, tool descriptions and configuration. The second is operational: temporarily remove the tool from the allowlist, reduce its permissions or put the action back behind mandatory human validation.

The rollback plan should state:

• previous server version;
• tool disablement mechanism;
• permissions to remove if the tool remains unstable;
• conversations or executions to audit after an incident;
• operator message if the agent is degraded.

In many cases, the fastest rollback is not a full redeployment. It is reducing the tool registry visible to the agent, paired with an operations note.

Conclusion

An MCP schema change is not an integration detail. For an agent that acts on internal systems, it is a production contract change.

The practical reflex is to validate the tool as an operational interface: clear intent, bounded inputs, limited identity, replayable evaluations, usable traces and ready rollback. The agent can then gain capability without becoming opaque.