AgentOps: diagnose MCP tool shadowing before a production action
A production runbook for detecting MCP tool collisions, proving which tool the agent actually selected and validating a canary catalog before any sensitive action.
Read article
Tag
evaluation
32 articles connected to this technical signal.
AgentOps: validate a fallback model before enabling multi-model routing
A production runbook for proving that an agent fallback model preserves structured outputs, tool boundaries, refusals and traceability before routing live traffic.
Read articleAgentOps: neutralize indirect prompt injection in MCP tool output before a production action
A production runbook for isolating hostile instructions carried by MCP tool output, preserving provenance, enforcing policy outside the model, and validating or rolling back write access.
Read articleAgentOps: calibrate the evaluator before trusting an agent quality gate
A production runbook for separating agent regression from evaluator drift by freezing outputs, replaying an adjudicated anchor set, measuring disagreement and keeping promotion rollbackable.
Read articleMicrosoft Foundry: diagnose a content-filter spike before weakening guardrails
A production runbook for separating blocked prompts, filtered outputs, application errors and policy drift before changing Microsoft Foundry guardrails.
Read articleAgentOps: invalidate stale evidence before a production action
A production runbook for bounding evidence age across retrieval, telemetry and tool results before an AI agent proposes, executes or rolls back an operational action.
Read articleAgentOps: validate regional failover before rerouting a production agent
A production runbook for qualifying an AI agent regional failover across model deployment, state, retrieval, tools, identity, traces, canary traffic, validation and rollback.
Read articleAgentOps: revalidate a stale approval before resuming a production action
A production runbook for detecting and revalidating stale human approval across state drift, request fingerprints, expiry, traces, refusal tests and rollback before an AI agent resumes an action.
Read articleAgentOps: validate session and memory isolation before production
A production runbook for detecting context leaks across AI agent sessions, separating memory, caches, retrieval, tools and identity, then enabling or rolling back persistence without losing audit traces.
Read articleAgentOps: diagnose context-window saturation before raising token limits
A production runbook for reconstructing an AI agent's context, isolating history, retrieval and tool outputs, then validating compaction or rolling back.
Read articleAgentOps: detect evaluation set contamination before promoting an agent
A production runbook for proving that an AgentOps score is not inflated by leakage across evaluations, prompts, retrieval or tuning data before promoting an AI agent.
Read articleMicrosoft Foundry Toolbox: validate a version before promoting it to agents
A production runbook to test a Toolbox version, its MCP tools, identities, approvals and traces before promotion, then return to the previous version without redeploying agents.
Read articleAgentOps: shadow a prompt or model change before production cutover
A production runbook for comparing a candidate agent with the active runtime on the same requests without duplicating actions, then deciding promotion, canary or rollback.
Read articleAgentOps: bound MCP tool timeouts, retries and circuit breakers before production
A production runbook for defining latency budgets, idempotency, retries, circuit breakers, traces and rollback for an MCP tool called by an AI agent.
Read articleAgentOps: validate tool output before a production write
A production runbook for qualifying structured AI agent tool output with schema, sources, diff, idempotence, policy, traces, human validation and rollback before writing to production.
Read articleAgentOps: diagnose MCP server drift before a production action
A production runbook for qualifying MCP server drift with tool manifests, schemas, identity, secrets, network path, traces, evaluations, validation and rollback before reauthorizing an AI agent.
Read articleAgentOps: scope an internal MCP server before exposing it to an agent
A production runbook for qualifying an internal MCP server with tool inventory, scopes, identities, secrets, audit, dry runs, evaluations, human validation and rollback before agent access.
Read articleAgentOps: diagnose a failed tool call before retrying a production action
A production runbook for qualifying a failed AI agent tool call with trace evidence, idempotence, identity, backend state, approvals, validation and rollback before retrying.
Read articleAgentOps: validate agent memory before production actions
A production runbook for qualifying AI agent memory with sources, traces, aging, permissions, evaluations, guardrails, human validation and rollback before it influences real actions.
Read articleAgentOps: validate an evaluation set before promoting an AI agent
A production runbook for qualifying an AI agent evaluation set with business cases, retrieval, tool calls, traces, thresholds, human validation and rollback before promotion.
Read articleMicrosoft Foundry: validate guardrails before exposing an agent to production
A production runbook for qualifying Microsoft Foundry agent guardrails with sources, refusals, tools, identity, traces, canary, human validation and rollback before user exposure.
Read articleAgentOps: diagnose prompt injection in retrieval before production actions
A production runbook for qualifying suspected prompt injection in an AI agent retrieval corpus with sources, traces, evaluation, tools, guardrails, validation and rollback.
Read articleAgentOps: validate an approval policy before production actions
A production runbook for qualifying an agent approval policy change with action scope, identity, traces, evaluation cases, guardrails, human validation and rollback.
Read articleAgentOps: diagnose Azure OpenAI rate limits before changing models
A production runbook for qualifying Azure OpenAI or Microsoft Foundry throttling with quota, deployment capacity, agent traces, retries, fallback, validation and rollback before changing models.
Read articleAgentOps: rotate an agent runtime identity before tool access fails
A production runbook for rotating or revoking an AI agent runtime identity with scoped permissions, dry-run tool calls, traces, approvals, validation and rollback before breaking production actions.
Read articleAgentOps: validate an agent tool before it can change production
A production runbook for qualifying a new AI agent tool with contract review, scoped identity, dry run, traces, approvals, evaluation cases and rollback before enabling real actions.
Read articleAgentOps: validate a retrieval index update before it changes production answers
A production runbook for qualifying a retrieval index update with source diff, metadata, chunking, evaluations, traces, human validation and rollback before changing an AI agent's answers.
Read articleAgentOps: diagnose missing agent traces before restoring a production action
A production runbook for qualifying incomplete AI agent traces with conversation events, sources, tool calls, identities, approvals, evaluations and rollback before restoring an action.
Read articleAgentOps: diagnose agent contract drift before redeployment
A production runbook for qualifying AI agent contract drift with prompts, tool manifests, sources, evaluations, traces, human validation and rollback.
Read articleMicrosoft Foundry: evaluate an agent before giving it a production action
A runbook for validating an AI agent before real action by separating sources, tools, identity, evaluation cases, human approvals, logs and rollback.
Read articleAgentOps: expose MCP tools without losing control of production actions
A production runbook for exposing MCP tools to an AI agent while keeping action scope, approvals, identities, logs, evaluation and rollback under control.
Read articleAgentOps: diagnose an AI agent that calls the wrong tool
A production runbook for qualifying an AI agent that selects the wrong tool, acts without evidence or hides an action behind a plausible answer.
Read article