AgentOps: validate tool output before a production write

A production runbook for qualifying structured AI agent tool output with schema, sources, diff, idempotence, policy, traces, human validation and rollback before writing to production.

31 Jul 2026 aiagentopsagentstoolsmcpmicrosoft-foundryautomationguardrailsobservabilityevaluationsecurityrunbookrollbackproduction

An AI agent can call the right tool, receive a valid response, and still prepare the wrong production write. The visible failure is not always the risk. The risk is often an output that looks usable: a configuration patch, a list of resources to clean up, a rule to publish, a prefilled change ticket or a rollback command. If that output is not qualified before execution, the team turns a probabilistic suggestion into a production change.

The use case is an operations assistant connected to Microsoft Foundry, an MCP server or an internal orchestrator. It reads runbooks, queries logs, builds a diff and proposes a bounded action: disable a feature flag, adjust a WAF rule, restart a job, update a policy or prepare rollback. The runbook goal is to decide whether tool output can become a write, must remain a draft, must be recomputed, or must be blocked until the contract is fixed.

Freeze Output As A Change Artifact

Treat tool output as a production artifact, not as a chat answer. It should carry intent, scope, sources, diff, idempotency key and rollback plan.

yaml tool-output-change-card.yml

change_candidate:
id: cand-20260731-0842
agent: ops-assistant-prod
conversation_id: conv-20260731-0834
tool: build_waf_policy_patch
transport: mcp
environment: production
requested_intent: reduce a WAF false positive on /partner/comment
proposed_write: add a targeted exclusion in wafpol-app-prod
execution_mode: draft_only

required_fields:
- approved_sources
- proposed_diff
- target_resource
- risk_level
- validation_checks
- idempotency_key
- human_approval
- rollback_patch

hard_stop_when_missing:
- target_resource
- rollback_patch
- source_evidence
- validation_checks

Output without rollback is not an action candidate. It is a hypothesis. It can support a review, but it should not write to production.

Validate Schema Before Meaning

Before deciding whether the proposal is smart, verify that it follows a strict contract. The schema should prevent dangerous free-form fields, ambiguous targets and implicit execution modes.

json production-write-output-schema.json

{
"tool": "build_waf_policy_patch",
"schema_version": "2026-07-31",
"required": [
  "target_resource_id",
  "operation",
  "proposed_diff",
  "evidence",
  "validation",
  "rollback"
],
"allowed_operations": [
  "create_draft_patch",
  "validate_existing_patch"
],
"forbidden_fields": [
  "execute_now",
  "disable_all_rules",
  "unbounded_scope",
  "secret_value",
  "approval_override"
],
"target_constraints": {
  "environment": "production",
  "resource_allowlist_required": true,
  "change_ticket_required": true
}
}

The contract should be enforced by the runtime or tool gateway, not only by the prompt. If the model can produce execute_now, the backend should mechanically reject that field.

Link Every Proposal To Evidence

Operable tool output should explain why it proposes the change. Sources must be identifiable: trace, log, runbook, current policy, Git diff, ticket or human observation. A sentence like “this should fix it” is not enough.

json tool-output-evidence.json

{
"candidate_id": "cand-20260731-0842",
"evidence": [
  {
    "kind": "kql_result",
    "source_id": "waf-blocks-20260731-0800",
    "claim": "rule 942430 blocks only /partner/comment for partner.example.com",
    "time_window": "2026-07-31T07:40:00Z/2026-07-31T08:20:00Z"
  },
  {
    "kind": "current_config",
    "source_id": "wafpol-app-prod@8f42c31",
    "claim": "no existing exclusion covers RequestArgNames comment"
  },
  {
    "kind": "runbook",
    "source_id": "azure-waf-add-targeted-owasp-crs-exclusion",
    "claim": "prefer narrow exclusion over disabling managed rule"
  }
],
"unsupported_claims": []
}

Ask for unsupported claims explicitly. If any remain, the output can stay as a draft, but it should not trigger the action.

Compare Diff Against Intent

The diff is the core validation surface. Output can match the schema and still exceed the original intent: change multiple resources, touch a neighboring environment, remove a rule or modify an unrequested field.

text diff-intent-review.txt

Requested intent
Fix a WAF false positive on partner.example.com /partner/comment
Keep OWASP/CRS in prevention
Do not modify other hosts
Do not add an IP allowlist

Proposed diff
Target: wafpol-app-prod
Add: exclusion for ruleId 942430
Variable: RequestArgNames
Selector: comment
Scope: listener partner.example.com
No prevention/detection mode change
No priority custom rule change

Decision
Compatible with intent if logs prove the same ruleId and field
Reject if output broadens to all hosts, all arguments or the full managed rule set

The right question is not “does the patch work?” It is “does the patch match exactly the proven problem?”

Test Idempotence And Collisions

An agentic write should be replayable or reversible without creating a second effect. Even a configuration patch needs a stable key and a check against current state before application.

yaml idempotency-and-collision-checks.yml

idempotency:
key: wafpol-app-prod:rule-942430:RequestArgNames:comment
safe_when:
  - proposed_patch_already_present_with_same_scope
  - current_config_sha_matches_reviewed_sha
  - no_parallel_change_for_target_resource
unsafe_when:
  - target_policy_changed_after_diff
  - another_candidate_changes_same_rule
  - patch_removes_existing_control
  - rollback_patch_does_not_restore_previous_state

pre_write_checks:
- refresh_current_config
- compare_config_sha
- search_open_change_candidates
- recompute_diff_if_target_changed

If current configuration changed after output generation, the clean decision is to recompute. Applying an old validated diff to a new state creates a change that is hard to explain.

Trace Output, Not Only Execution

Observability should show the proposed output before the write. Otherwise the team sees only the final action and loses the reasoning that turned tool response into change.

kusto agent-tool-output-validation.kql

let CandidateId = "cand-20260731-0842";
AgentToolOutputEvents
| where TimeGenerated > ago(24h)
| where CandidateId == CandidateId
| project TimeGenerated,
        AgentName,
        ConversationId,
        ToolName,
        OutputSchemaVersion,
        TargetResource,
        ProposedOperation,
        EvidenceCount,
        UnsupportedClaimCount,
        DiffHash,
        CurrentConfigHash,
        IdempotencyKey,
        PolicyDecision,
        ApprovalState,
        RollbackHash
| order by TimeGenerated asc

Adapt table names to your telemetry pipeline. The important signal is the transition: output generated, output validated, output approved, write executed, post-change validation completed, rollback ready.

Put Output Through Policy

Policy should reject dangerous writes even when the answer is well written. It should bound operations, resources, environments, mutable fields and approval level.

yaml tool-output-policy.yml

policy: production_tool_output_gate
default: deny
allow_when:
- output_schema_valid
- target_resource_allowlisted
- current_config_hash_matches
- evidence_count_at_least: 2
- unsupported_claim_count: 0
- rollback_patch_present
- human_approval_state: approved

deny_when:
- operation_requests_direct_execution
- diff_touches_unrequested_resource
- output_contains_secret_or_token
- approval_scope_differs_from_target
- rollback_changes_more_than_forward_patch
- validation_checks_missing

write_modes:
draft: allowed_without_approval
plan: approval_required
apply: human_required_and_breakglass_forbidden

This keeps control out of the conversation alone. The model proposes, policy decides, and a human approves sensitive scope.

Decide Publish, Recompute Or Roll Back

Close the runbook with an explicit decision. Validated output is publishable only when diff, evidence, policy and rollback hold together.

text tool-output-decision-card.txt

Publish the write
Schema is valid and versioned
Production target is unique and allowlisted
Evidence is linked to every operational claim
Diff matches the original intent
Current configuration matches the reviewed configuration
Idempotence and collisions are checked
Policy allows and human approval is present
Post-change validation and rollback have been tested

Recompute output
Target configuration changed after the diff
Evidence is incomplete or log window is too old
Concurrent change detected
User intent was reformulated

Block or roll back
Output is outside schema
Direct execution field is present
Scope is broader than the incident
Rollback is missing or riskier than the change
Output cannot be explained from sources

The goal is not to trust the agent. It is to make its proposal operable: traceable, bounded, validatable and reversible. When those conditions are not met, keeping output in draft is a production decision, not a delay.