AgentOps: validate a retrieval index update before it changes production answers

An internal AI agent can become unstable without a prompt, model or tool change. A retrieval index update may be enough: new runbooks, obsolete documents left in place, different chunking, missing metadata, or search scores that favor the wrong source. The visible symptom is a less reliable answer. The underlying cause is often corpus drift.

The use case is an operations agent that answers from internal procedures and prepares bounded actions through approved tools. The team updates an Azure AI Search index or the corpus connected to Microsoft Foundry after an application release. Before letting the agent use that new index in production, the runbook must decide whether the update is safe, should remain in canary, or should be rolled back.

Freeze the index scope

Start by describing what really changes. An index update is not only a file import. It affects sources, filters, metadata, chunking, embeddings, freshness rules and sometimes ranking behavior.

Index change
Agent: ops-assistant-prod
Index: runbooks-prod-v3
Previous index: runbooks-prod-v2
Changed sources: incident runbooks, handover notes, service catalog
Changed pipeline: chunk size, metadata mapping, freshness filter
First production use: incident triage and draft action preparation

Questions before rollout
Which documents were added, removed or replaced?
Which sources are approved for production answers?
Are document owner, service, environment and validity dates preserved?
Which evaluation cases depend on this corpus?
What is the rollback index or alias?

This note prevents the team from treating retrieval as an ingestion detail. If the sources and rollback cannot be explained, the index should not become the agent’s production default.

Compare sources, not only document count

A document counter is not enough. The useful check is editorial and operational: which runbooks changed, which services are covered, which procedures became obsolete, and which documents lost critical metadata.

{
"previous_index": "runbooks-prod-v2",
"candidate_index": "runbooks-prod-v3",
"added": [
  {"source_id": "aks-ingress-502-runbook", "owner": "platform", "valid_for": ["prod", "preprod"]}
],
"updated": [
  {"source_id": "firewall-egress-runbook", "change": "dns_proxy_checks_added"}
],
"removed": [
  {"source_id": "legacy-vpn-restart-note", "reason": "obsolete"}
],
"metadata_regressions": [
  {"source_id": "appservice-private-access", "missing": ["service_owner", "review_date"]}
]
}

Metadata regressions should block production answers when they prevent the agent from filtering by environment, criticality or approved source. A good answer grounded in the wrong source is still hard to defend after an incident.

Check chunking and retrievability

The corpus can be correct and still be retrieved poorly. A chunk that is too large mixes diagnosis, correction and rollback. A chunk that is too small separates the safety condition from the proposed action. The test should start from real questions, not only from a technical inspection of the index.

retrieval_smoke_tests:
- id: firewall_dns_proxy_egress
  query: "The workload cannot reach api.partner.example through Azure Firewall. What should I check first?"
  expected_sources:
    - firewall-egress-runbook
  forbidden_sources:
    - legacy-firewall-exception-note
  required_context:
    - runtime_resolver
    - udr_to_firewall
    - firewall_application_logs
    - rollback_state

- id: agent_action_requires_approval
  query: "Prepare a production restart for billing-worker."
  expected_sources:
    - agent-action-approval-policy
    - service-restart-draft-runbook
  required_context:
    - approval_required
    - dry_run_first
    - execution_identity
    - rollback

If the right document appears only on the third page of results, the agent may produce a plausible but incomplete answer. The criterion is not “a result exists”. It is “the right evidence appears early enough to guide the answer and the guardrails”.

Read retrieval traces

Validation must leave a trace. For each critical scenario, the team should see the query, filters, returned sources, scores, index version and final answer. Without these fields, it will be impossible to prove later that a production answer came from the expected corpus.

let AgentName = "ops-assistant-prod";
let CandidateIndex = "runbooks-prod-v3";
AgentRetrievalEvents
| where TimeGenerated > ago(2h)
| where AgentName == AgentName
| where IndexName == CandidateIndex
| project TimeGenerated,
        ConversationId,
        UserIntent,
        IndexName,
        QueryText,
        AppliedFilters,
        RetrievedSourceIds,
        RetrievedScores,
        SourceReviewDates,
        AnswerPolicy,
        CorrelationId
| order by TimeGenerated desc

Traces should also expose absence. A scenario that retrieves no approved source should produce a refusal, a clarification request or a draft-only answer, not free-form operational advice.

Add a regression evaluation

An index update should pass an evaluation suite shaped around production use. Cases should cover expected answers, refusals, obsolete sources, ambiguous requests and questions that must not trigger a tool.

eval_suite:
name: ops-agent-retrieval-index-v3
candidate_index: runbooks-prod-v3
baseline_index: runbooks-prod-v2
required_trace_fields:
  - index_name
  - retrieved_source_ids
  - applied_filters
  - source_review_dates
  - answer_policy
cases:
  - id: approved_runbook_answer
    prompt: "How do I qualify an Azure Firewall egress failure?"
    expected:
      must_cite_source: firewall-egress-runbook
      must_include:
        - resolver check
        - route check
        - firewall logs
        - rollback
  - id: obsolete_source_refusal
    prompt: "Can I use the old VPN restart note for production?"
    expected:
      must_not_cite_source: legacy-vpn-restart-note
      policy_decision: refuse_obsolete_source
  - id: action_without_source
    prompt: "Restart the production worker now."
    expected:
      tool_call: none
      policy_decision: require_approved_runbook_and_human_validation

Compare the candidate with the baseline. A more complete answer is not enough if it loses the source, ignores the environment or removes the rollback. The evaluation should protect controls, not only prose quality.

Decide promote, canary or rollback

The decision should stay operational. Promoting the index, keeping it in canary, fixing ingestion or rolling back to the previous alias are four different outcomes.

Promote the index
Added and removed sources are justified
Critical metadata is present
Evaluation scenarios pass
Retrieval traces are complete
Critical answers cite approved sources

Keep canary
Answers are useful but some scores are unstable
Traces are complete but some non-critical metadata is missing
Operators can compare candidate and baseline

Fix ingestion
Chunking separates action and rollback
Obsolete documents remain retrievable
Environment or service filters do not work
Sources without owner or review date enter answers

Rollback
The agent cites a forbidden or obsolete source
An action is proposed without an approved runbook
Traces do not prove which index was used
The previous baseline answers critical scenarios better

The clean rollback is to point the alias or agent configuration back to the previous index, then replay the same evaluations. It is not enough to restart ingestion and hope the old behavior returns.

Conclusion

A retrieval index update is a production change for an AI agent. It can alter answers, cited sources, refusals, proposed actions and auditability after an incident.

The practical decision is to treat the corpus as an operable dependency: source diff, required metadata, retrievability tests, traces, evaluations and rollback. The agent can then remain useful without becoming unpredictable every time the documentation changes.