AgentOps: shadow a prompt or model change before production cutover

A system prompt revision performs better in offline evaluations. A newer model reduces unnecessary refusals. Replacing the active version and watching support tickets may seem like the shortest route. The risk emerges after cutover: real requests are more ambiguous than the test set, retrieval returns different evidence, and a tool receives arguments that nobody exercised before release.

Consider an internal agent used by an operations team to triage incidents. It reads runbooks, queries service metadata and can prepare an action for approval. The current runtime remains responsible for the user-facing response. A candidate receives a controlled copy of the same input, but its output and tool calls must have no operational effect. The runbook must decide whether that candidate is ready for a canary, needs more evaluation or should be rejected.

Version the release bundle, not just the model

An agent release is more than a model deployment name. Behavior also depends on the prompt, inference settings, sources, tool schemas, guardrails and approval policy. Record those components under immutable release identifiers for both the reference and candidate.

reference:
release_id: ops-agent-2026-08-01.2
prompt_sha: <immutable-digest>
model_deployment: <reference-deployment>
retrieval_index: runbooks-2026-08-01
tool_contract: tools-v14
guardrail_policy: guardrails-v9
approval_policy: approvals-v6

candidate:
release_id: ops-agent-2026-08-07.1
prompt_sha: <immutable-digest>
model_deployment: <candidate-deployment>
retrieval_index: runbooks-2026-08-01
tool_contract: tools-v14
guardrail_policy: guardrails-v9
approval_policy: approvals-v6

change_scope:
intended: [system_prompt, model_deployment]
forbidden_drift: [retrieval_index, tool_contract, approval_policy]
rollback_target: ops-agent-2026-08-01.2

When several components change intentionally, treat the combined surface as the release under test. Unplanned drift in the index or tool contract makes the comparison inconclusive: a regression can no longer be attributed to the prompt, the model or a dependency.

Build a shadow boundary with no side effects

Shadow traffic is not a second production path. It is an asynchronous, sampled copy taken after access controls, with an explicit data policy. Only the reference path may return an answer or affect a business system.

Active path
authorized request -> reference runtime -> approved tools -> user response

Shadow path
sampled copy -> minimization/redaction -> candidate runtime
-> read-only retrieval -> simulated or read-only tools -> evaluation and traces

Prohibited
no candidate response delivered to the user
no write to a ticket, job or cloud resource
no shared conversation cache with the active runtime
no unbounded retry
no retention beyond the approved policy

Do not mirror every conversation by default. Sample by scenario family, sensitivity, language, length and expected tool use. Exclude flows that cannot be minimized correctly. Correlation identifiers should connect both runs without copying a user’s full identity into evaluation traces.

There are three defensible tool modes: deterministic simulation from recorded responses, read-only calls against a test environment, or schema validation without execution. A dryRun flag is a control only when the server enforces it. Telling the model not to write is not an execution boundary.

Capture a comparison pair that can be explained

Each shadow run needs enough context to explain divergence without turning the evaluation platform into a replica of production data. The comparison envelope connects a redacted input to release versions, sources, decisions, tools and checks.

{
"comparisonId": "cmp-7f82",
"capturedAt": "<timestamp>",
"scenario": "incident-triage",
"dataClass": "internal-redacted",
"reference": {
  "releaseId": "ops-agent-2026-08-01.2",
  "traceId": "trace-reference",
  "decision": "draft-diagnostic",
  "tools": ["search_runbooks", "read_service_metadata"]
},
"candidate": {
  "releaseId": "ops-agent-2026-08-07.1",
  "traceId": "trace-candidate",
  "decision": "draft-diagnostic",
  "tools": ["search_runbooks", "read_service_metadata"],
  "toolMode": "simulated"
},
"checks": {
  "approvedSourcesOnly": true,
  "forbiddenToolCall": false,
  "approvalBoundaryRespected": true,
  "candidateOutputDelivered": false
}
}

Both runtimes must start from the same logical input. When a dependency can change between executions, pin its version or recorded response. Otherwise, a difference may come from an updated index, a service status change or a nondeterministic tool rather than the candidate itself.

Compare operational decisions before prose

High textual similarity does not prove that two agents made the same decision. Start with operational invariants: approved evidence, tool selection, argument scope, approval request, uncertainty handling and absence of side effects.

blocking_checks:
- candidate_output_reached_user
- write_tool_executed
- forbidden_tool_selected
- approval_boundary_bypassed
- source_outside_approved_scope
- tool_error_reported_as_success

comparative_checks:
- task_outcome
- evidence_quality
- tool_argument_scope
- correct_refusal
- unnecessary_tool_calls
- latency_budget
- token_and_tool_cost

review_rules:
critical_disagreement: mandatory_human_review
evaluator_uncertainty: keep_unresolved
aggregate_score: never_overrides_blocking_check

An automated evaluator can speed up triage, but it should not be the only judge for critical cases. Keep a sample reviewed by operators and oversample disagreements involving the wrong tool, broader scope, a missing refusal or a confident answer built on weak evidence.

Read divergence as a production signal

Averages hide small, high-risk segments. Group results by scenario, language, requested tool, source type and approval policy. A regression isolated to ambiguous requests involving a sensitive action deserves more attention than a small latency increase on documentation questions.

AgentShadowComparisons
| where TimeGenerated > ago(24h)
| where CandidateReleaseId == "ops-agent-2026-08-07.1"
| extend Critical = CandidateOutputDelivered == true
  or WriteToolExecuted == true
  or ForbiddenToolSelected == true
  or ApprovalBoundaryRespected == false
  or ApprovedSourcesOnly == false
| summarize
  Samples = count(),
  CriticalDisagreements = countif(Critical),
  ToolScopeRegressions = countif(ToolArgumentScope == "broader"),
  P95CandidateLatencyMs = percentile(CandidateLatencyMs, 95)
by Scenario, Language, RequestedTool
| order by CriticalDisagreements desc, ToolScopeRegressions desc

AgentShadowComparisons is a normalized example table to adapt to the actual telemetry schema. Monitor the shadow pipeline itself as well. A sudden fall in comparison volume is not an improvement; it may indicate broken duplication, rate limiting or an unavailable evaluator.

Move from shadow traffic to a canary

Shadowing exposes a candidate to real request shapes, but not to real user reactions or every consequence of a tool call. It qualifies the release for a canary; it does not justify a global cutover on its own.

Allow a bounded canary
representative volume reached for every critical scenario
zero blocking violation
sensitive disagreements reviewed and explained
latency and cost within the operating budget
complete reference and candidate traces
rollback bundle remains deployable

Keep shadowing
insufficient volume for an important segment
unstable automated evaluation
unattributable drift between bundles
higher refusal rate still under review

Block
write or user response escaped from the shadow path
forbidden tool call or broader argument scope
approval bypassed
sensitive or unapproved source used
dependency error represented as success

The canary should restrict users, scenarios and tools. Start with read-only cases, retain stronger human approval and raise traffic in explicit stages. At every stage, compare canary traces with the shadow baseline instead of watching only HTTP error rates.

Decide cutover and prepare the complete rollback

Promote the candidate when blocking invariants remain intact, critical segments have enough evidence and the canary confirms behavior with real users. Record the exact promoted bundle identifier and keep a tested route back.

Roll back when a guardrail violation appears, tool scope drifts, traces become incomplete or the canary regresses a critical scenario. Restore prompt, model, inference settings, index, tool contract, guardrails and approval policy together. Disabling only the new model leaves every other possible source of regression in place.

After rollback, replay the failed comparisons against the restored bundle. If they still fail, the candidate change was probably not the only cause. Keep the incident open and isolate the shared dependency before attempting another promotion.

Conclusion

Shadow traffic turns a prompt or model cutover into an observable production decision. It exposes the candidate to real request shapes without allowing it to answer or act, then compares decisions, sources, tools and approvals before writing style.

The runbook ends with a deliberate choice: authorize a bounded canary with evidence, continue shadowing while coverage is incomplete, or block and restore the previous bundle. That discipline lets an agent evolve without confusing average quality gains with operational safety.