OpenTelemetry Collector: diagnose telemetry drops before adding memory

A dashboard develops gaps after a traffic peak. Traces arrive intermittently, log volume falls, and the Collector restarts once or twice. The immediate proposal is often to raise the memory limit. That may postpone the next failure, but it does not establish whether data was refused at the receiver, dropped by a processor, stranded in an exporter queue, rejected by the backend, or lost during a restart.

The use case is an OpenTelemetry Collector running as a Kubernetes gateway between applications and one or more remote observability backends. The runbook goal is to locate the first measurable loss, contain the affected pipeline, and decide whether to tune capacity, correct the downstream path, scale out, or restore the previous configuration.

Freeze the pipeline and incident window

Do not start from the pod that used the most memory. Record the route followed by the missing signal and the versions active during the gap.

incident:
start: <timestamp>
end: <timestamp>
missing_signal: traces
affected_services:
  - checkout-api

pipeline:
clients: application SDKs
receiver: otlp
processors:
  - memory_limiter
  - resource
  - batch
exporter: otlphttp/primary
backend: <observability-backend>

versions:
collector_image: <immutable-image-reference>
configuration: <config-version>
deployment: <deployment-revision>

evidence:
- collector internal metrics
- collector logs and restart events
- backend ingestion status
- client export errors
- configuration and rollout timeline

Separate traces, metrics and logs even if they use the same receiver and exporter. They can have different rates, batch sizes, processor chains and failure behavior.

Find the first counter that diverges

Follow the pipeline in order. Compare accepted and refused items at the receiver, processor drops, exporter sends and failures, then queue occupancy. Metric names and labels can vary with the Collector version and distribution, so inventory the endpoint before writing a permanent query.

# Receiver pressure
sum by (receiver) (rate(otelcol_receiver_refused_spans[5m]))

# Explicit processor drops, when exposed by the processor
sum by (processor) (rate(otelcol_processor_dropped_spans[5m]))

# Export failures
sum by (exporter) (rate(otelcol_exporter_send_failed_spans[5m]))

# Queue saturation
max by (exporter) (
otelcol_exporter_queue_size
/ clamp_min(otelcol_exporter_queue_capacity, 1)
)

Use equivalent item counters for metrics and logs. A rising receiver refusal points upstream of processing. Stable acceptance followed by processor drops points to an explicit policy or resource protection. Export failures with a filling queue point downstream: DNS, TLS, authentication, throttling, backend availability or insufficient exporter throughput.

Correlate loss with resource pressure

Collector memory alone is not a diagnosis. Put queue fill, refusal rates, CPU throttling, resident memory, garbage collection pressure and restarts on the same timeline.

Queue rises, exporter failures rise, memory follows
Diagnose the downstream destination and exporter throughput first

Receiver refusals rise while the memory limiter is active
Confirm sustained load, memory budget and upstream retry behavior

Pod restarts without refusal or exporter evidence
Check OOMKilled, probes, node pressure and missing internal telemetry

Backend volume falls but Collector counters remain balanced
Check backend ingestion, indexing, query scope and retention

Only one service is missing
Check that SDK, endpoint, sampling and resource attributes before resizing the gateway

The important relationship is temporal. A queue that fills before memory rises suggests buffered downstream pressure. Memory rising before queues move suggests processing cost, cardinality, oversized batches or another in-process component.

Prove the downstream path

Exporter errors need to be classified, not counted as one family. Read status codes and transport errors alongside DNS, TLS and authentication evidence. A larger queue cannot repair a wrong certificate, expired credential, blocked egress path or backend throttle.

Use a debug exporter only on a bounded diagnostic pipeline or canary. It can prove that the Collector receives and processes a sample, but enabling verbose payload output across production traffic can expose sensitive telemetry and create additional load.

exporters:
debug/incident:
  verbosity: basic

service:
pipelines:
  traces/incident:
    receivers: [otlp/incident]
    processors: [memory_limiter, batch]
    exporters: [debug/incident]

operating_rule:
scope: isolated test receiver and synthetic trace only
duration: one diagnostic window
forbidden: production payload dump

The operating_rule section is an operational note, not Collector configuration. Keep it in the change record rather than the deployed configuration.

Size queues from a failure budget

Do not enlarge a queue because it reached 100%. Derive its capacity from the accepted input rate, average request size, tolerable backend outage, available memory or persistent storage, and recovery throughput.

Inputs
accepted items or requests per second
average and high-percentile request size
maximum downstream outage to absorb
memory or disk budget available to the Collector
export throughput after recovery

Validation questions
Can the queue absorb the intended outage window?
Can the exporter drain faster than new data arrives?
Does retry expiry exceed the supported outage window?
Does the queue fit below the Collector resource limit?
Is data loss acceptable when the failure budget is exceeded?

An in-memory queue trades memory for short-term resilience and is emptied by a restart. A persistent queue changes the recovery and storage failure modes; it must be tested with the actual storage extension, volume and shutdown behavior. Neither option removes the need for a bounded retry horizon.

Change one bottleneck at a time

A defensible configuration change should name the diagnosed boundary. If the backend was unavailable, correct reachability or authentication before changing memory. If export throughput is lower than steady input, validate exporter concurrency or scale out. If the memory limiter refuses traffic under expected load, recalculate the pod and limiter budgets together.

processors:
memory_limiter:
  check_interval: <tested-interval>
  limit_mib: <derived-hard-limit>
  spike_limit_mib: <derived-spike-budget>
batch:
  send_batch_size: <measured-batch-size>
  timeout: <tested-timeout>

exporters:
otlphttp/primary:
  endpoint: <backend-endpoint>
  sending_queue:
    enabled: true
    queue_size: <derived-capacity>
  retry_on_failure:
    enabled: true
    max_elapsed_time: <supported-outage-window>

Place resource protection early in the processor chain and keep batching after filtering or transformations. Confirm the exact schema supported by the deployed Collector version before rollout; component options evolve.

Canary the correction with loss accounting

Send a synthetic, uniquely identifiable stream through one canary Collector. Increase load gradually and inject a bounded downstream failure. The test must observe both telemetry completion and Collector stability.

canary:
signals: [traces, metrics, logs]
stages:
  - steady_baseline
  - expected_peak
  - bounded_backend_failure
  - recovery_and_queue_drain

acceptance:
receiver_refusals: zero_during_supported_load
unexplained_processor_drops: zero
queue: drains_after_backend_recovery
exporter_failures: return_to_baseline
restarts: zero
backend_count: reconciled_with_sent_synthetic_items
recovery: faster_than_new_input

negative_checks:
- unsupported_outage_exceeds_budget_and_emits_clear_loss_signal
- debug_exporter_does_not_receive_production_traffic
- previous_configuration_remains_deployable

Reconcile counts using synthetic traffic, not sampled business traces. Sampling and filtering are intentional loss boundaries and must be excluded from an end-to-end conservation check.

Decide and roll back

Keep the change when the first loss boundary is removed under expected peak load, the queue drains after recovery, memory stays within the tested budget, and the backend receives the synthetic stream. Scale out when throughput, rather than transient outage absorption, is the limiting factor. Fix the downstream path when exporter failures remain the trigger.

Roll back when the canary introduces new refusals, increases recovery time, restarts under the same load, or hides loss without improving end-to-end counts. Restore the previous image and configuration together, remove the canary route, and replay the same synthetic test. If the previous version also loses data, keep the incident open: rollback restored state, not service quality.

Conclusion

Telemetry loss becomes operable when every pipeline boundary has an accepted, refused, dropped, queued or failed signal. Memory is only one part of that chain.

Find the first divergent counter, prove the downstream path, size resilience from an explicit failure budget, and canary the correction with synthetic count reconciliation. The production decision is then clear: keep the tuned Collector, scale the pipeline, fix the backend path, or restore the previous bundle with evidence that the rollback actually helped.