Unifying Multi-Service Telemetry Pipelines with the OpenTelemetry Collector Gateway Pattern

When you're running 10, 20, or more microservices, telemetry inevitably becomes a real headache. Sending traces to Jaeger, metrics to Prometheus, and logs to Elasticsearch separately means SDK configuration differs across every service, and every time you add a new backend, you have to touch dozens of services. At first I thought, "Can't each service just export directly?" — but as services multiplied, I experienced firsthand how terrifying a technical debt that approach becomes.

The OpenTelemetry Collector Gateway pattern addresses this problem at its root. Each service only needs to know a single OTLP endpoint, and all logic for adding, swapping, or filtering backends is handled entirely at the Gateway config file level. In this post, I'll walk through — with real code — how to safely transform data using Processor chaining and how to design Exporter fan-out to distribute data to multiple backends simultaneously. The focus is on scenarios commonly encountered in production: Tail sampling, Routing Connector, and loadbalancingexporter.

Why the Gateway Pattern Matters Now

What Happens in the Field

A setup where each service sends telemetry directly to multiple backends looks simple at first. However, according to the OpenTelemetry Collector Follow-up Survey Analysis (2026), a majority of respondents are running multiple Collector instances, and VM-based deployments alongside Kubernetes are also on the rise (see the survey itself for exact figures). This reflects how much more complex telemetry flows have become as hybrid and multi-cloud environments have gone mainstream.

In such environments, the "each service sends directly" approach creates three problems:

1. Cost of change: Adding or swapping a backend requires redeploying every service.
2. Tail sampling is impossible: When spans from the same trace are scattered across multiple services, Tail sampling — which decides whether to sample only after seeing the complete trace — becomes fundamentally impossible.
3. Resource waste: Having each service run heavy transformation and filtering logic at the SDK level consumes application resources.

The Agent-to-Gateway Two-Tier Architecture

The most common deployment topology for the Gateway pattern is the Agent-to-Gateway two-tier architecture. A lightweight Agent Collector runs on each node (or Pod) and handles only local data collection, while a central Gateway Collector takes ownership of all actual processing.

Agents are kept lightweight with minimal configuration. CPU- and memory-intensive operations like Tail sampling and complex transformations are all concentrated at the Gateway. In a Kubernetes environment, deploy the Agent as a DaemonSet and the Gateway as a Deployment, attaching HPA only to the Gateway.

Processor Chaining — Order Is Stability

Why Order Matters

Honestly, at first I didn't think much about Processor order. "It all gets processed eventually, right?" But after experiencing a Collector crash due to OOM from placing memory_limiter near the end of the pipeline, my thinking changed.

The recommended Processor execution order is:

memory_limiter → resource → attributes → filter → tail_sampling → batch

There's a reason for this order. memory_limiter must be first so it can reject data under memory pressure before heavy transformation and sampling logic runs on data that would be discarded anyway. batch must always come last. Placing batch before tail_sampling causes incomplete traces to be bundled into batches, making sampling decisions unpredictable.

A Real Gateway Collector Configuration Example

Below is a Gateway configuration for a SaaS environment that receives traces, applies Tail sampling, and fans out to two backends. Since this is a production example, retry_on_failure and sending_queue are explicitly defined, and error_mode: ignore is set on the filter processor so that OTTL expression evaluation errors don't destabilize the pipeline.

# gateway-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
 
processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 1500
    spike_limit_mib: 300
 
  resource:
    attributes:
      - key: deployment.environment
        value: production
        action: upsert
 
  attributes:
    actions:
      - key: http.user_agent
        action: delete  # Remove PII
 
  filter/drop_health:
    error_mode: ignore  # Ignore expression evaluation errors when attribute is absent
    traces:
      span:
        - 'attributes["http.target"] == "/health"'
 
  tail_sampling:
    decision_wait: 30s
    num_traces: 50000
    policies:
      - name: errors-policy
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: slow-traces-policy
        type: latency
        latency:
          threshold_ms: 1000
      - name: probabilistic-policy
        type: probabilistic
        probabilistic:
          sampling_percentage: 10
 
  batch:
    send_batch_size: 1000
    timeout: 5s
 
exporters:
  otlphttp/jaeger:
    endpoint: https://jaeger-collector:4318
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 5m
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 5000
 
  datadog:
    api:
      key: ${env:DD_API_KEY}
    retry_on_failure:
      enabled: true
    sending_queue:
      enabled: true
      queue_size: 5000
 
  prometheusremotewrite:
    endpoint: <YOUR_GRAFANA_CLOUD_PROM_ENDPOINT>  # e.g. https://prometheus-prod-XX-prod-us-east-0.grafana.net/api/prom/push
    retry_on_failure:
      enabled: true
 
service:
  # Note: tail_sampling here assumes a single Gateway instance.
  # If you scale out to 2 or more Gateway instances, you must also apply the
  # loadbalancingexporter configuration in the 'Tail Sampling and Gateway Scale-Out'
  # section below to prevent spans from being distributed across instances.
  pipelines:
    traces:
      receivers: [otlp]
      processors:
        - memory_limiter
        - resource
        - attributes
        - filter/drop_health
        - tail_sampling
        - batch
      exporters: [otlphttp/jaeger, datadog]
    metrics:
      receivers: [otlp]
      processors:
        - memory_limiter
        - resource
        - batch
      exporters: [prometheusremotewrite]

Using a slash in a name like filter/drop_health lets you define multiple Processors of the same type. Filtering out health check traffic matters more than you might think — without it, Tail sampling buckets fill up quickly with meaningless spans.

Exporter Fan-Out — Services Don't Need to Know About New Backends

How Fan-Out Works

When you list multiple Exporters under exporters in service.pipelines, the Collector internally creates a fan-out consumer that copies the output of the last Processor and delivers it to all Exporters simultaneously. No changes are needed in service code.

One SaaS team needed to send metrics to Grafana Cloud, traces to Datadog, and logs to Elasticsearch — and accomplished it purely through Gateway configuration without touching any service code. When swapping backends, it was as simple as updating the config file and restarting the Collector.

Using Fan-Out for Backend Migration

Another use of fan-out is parallel delivery during migration. When moving from Jaeger to a new tracing backend, you can send data to both backends simultaneously during a validation period, and once validation is complete, simply remove the old Exporter entry. Migration completes without any service redeployment.

# Mid-migration configuration (conceptual example; retry and queue settings omitted)
exporters:
  otlphttp/old_jaeger:
    endpoint: https://old-jaeger:4318
  otlphttp/new_backend:
    endpoint: https://new-tracing-backend:4318
 
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, resource, tail_sampling, batch]
      exporters: [otlphttp/old_jaeger, otlphttp/new_backend]  # Simultaneous delivery

Routing Connector — Separate Pipelines per Service

From routingprocessor to Routing Connector

routingprocessor is an older component. After checking the status and recommendations documented in the routingprocessor README and routingconnector README in the contrib repository, adopting routing connector for new designs has become standard practice (determine the exact status based on your Collector version and the latest guidance in each README). A Connector is a special component that acts as an Exporter in one pipeline and a Receiver in the next, enabling pipeline-level routing that is far more flexible for multi-tenancy designs.

Using OTTL (OpenTelemetry Transformation Language) condition expressions, you can route data from specific services or attributes into separate pipelines.

connectors:
  routing:
    default_pipelines: [traces/default]
    error_mode: ignore
    table:
      - statement: route() where resource.attributes["service.name"] == "payment-service"
        pipelines: [traces/payment]
      - statement: route() where resource.attributes["service.name"] == "auth-service"
        pipelines: [traces/auth]
 
service:
  pipelines:
    traces/ingress:
      receivers: [otlp]
      processors: [memory_limiter]
      exporters: [routing]  # Connector acts as Exporter
 
    traces/payment:
      receivers: [routing]  # Connector acts as Receiver
      processors: [tail_sampling, batch]
      exporters: [otlphttp/payment_backend]
 
    traces/auth:
      receivers: [routing]
      processors: [tail_sampling, batch]
      exporters: [otlphttp/auth_backend]
 
    traces/default:
      receivers: [routing]
      processors: [tail_sampling, batch]
      exporters: [otlphttp/jaeger]

Tail Sampling and Gateway Scale-Out

Tail sampling works correctly only when all spans of the same trace are guaranteed to arrive at a single decision point. The Gateway configuration example earlier included tail_sampling without any countermeasure for this issue — but the moment you scale out to two or more Gateway instances, spans become distributed across different instances and incomplete traces lead to incorrect sampling decisions. This is exactly where the tradeoff table's condition "minimum 2 instances required" creates a direct conflict.

The answer to this problem is loadbalancingexporter. It applies consistent hashing based on traceID at the Agent level so that spans with the same traceID always reach the same Gateway instance.

# Agent Collector configuration (deployed as DaemonSet on each node)
exporters:
  loadbalancing:
    protocol:
      otlp:
        tls:
          insecure: true
    resolver:
      dns:
        hostname: gateway-collector-headless.observability.svc.cluster.local
        port: 4317
 
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [loadbalancing]

The dns resolver uses a Kubernetes headless service to dynamically retrieve the list of Gateway Pods. When Gateway Pods are added or removed, this is reflected automatically.

A common misconception here is that loadbalancingexporter is a separate, centralized load balancer component. In fact, each Agent Pod embeds its own loadbalancingexporter instance, and those instances each independently query the same DNS resolver and route to a specific Gateway based on traceID hashing.

Tradeoffs and Common Mistakes in Practice

Structural Pros and Cons

Mistakes That Actually Come Up in Practice

Items already covered in the main text — such as Processor order (memory_limiter first, batch strictly after tail_sampling) — are not repeated here. This section covers only pitfalls not addressed elsewhere.

1. Setting decision_wait too short

If decision_wait is shorter than the actual trace completion latency, sampling decisions are made before all spans have arrived. Measure P99 trace latency in your production environment and add a buffer on top of that value.

2. Omitting retry and queue settings on Exporters

Leaving out retry_on_failure and sending_queue means data is simply lost during a temporary backend outage. For production Exporters, don't leave defaults as-is — explicitly enable them and define queue sizes.

3. Missing Collector self-observability

From the start, your design should include a way to verify that the Gateway is operating correctly. The Collector's own internal metrics (queue size, drop count, receive/send latency, etc.) are exposed separately via service.telemetry.metrics.address. This is a different concept from the prometheus exporter that exposes pipeline data, so don't confuse the two.

# Collector self-observability configuration
extensions:
  zpages:
    endpoint: 0.0.0.0:55679
 
service:
  extensions: [zpages]
  telemetry:
    metrics:
      level: detailed
      address: 0.0.0.0:8888  # Endpoint for exposing the Collector's own internal metrics

4. Tail sampling behaves strangely after scaling out the Gateway

When multiple Gateway instances are running, not using loadbalancingexporter on the Agent side causes traceIDs to scatter across instances, resulting in incomplete decisions. This must be understood as a paired requirement with the Agent configuration in the previous section.

Closing Thoughts

The essence of the Gateway pattern is ultimately separation of concerns. Each microservice doesn't need to know "where to send," and the Gateway owns that decision entirely. Processor chaining explicitly declares the order in which data is transformed, and Exporter fan-out distributes the results to multiple backends simultaneously.

Separating pipelines per service with the Routing Connector allows each team in a multi-tenancy environment to have its own independent backend, and combining loadbalancingexporter with Tail sampling enables complete trace-based sampling even in distributed environments.

The real value of adopting the Gateway in production is that a few lines in a config file replace dozens of service redeployments. You don't have to file a ticket with the application team when swapping a backend, and when validating a new backend, you can just add a fan-out and watch for a few days without any code changes. Once you have a solid grasp of the Processor ordering rules and the Connector concept, most future expansion converges to editing pipeline definitions.

References

• Gateway deployment pattern — OpenTelemetry official docs
• Agent-to-gateway deployment pattern — OpenTelemetry official docs
• Architecture — OpenTelemetry official docs
• Configuration — OpenTelemetry official docs
• Routing Connector README — opentelemetry-collector-contrib
• Routing Processor README — opentelemetry-collector-contrib
• Tail Sampling Processor README — opentelemetry-collector-contrib
• Load Balancing Exporter README — opentelemetry-collector-contrib
• OpenTelemetry Collector Follow-up Survey Analysis — OpenTelemetry Blog