Argo Rollouts Canary Deployment: Letting Prometheus Metrics Drive Traffic Percentage Instead

If you've ever felt anxious checking Slack notifications the morning after a late-night deployment, you've probably had a similar experience to mine. When you find yourself rolling back at 9 AM wondering "why are 5xx errors firing when this release didn't touch anything?", it's only natural to wish something could make that judgment automatically while you sleep.

Argo Rollouts' Canary strategy and AnalysisTemplate actually implement that idea. It exposes the new version to just 5–10% of total traffic first, automatically advances to the next step when Prometheus metric thresholds pass, and executes a rollback before a human can even intervene if it fails. This post breaks down how it works, walks through real YAML examples, and covers the common pitfalls.

This is aimed at people who are already running Kubernetes, have used kubectl rollout undo more than a few times, and are thinking "I want to deploy smarter next time."

The Limits of Basic Deployments and the Gap Argo Rollouts Fills

What Basic Rolling Updates Can't Do

Kubernetes' built-in Deployment rolling update is a well-crafted feature. But it has two fundamental limitations.

First, traffic ratio and Pod ratio are coupled. When 30% of new-version Pods come up, roughly 30% of traffic flows to them. For services with few replicas, one Pod becomes 33% of the total, making fine-grained traffic control impossible.

Second, there is no metric-based decision logic. If a new Pod is in Ready state, Kubernetes considers the deployment successful. Even if response times internally doubled, or 5xx errors increased by 2%, the health check still passes.

Argo Rollouts replaces Deployment with the Rollout CRD and solves both problems. It runs stable and canary ReplicaSets simultaneously and controls traffic independently of Pod count by integrating with an Ingress controller or service mesh.

Its Place in a GitOps Pipeline

The ArgoCD + Argo Rollouts combination has been widely adopted in GitOps pipelines. ArgoCD synchronizes the Rollout manifest, and Argo Rollouts executes the progressive deployment. We recommend checking the official GitHub Releases for the version of Argo Rollouts you're using and its support matrix. Compatibility information for traffic routing providers (Istio, NGINX, ALB, etc.) and metric providers (Prometheus, Datadog, etc.) is documented in the release notes.

Rollout Resource Structure — Declaring Deployment Scenarios with Steps

Basic Rollout Manifest

Converting a Deployment to a Rollout is simpler than you might think. Change apiVersion and kind, and add spec.strategy. However, you must not run a Deployment and a Rollout with the same name simultaneously. The order matters: delete the existing Deployment first, then create the Rollout.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: my-service
spec:
  replicas: 10
  selector:
    matchLabels:
      app: my-service
  template:
    metadata:
      labels:
        app: my-service
    spec:
      containers:
        - name: my-service
          image: my-service:v2.1.0
  strategy:
    canary:
      canaryService: my-service-canary
      stableService: my-service-stable
      steps:
        - setWeight: 10
        - pause: {duration: 5m}
        - setWeight: 30
        - pause: {duration: 10m}
        - setWeight: 60
        - analysis:
            templates:
              - templateName: success-rate
            args:
              - name: service-name
                value: my-service
        - setWeight: 100

steps is the core. setWeight declares the traffic percentage to send to canary, pause declares a wait time, and analysis declares an AnalysisTemplate execution. The controller automatically runs them in this order.

Traffic Management Integration — NGINX and Istio

The method of controlling traffic weights differs by environment. With NGINX Ingress, it works via annotations.

strategy:
  canary:
    canaryService: my-service-canary
    stableService: my-service-stable
    trafficRouting:
      nginx:
        stableIngress: my-service-ingress

Argo Rollouts automatically updates the nginx.ingress.kubernetes.io/canary-weight annotation on the Ingress at each deployment step.

In an Istio environment, it directly adjusts the weights in the VirtualService. Direct adjustment via Istio is far more precise than splitting traffic only by Pod count ratio. For example, if you want to send just 5% of traffic to canary in a service with 2 replicas, it's impossible to implement with Pod count alone. This is why an Istio VirtualService-based approach has become the de facto standard.

trafficRouting:
  istio:
    virtualService:
      name: my-service-vsvc
      routes:
        - primary
    destinationRule:
      name: my-service-destrule
      canarySubsetName: canary
      stableSubsetName: stable

AnalysisTemplate — How Metrics Determine Deployment Progression

How It Works

When an analysis step is reached, Argo Rollouts creates an AnalysisRun resource. The AnalysisRun sends a PromQL query to Prometheus at each specified interval and checks whether the result satisfies the successCondition. If it fails more than failureLimit times, it automatically resets the canary weight to 0 and rolls back.

Error Rate-Based AnalysisTemplate

This is the most commonly used pattern. It treats an HTTP 5xx error rate exceeding 5% as a failure.

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  args:
    - name: service-name
  metrics:
    - name: success-rate
      interval: 1m
      successCondition: result[0] >= 0.95
      failureLimit: 3
      provider:
        prometheus:
          address: http://prometheus:9090
          query: |
            sum(rate(http_requests_total{
              service="{{args.service-name}}",
              status!~"5.."
            }[2m])) /
            sum(rate(http_requests_total{
              service="{{args.service-name}}"
            }[2m]))

failureLimit: 3 is important. If a single failure from a momentary traffic spike triggers a rollback, there will be too many false positives. failureLimit is the upper bound on the number of measurements that fail to satisfy successCondition, so up to 3 failures are tolerated and rollback is triggered on the 4th. Note that there is a separate field called consecutiveErrorLimit, which is unrelated to the successCondition evaluation — it is the consecutive upper bound for provider errors where the PromQL query itself fails (network timeout, query parse failure, etc.). The two fields have different roles, so be careful not to confuse them.

P99 Latency-Based Analysis

If you want to catch performance regressions early in the deployment, you can add a latency-based condition.

metrics:
  - name: latency-p99
    interval: 1m
    successCondition: result[0] < 0.3
    failureLimit: 3
    provider:
      prometheus:
        address: http://prometheus:9090
        query: |
          histogram_quantile(0.99,
            sum(rate(http_request_duration_seconds_bucket{
              service="my-service"
            }[5m])) by (le)
          )

successCondition: result[0] < 0.3 means success is declared when P99 latency is below 300ms. If this threshold is not met, traffic will not be raised to 60% and the rollout will stop.

You can also declare multiple metrics in a single AnalysisTemplate. It is common to monitor error rate and latency simultaneously and require both to pass before proceeding to the next step.

Full Deployment Flow Visualization

Configuration Patterns by Real-World Scenario

Unattended Overnight Deployment — Automatic Rollback Based on Error Rate

This is the pattern SRE teams adopt first. It lets you deploy during hours when no one is monitoring while still ensuring quality.

strategy:
  canary:
    steps:
      - setWeight: 5
      - pause: {duration: 10m}
      - analysis:
          templates:
            - templateName: success-rate
          args:
            - name: service-name
              value: my-service
      - setWeight: 25
      - pause: {duration: 15m}
      - analysis:
          templates:
            - templateName: success-rate
            - templateName: latency-p99
          args:
            - name: service-name
              value: my-service
      - setWeight: 100

The key point is repeating the analysis after each weight step. At the initial 5% stage, only the error rate is checked; at the 25% stage, both error rate and latency are validated together.

Header-Based Routing — Early Access for Internal QA Team

Routing only requests with a specific HTTP header to canary is useful because it lets you expose the new version exclusively to an internal tester group without touching overall traffic. With Istio, you define two routes in the VirtualService (a header-matching route and a default route) and reference both route names in the Rollout spec.

First, in the VirtualService, requests with an X-Canary: true header are directed to the canary subset, while the rest are split by weight.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: my-service-vsvc
spec:
  hosts:
    - my-service
  http:
    - name: qa-internal
      match:
        - headers:
            X-Canary:
              exact: 'true'
      route:
        - destination:
            host: my-service
            subset: canary
          weight: 100
        - destination:
            host: my-service
            subset: stable
          weight: 0
    - name: primary
      route:
        - destination:
            host: my-service
            subset: stable
          weight: 100
        - destination:
            host: my-service
            subset: canary
          weight: 0

In the Rollout, specify both routes in the routes array so that Argo Rollouts adjusts the weights of both routes together as the deployment progresses.

trafficRouting:
  istio:
    virtualService:
      name: my-service-vsvc
      routes:
        - primary
        - qa-internal
    destinationRule:
      name: my-service-destrule
      canarySubsetName: canary
      stableSubsetName: stable

The primary route handles weight-based canary, and the qa-internal route sends matching requests directly to the canary subset at all times. While the QA team independently validates the new version, regular users are served by the stable version.

Checking Real-Time Status with the kubectl Plugin

kubectl argo rollouts get rollout my-service --watch
 
kubectl argo rollouts promote my-service
 
kubectl argo rollouts abort my-service
kubectl argo rollouts undo my-service

The get rollout --watch command shows the canary weight, AnalysisRun status, and each step's progress in real time. When verifying your initial setup, keeping this command open and watching the steps progress makes it very intuitive to understand.

Trade-offs and Common Mistakes

What You Gain

What You Have to Accept

Mistakes Commonly Encountered in Practice

Setting failureLimit too low: Setting failureLimit: 1 means a single momentary spike triggers a rollback. Starting around 3–5 is a good approach.

Using P99 latency for low-traffic services: If total service RPS is 10 with canary at 5% weight, requests reaching canary are about 30 per minute. Calculating P99 from that sample is meaningless. For low-traffic services, it's better to only look at error rate, or start latency analysis only after raising the weight to at least 30%.

Setting thresholds without baseline metrics: If you set successCondition: result[0] >= 0.95 but the service's normal success rate is 98%, a 5% margin is too generous. Conversely, if the normal rate is 97% due to external dependencies, a threshold of 0.95 will always pass and becomes meaningless. It's recommended to look at at least two weeks of Prometheus data to establish a baseline.

Running Deployment and Rollout simultaneously: If a Deployment and Rollout with the same name coexist, the two controllers conflict over the same ReplicaSet. Either clean up the Deployment first, or follow the procedure in the official Argo Rollouts migration guide.

Where to Start

When first adopting this, the official Best Practices recommends a specific order. Start with Blue/Green deployment to understand your metrics and application characteristics, then transition to Canary. Even for Canary itself, it's realistic to start simply with just weights and pauses without AnalysisTemplate, and progressively add analysis steps once core KPIs are confirmed.

Secure a metric baseline, decide on a traffic router (NGINX or Istio), and choose a service with sufficient traffic for your first rollout. With those three things in place, the rest can be expressed as manifests.

In the end, all of this configuration has one purpose: the morning after a late-night deployment, instead of Slack alerts, being able to drink your coffee while reading "the new version automatically rolled out to 100%." The moment you delegate rollback decisions from humans to metrics, the anxiety of the morning after a deployment noticeably changes.

References

• Argo Rollouts Official Docs — Analysis & Progressive Delivery
• Argo Rollouts Official Docs — Prometheus Metric Provider
• Argo Rollouts Official Docs — Canary Strategy
• Argo Rollouts Official Docs — Best Practices
• Argo Rollouts Official Docs — Migrating from Deployments
• Argo Rollouts GitHub Releases
• Canary Deployment in Kubernetes Part 3 — Smart Canary with Argo Rollouts and Prometheus
• Progressive Canary Deployments on Kubernetes with Argo Rollouts and Istio
• Canary Deployment in Kubernetes Using Argo Rollouts and Istio — Deckhouse Blog
• Progressive Canary Releases with Argo Rollouts Analysis and Linkerd Metrics
• Automating Blue-Green & Canary Deployments with Argo Rollouts — Akuity