Deploy a Release Using a Canary Deployment

This tutorial walks you through deploying a canary release in Kubernetes. A canary deployment allows you to safely test a new application version with a small portion of production traffic before rolling it out completely. This approach helps minimize risk and enables rapid feedback from real users.

For an overview of canary deployments, see Canary deployments in the Managing Workloads concept page.

Objectives

  • Deploy a stable version of an application
  • Deploy a canary version alongside the stable version
  • Route traffic to both versions using a Service
  • Monitor the canary deployment
  • Complete the rollout by scaling the canary and removing the stable version

Before you begin

You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:

Understanding canary deployments

A canary deployment is a strategy where you deploy a new version of your application alongside the existing version. The new version (canary) receives a small percentage of traffic, allowing you to:

  • Test the new version with real production traffic
  • Monitor for errors, performance issues, or unexpected behavior
  • Roll back quickly if issues are detected
  • Gradually increase traffic to the new version if it performs well

In this tutorial, you'll use the track label to differentiate between the stable and canary releases. The stable release uses track: stable, and the canary release uses track: canary. Both deployments share common labels (app.kubernetes.io/name: rollout-demo) that allow the Service to route traffic to both sets of Pods.

You'll deploy the stable version with 3 replicas and the canary version with 1 replica. Since both versions share the same Service selector (app.kubernetes.io/name: rollout-demo), Kubernetes will load-balance traffic across all 4 pods. With this ratio, approximately 25% of traffic goes to the canary and 75% to the stable version.

Deploying the stable version

First, deploy the stable version of your application:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: rollout-demo-stable
  labels:
    app.kubernetes.io/name: rollout-demo
    track: stable
spec:
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/name: rollout-demo
      track: stable
  template:
    metadata:
      labels:
        app.kubernetes.io/name: rollout-demo
        track: stable
    spec:
      containers:
      - name: rollout-demo
        image: gcr.io/google-samples/hello-app:1.0
        ports:
        - containerPort: 8080
        env:
        - name: VERSION
          value: "v1"

Apply the stable Deployment:

kubectl apply -f https://k8s.io/examples/application/canary/app-v1-deployment.yaml

Verify that the Deployment was created and the Pods are running:

kubectl get deployments -l app.kubernetes.io/name=rollout-demo

The output is similar to:

NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
rollout-demo-stable    3/3     3            3           10s

Check the Pods:

kubectl get pods -l app.kubernetes.io/name=rollout-demo

The output is similar to:

NAME                                  READY   STATUS    RESTARTS   AGE
rollout-demo-stable-7d4b9b8c5d-abc12   1/1     Running   0          15s
rollout-demo-stable-7d4b9b8c5d-def34   1/1     Running   0          15s
rollout-demo-stable-7d4b9b8c5d-ghi56   1/1     Running   0          15s

Creating a service

Create a Service to expose your application. The Service selector uses the common label (app.kubernetes.io/name: rollout-demo) and omits the track label, which allows it to route traffic to both the stable and canary Pods:

apiVersion: v1
kind: Service
metadata:
  name: rollout-demo-service
  labels:
    app.kubernetes.io/name: rollout-demo
spec:
  type: ClusterIP
  selector:
    app.kubernetes.io/name: rollout-demo
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080

Apply the Service:

kubectl apply -f https://k8s.io/examples/application/canary/app-service.yaml

Verify the Service was created:

kubectl get service rollout-demo-service

The output is similar to:

NAME                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
rollout-demo-service   ClusterIP   10.96.123.45    <none>        80/TCP    5s

Test the Service by creating a temporary Pod and making a request:

kubectl run curl-test --image=curlimages/curl:latest --rm -it --restart=Never -- curl http://rollout-demo-service

You should see responses from the stable Pods. Run this command multiple times to see different Pod hostnames, but all responses should show version v1.

At this point, your application is in a steady state. In a real-world scenario, you would typically pause here and operate the stable version until you have a new version to deploy. The next steps demonstrate how to introduce a canary version.

Deploying the canary version

To create a canary Deployment, you can copy the stable Deployment manifest and make a few changes:

  • Change the metadata.name (for example, to rollout-demo-canary)
  • Change the track label from stable to canary in both metadata.labels and spec.selector.matchLabels/spec.template.metadata.labels
  • Set a lower number of replicas (for example, 1)
  • Update the container image to the new version

Now deploy the canary version alongside the stable version:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: rollout-demo-canary
  labels:
    app.kubernetes.io/name: rollout-demo
    track: canary
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: rollout-demo
      track: canary
  template:
    metadata:
      labels:
        app.kubernetes.io/name: rollout-demo
        track: canary
    spec:
      containers:
      - name: rollout-demo
        image: gcr.io/google-samples/hello-app:2.0
        ports:
        - containerPort: 8080
        env:
        - name: VERSION
          value: "v2"

Apply the canary Deployment:

kubectl apply -f https://k8s.io/examples/application/canary/app-v2-deployment.yaml

Verify both Deployments are running:

kubectl get deployments -l app.kubernetes.io/name=rollout-demo

The output is similar to:

NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
rollout-demo-stable    3/3     3            3           2m
rollout-demo-canary    1/1     1            1           10s

Check all Pods:

kubectl get pods -l app.kubernetes.io/name=rollout-demo -o wide

The output is similar to:

NAME                                  READY   STATUS    RESTARTS   AGE   IP           NODE
rollout-demo-stable-7d4b9b8c5d-abc12   1/1     Running   0          2m    10.244.1.5   node1
rollout-demo-stable-7d4b9b8c5d-def34   1/1     Running   0          2m    10.244.1.6   node1
rollout-demo-stable-7d4b9b8c5d-ghi56   1/1     Running   0          2m    10.244.2.7   node2
rollout-demo-canary-8e5c0d9f6a-xyz78   1/1     Running   0          15s   10.244.2.8   node2

Notice that you now have 4 Pods total: 3 stable Pods and 1 canary Pod.

Testing traffic distribution

Since both Deployments use the same Service selector (app.kubernetes.io/name: rollout-demo), the Service routes traffic to all Pods. With 3 stable Pods and 1 canary Pod, approximately 25% of requests go to the canary.

Test the Service multiple times to see traffic distribution:

kubectl run curl-test --image=curlimages/curl:latest --rm -it --restart=Never -- \
  sh -c 'for i in $(seq 1 10); do curl -s http://rollout-demo-service; echo; done'

You should see responses from both stable and canary versions. The ratio may vary, but you should see some canary responses mixed with stable responses.

Check the Service EndpointSlices to verify both versions are receiving traffic:

kubectl get endpointslices -l kubernetes.io/service-name=rollout-demo-service

The output is similar to (one EndpointSlice per address family; the ENDPOINTS column is truncated by default):

NAME                          ADDRESSTYPE   PORTS   ENDPOINTS                              AGE
rollout-demo-service-abc12    IPv4          8080    10.244.1.5,10.244.1.6,10.244.2.7 + 1 more...   3m

To see all endpoint addresses, use -o yaml.

Monitoring the canary deployment

Monitor your canary deployment for errors, performance issues, or unexpected behavior:

Check Pod logs for the canary:

kubectl logs -l app.kubernetes.io/name=rollout-demo,track=canary --tail=50

Monitor Pod status:

kubectl get pods -l app.kubernetes.io/name=rollout-demo -w

Press Ctrl+C to stop watching.

Check resource usage:

kubectl top pods -l app.kubernetes.io/name=rollout-demo

Adjusting traffic distribution

If the canary is performing well, you can gradually increase traffic to it by scaling it up:

Scale the canary to 2 replicas (now 40% of traffic):

kubectl scale deployment/rollout-demo-canary --replicas=2

Verify the new Pod is running:

kubectl get pods -l app.kubernetes.io/name=rollout-demo

Continue monitoring. If everything looks good, you can scale the canary further and scale down the stable version.

Completing the rollout

If your monitoring instead uncovered issues with the canary, skip ahead to Rolling back a canary deployment.

Once you're confident that the canary is stable and performing well, complete the rollout:

Scale the canary to the desired number of replicas (e.g., 3):

kubectl scale deployment/rollout-demo-canary --replicas=3

Scale down the stable version to 0:

kubectl scale deployment/rollout-demo-stable --replicas=0

Verify all traffic is going to the canary:

kubectl get pods -l app.kubernetes.io/name=rollout-demo

The output should show only canary Pods:

NAME                                  READY   STATUS    RESTARTS   AGE
rollout-demo-canary-8e5c0d9f6a-xyz78   1/1     Running   0          5m
rollout-demo-canary-8e5c0d9f6a-abc12   1/1     Running   0          2m
rollout-demo-canary-8e5c0d9f6a-def34   1/1     Running   0          2m

Test the Service to confirm all responses are from the canary:

kubectl run curl-test --image=curlimages/curl:latest --rm -it --restart=Never -- curl http://rollout-demo-service

All responses should now show version v2.

To complete the rollout with a single Deployment, first scale the stable Deployment back up so you do not lose capacity if the new image fails to pull. Then update the image and the VERSION environment variable to match the new version, wait for the rollout to complete, and remove the canary Deployment:

kubectl scale deployment/rollout-demo-stable --replicas=3
kubectl set image deployment/rollout-demo-stable rollout-demo=gcr.io/google-samples/hello-app:2.0
kubectl set env deployment/rollout-demo-stable VERSION=v2
kubectl rollout status deployment/rollout-demo-stable
kubectl delete deployment rollout-demo-canary

Rolling back a canary deployment

If you detect issues with the canary version, you can quickly roll back:

Scale down the canary deployment:

kubectl scale deployment/rollout-demo-canary --replicas=0

Scaling to zero preserves the canary Deployment so you can inspect its configuration while you investigate. Once you've finished, delete it with kubectl delete deployment rollout-demo-canary.

Scale the stable version back up if needed:

kubectl scale deployment/rollout-demo-stable --replicas=3

Investigate the issues with the canary before attempting another canary deployment.

Splitting traffic using HTTPRoute (optional)

If you're using the Gateway API, you can use HTTPRoute to have more precise control over traffic splitting between the stable and canary versions. This approach allows you to specify exact percentages for traffic distribution rather than relying on replica counts.

First, create separate Services for the stable and canary versions. This approach is also useful for debugging, even if you are not using Gateway API. By having separate Services, you can directly test or monitor each version independently.

apiVersion: v1
kind: Service
metadata:
  name: rollout-demo-stable-service
spec:
  selector:
    app.kubernetes.io/name: rollout-demo
    track: stable
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: rollout-demo-canary-service
spec:
  selector:
    app.kubernetes.io/name: rollout-demo
    track: canary
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080

Then create an HTTPRoute that splits traffic between the two Services:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: rollout-demo-route
spec:
  parentRefs:
  - name: example-gateway
  hostnames:
  - "rollout-demo.example.com"
  rules:
  - backendRefs:
    - name: rollout-demo-stable-service
      port: 80
      weight: 90
    - name: rollout-demo-canary-service
      port: 80
      weight: 10

This configuration routes 90% of traffic to the stable Service and 10% to the canary Service, regardless of the number of replicas.

You can also use header-based routing to send specific traffic to the canary:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: rollout-demo-route
spec:
  parentRefs:
  - name: example-gateway
  hostnames:
  - "rollout-demo.example.com"
  rules:
  # Rule 1: Route traffic with canary header to canary service
  - matches:
    - headers:
      - type: Exact
        name: env
        value: canary
    backendRefs:
    - name: rollout-demo-canary-service
      port: 80
  # Rule 2: Split remaining traffic 90/10
  - backendRefs:
    - name: rollout-demo-stable-service
      port: 80
      weight: 90
    - name: rollout-demo-canary-service
      port: 80
      weight: 10

This configuration sends all traffic with the env: canary header to the canary Service, while other traffic is split 90/10 between stable and canary.

Automating canary rollouts

In production environments, canary rollouts are typically managed by controllers or CI/CD systems that automate traffic shifting, monitoring, and promotion or rollback. Tools such as Flux, Argo Rollouts, GitLab, and many others can handle progressive delivery, analysis, and rollback for you. This reduces manual steps and helps ensure safe, repeatable deployments. For more information, see the documentation for your chosen deployment tool or controller.

Cleaning up

Delete the resources created in this tutorial:

kubectl delete deployment rollout-demo-stable rollout-demo-canary
kubectl delete service rollout-demo-service

If you created separate Services for HTTPRoute, delete those as well:

kubectl delete service rollout-demo-stable-service rollout-demo-canary-service
kubectl delete httproute rollout-demo-route

What's next

  • Learn more about canary deployments in the Managing Workloads concept page.
  • Read about Deployments and how they manage your application lifecycle.
  • Read about Services and how they enable service discovery and load balancing.
  • Explore the Gateway API for advanced traffic management capabilities.
  • Consider using Horizontal Pod Autoscaling to automatically adjust replica counts based on metrics.