Skip to content

Canary deployments

DETAILS: Tier: Free, Premium, Ultimate Offering: GitLab.com, Self-managed, GitLab Dedicated

Canary deployments are a popular continuous deployment strategy, where a small portion of the fleet is updated to the new version of your application.

When embracing continuous delivery, an organization needs to decide what type of deployment strategy to use. One of the most popular strategies is canary deployments, where a small portion of the fleet is updated to the new version first. This subset, the canaries, then serve as the proverbial canary in the coal mine.

If there is a problem with the new version of the application, only a small percentage of users are affected and the change can either be fixed or quickly reverted.

Use cases

Canary deployments can be used when you want to ship features to only a portion of your pods fleet and watch their behavior as a percentage of your user base visits the temporarily deployed feature. If all works well, you can deploy the feature to production knowing that it shouldn't cause any problems.

Canary deployments are also especially required for backend refactors, performance improvements, or other changes where the user interface doesn't change, but you want to make sure the performance stays the same, or improves. Developers need to be careful when using canaries with user-facing changes, because by default, requests from the same user are randomly distributed between canary and non-canary pods, which could result in confusion or even errors. If needed, you may want to consider setting service.spec.sessionAffinity to ClientIP in your Kubernetes service definitions, but that is beyond the scope of this document.

Advanced traffic control with Canary Ingress

Canary deployments can be more strategic with Canary Ingress, which is an advanced traffic routing service that controls incoming HTTP requests between stable and canary deployments based on factors such as weight, sessions, cookies, and others. GitLab uses this service in its Auto Deploy architecture to let users quickly and safely roll out their new deployments.

How to set up a Canary Ingress in a canary deployment

A Canary Ingress is installed by default if your Auto DevOps pipeline uses v2.0.0+ of auto-deploy-image. A Canary Ingress becomes available when you create a new canary deployment and is destroyed when the canary deployment is promoted to production.

Here's an example setup flow from scratch:

  1. Prepare an Auto DevOps-enabled project.
  2. Set up a Kubernetes Cluster in your project.
  3. Install NGINX Ingress in your cluster.
  4. Set up the base domain based on the Ingress Endpoint assigned above.
  5. Check if v2.0.0+ of auto-deploy-image is used in your Auto DevOps pipelines. If it isn't, follow the documentation to specify the image version.
  6. Run a new Auto DevOps pipeline and make sure that the production job succeeds and creates a production environment.
  7. Configure a canary deployment job for Auto DevOps pipelines.
  8. Run a new Auto DevOps pipeline and make sure that the canary job succeeds and creates a canary deployment with Canary Ingress.

Show Canary Ingress deployments on deploy boards (deprecated)

WARNING: This feature was deprecated in GitLab 14.5.

To view canary deployments you must properly configure deploy boards:

  1. Follow the steps to enable deploy boards.
  2. To track canary deployments you must label your Kubernetes deployments and pods with track: canary. To get started quickly, you can use the Auto Deploy template for canary deployments that GitLab provides.

Depending on the deploy, the label should be either stable or canary. GitLab assumes the track label is stable if the label is blank or missing. Any other track label is considered canary (temporary). This allows GitLab to discover whether a deployment is stable or canary (temporary).

Once all of the above are set up and the pipeline has run at least once, Go to the environments page under Pipelines > Environments. As the pipeline executes, deploy boards clearly mark canary pods, enabling quick and clear insight into the status of each environment and deployment.

Canary deployments are marked with a yellow dot in the deploy board so that you can quickly notice them.

Canary deployments on deploy board

How to check the current traffic weight on a Canary Ingress (deprecated)

WARNING: This feature was deprecated in GitLab 14.5.

  1. Visit the deploy board.

  2. View the current weights on the right.

    Rollout Status Canary Ingress

How to change the traffic weight on a Canary Ingress (deprecated)

WARNING: This feature was deprecated in GitLab 14.5.

You can change the traffic weight in your environment's deploy board by using GraphiQL, or by sending requests to the GraphQL API.

To use your deploy board:

  1. Go to Operate > Environments for your project.
  2. Set the new weight with the dropdown list on the right side.
  3. Confirm your selection.

Here's an example using GraphiQL:

  1. Visit GraphiQL Explorer.

  2. Execute the environmentsCanaryIngressUpdate GraphQL mutation:

    mutation {
  environmentsCanaryIngressUpdate(input:{
    id: "gid://gitlab/Environment/29",              # Your Environment ID. You can get the ID from the URL of the environment page.
    weight: 45                                      # The new traffic weight. for example, If you set `45`, 45% of traffic goes to a canary deployment and 55% of traffic goes to a stable deployment.
  }) {
    errors
  }
}

  3. If the request succeeds, the errors response contains an empty array. GitLab sends a PATCH request to your Kubernetes cluster for updating the weight parameter on a Canary Ingress.