Beta Acid

Best-in class software innovates and evolves to keep customers engaged. One way to improve and add features without disrupting the current user experience is to deploy a canary release.

Canary releases get their name from an old coal mining practice of taking canaries into the mine to test for deadly carbon monoxide. Carbon monoxide is odorless, and would affect a small bird much faster than a miner. So if you were deep in a mine and your canary stopped singing and became sluggish, you knew it was time to get out quickly. Here in the coding world, we can use a similar warning system to test new features and code changes.

Here’s how it works

When you want to introduce a new feature or code change, you run two different versions of your code in parallel. Most of your users will still be accessing the current version of your site or service – think of them as your miners. Then, a small percentage of users – the canaries – will experience the new version. This allows you to test your new features or changes with real world users while limiting the potential impact of any bugs in the new release.

Once you’re confident that the new version is working as it should, you can then increase the percentage of users using the new version of your code. This slow rollout will allow you to diagnose any scaling issues that might not initially show up with your small test group.

AWS Cloudfront

Cloudfront is a Content Delivery Network (CDN) provided by AWS. It helps developers make their sites scale better and with lower latencies by caching site content locally across multiple regions of the world, while only maintaining a single instance of your site. It’s a great tool for developers and is widely used, but AWS Cloudfront doesn’t have native support for canary releases. Fortunately, there’s a workaround: it does have support for running custom code on all requests through Cloudfront Functions and lambda@edge. So let’s talk about how to leverage lambda@edge functions to implement a canary release.

lambda@edge

The lambda@edge AWS service allows developers to deploy lambda functions to all Cloudfront regions, so they can be called by Cloudfront with low latencies. lambda@edge functions have some different limitations than regular lambda functions, and you can check those out here.

When you associate a lambda@edge function with a Cloudfront distribution, the function version that is associated will be deployed to all regions. This allows for lower latency, but be aware that the logs for the lambda function will be created in the region where the request was handled.

A Canary Release on Cloudfront

Now, let’s talk details. Hosting a SPA on S3 using Cloudfront as a cache is a simple way to host your site. You can use the static site hosting option from S3 for a public site, or you can keep your S3 bucket private and access the content through Cloudfront using an OAI. If you plan to use an OAI, you’ll probably need to do some editing to the requests with a lambda@edge or a Cloudfront function to handle paths correctly.

To run a canary release, just add a lambda@edge function to handle the requests that cloudfront will be making to the origin, in this case the S3 bucket. The code below will change the origin based on the value of the header bucket. This header should be the header used in the cache behavior in Cloudfront. This will reduce the number of times the lambda functions need to be called.

"use strict";
exports.handler = (event, context, callback) => {
  const request = event.records[0].cf.request;
  if (request.headers.cookie) {
    request.headers.cookie.foreach(function (cookie) {
      if (cookie.value.startswith("mycookiename=")) {
        const domainname = "new-release-bucket.s3.amazonaws.com";
        request.origin.s3.domainname = domainname;
        request.headers["host"] = [{ key: "host", value: domainname }];
      }
    });
  }
  callback(null, request);
};

It’s best practice to set a cookie before making the request, but you can also make use of another function attached to the viewer request or response event to set the cookie. Using a cookie will also keep the user on the new release throughout all the requests made to the site.

Where can you grow from here?

The great thing about deploying canary releases is that you can dream big and keep risks small. What are some of the features and add-ons your customer’s have been asking for? What are the best-in-class apps doing that you’d like to be doing? What new feature would make your software stand out? Pick one, start coding, and let the canaries lead the way!