Ex
Ratione
About • GitHub • Blog • Archives • Feed

A Node.js Package to Deploy CloudFormation Stacks

By Reason September 6th, 2015 Permalink

I have long been disappointed with the lack of tooling in the Node.js ecosystem for the use of AWS CloudFormation. Outside of the DSL tool coffin to help with the programmatic construction of CloudFormation templates (a package that is, sadly, written in Coffescript rather than Javascript) there is very little out there. This is unfortunate, because using CloudFormation robustly in your deployment or continuous integration process requires more infrastructure than just making a few calls to the Node.js AWS SDK's CloudFormation API. That API provides a good set of bricks, but you still have to build the house.

Introducing cloudformation-deploy

Out of sheer frustration with this state of affairs, and because I was working on another Node.js project that absolutely required infrastructure for CloudFormation deployment, I recently wrote and published cloudformation-deploy. This package manages the details and edge cases for one common workflow in CloudFormation stack deployment:

  • Create a new version of the CloudFormation template for your application stack.
  • Validate the template.
  • Request stack creation.
  • Wait on the stack status to show show success.
  • Update existing resources to point to the new stack.
  • Delete the prior instance of the stack once the switch is done.

For example, the stack for a simple stateless web application might be built around the core of an Elastic Load Balancer and Auto Scaling Group. The switch of resources on successful deployment of a new version of the stack consists of updating a Route 53 CNAME record to point at the new stack and waiting 60 seconds for DNS propagation to complete.

Stack Update Versus Creation, Switch, and Deletion

The CloudFormation API does offer the ability to update an existing stack as an alternative to the create, switch, and destroy workflow described above. Fewer requests are involved, but it is much more complex under the hood, and you have far less control over how the update proceeds. When using update as a part of devops and deployment processes you have to be very sure to understand exactly how it is going to affect the resources in the stack, and that can change dramatically for comparatively small differences in the CloudFormation template. There is little written on this topic, and the best way to gain knowledge is to experiment with stacks to see what happens.

It is fairly clear, however, that it is almost never the case that updating an existing stack is acceptable when EC2 servers are involved, or when downtime must be minimized, or when the switchover to the new version must be close to atomic. The best way to eliminate downtime in deployment, as is the case in cloudformation-deploy, is to (a) deploy new infrastructure in parallel to old infrastructure and then (b) make some atomic or near-atomic change that points users and resources to the new stack.

Using cloudformation-deploy

There are examples of use in the cloudformation-deploy codebase, and the project documentation explains the configuration object details. Here is an example that deploys one of the sample EC2 CloudFormation templates provided by Amazon. 

/**
 * @fileOverview Deploy an EC2 stack from one of the example AWS templates.
 *
 * Depending on the instanceType specified it can be made to succeed or fail
 * due to the availability of virtualization support. This is helpful when
 * wanting to demonstrate behavior of the deployment code on success or failure.
 *
 * Fail on: t2.micro.
 * Succeed on: m1.small.
 */

// Core.
var fs = require('fs');
var path = require('path');
var util = require('util');

// NPM.
var cloudFormationDeploy = require('cloudformation-deploy');

// Variables.
var unixTimestamp = Math.round((new Date()).getTime() / 1000);

var config = {
  // The baseName is used to tag stacks and identify which stacks are
  // prior instances of this application.
  baseName: 'ec2-cloudformation-deploy',
  version: '0.1.0',
  // This should usually be a build ID generated by a task manager, such
  // as BUILD_NUMBER in Jenkins. Using a Unix timestamp is a fair fallback
  // for the sake of making this example run.
  deployId: unixTimestamp,
  progressCheckIntervalInSeconds: 3,
  // Parameters provided to the CloudFormation template.
  parameters: {
    KeyName: 'cloudformation-deploy-example',
    InstanceType: 'm1.small',
    SSHLocation: '0.0.0.0/0'
  },
  // A callback invoked once for each new event during stack creation and
  // deletion.
  onEventFn: function (event) {
    console.log(util.format(
      'Event: %s',
      JSON.stringify(event)
    ));
  },
  // Invoked after stack creation is successful, but before any prior stacks
  // are deleted. Usually used to switch over resources to point to the new
  // stack, but here just an excuse for more example logging.
  postCreationFn: function (stackDescription, innerCallback) {
    console.log(util.format(
      'Deployed stack description: %s',
      JSON.stringify(stackDescription, null, '  ')
    ));

    innerCallback();
  }
};

// Load a local template JSON file. We can pass in the template as a JSON
// string, and object, or as an S3 URL pointing to an uploaded template.
//
// Here we are using an example EC2 template provided by Amazon. See:
// https://github.com/exratione/cloudformation-deploy/blob/master/examples/templates/ec2.json
var templatePath = path.join(__dirname, '../templates/ec2.json');
var template = fs.readFileSync(templatePath, {
  encoding: 'utf8'
});

cloudFormationDeploy.deploy(config, template, function (error, result) {
  // This enables error messages to show up in the JSON output. Not something to
  // be used outside of example code.
  Object.defineProperty(Error.prototype, 'message', {
    configurable: true,
    enumerable: true
  });

  // This will be a large set of data even for smaller deployments, including
  // events, the stack description object, and stack traces for errors.
  console.log(util.format(
    'Result: %s',
    JSON.stringify(result, null, '  ')
  ));

  if (error) {
    console.error(error.message, error.stack || '');
    console.error('Deployment failed.');
  }
  else {
    console.log('Deployment successful.');
  }
});

Process and Edge Cases

What happens when the new stack deployment fails to complete? Cloud services such as AWS remain inherently fallible and one has to expect a low but meaningful error rate on any set of operations even when all of the configuration is correct. In a production situation, the failed stack should be deleted. In development, you may want to keep it around to inspect. Equally, when running in development you may or may not actually want the prior stack for the application to be removed following a successful deployment.

The following optional configuration object parameters determine this behavior in cloudformation-deploy: 

// Default behavior: delete the prior application stack on successful deployment.
config.priorInstance = cloudFormationDeploy.priorInstance.DELETE;
// Do not delete the prior application stack.
config.priorInstance = cloudFormationDeploy.priorInstance.DO_NOTHING;

// Default behavior: delete the deployed stack on failure.
config.onFailure = cloudFormationDeploy.onFailure.DELETE;
// Do not delete the deployed stack on failure.
config.onFailure = cloudFormationDeploy.onFailure.DO_NOTHING;

Deploying Multiple Stacks

The cloudformation-deploy package only explicitly supports deployment of a single stack. Deploying multiple linked stacks for a single application is not uncommon, though it should be replaced these days by the use of nested stacks. In that case a single template is deployed, and that template contains includes AWS::CloudFormation::Stack resources that refer to templates stored in S3. All of the referenced templates are deployed as distinct stacks.

If not using nested stacks, it is still possible to deploy multiple related stacks concurrently with cloudformation-deploy. The postCreationFn function passed in with the configuration object is asynchronous, so multiple stacks can build and wait on one another. A simple example: 

var complete = {};
var failed = false;

function isComplete () {
  return complete.a && complete.b;
}

configA.postCreationFn = function (stackDescription, callback) {
  complete.a = true;

  function continueWhenComplete () {
    if (failed) {
      return callback(new Error('The other template failed.'));
    }
    else if (!isComplete()) {
      return setTimeout(continueWhenComplete, 1000);
    }

    // Perform switchover tasks here...

    callback();
  });
   
  continueWhenComplete();
};

configB.postCreationFn = function (stackDescription, callback) {
  complete.b = true;  

  function continueWhenComplete () {
    if (failed) {
      return callback(new Error('The other template failed.'));
    }
    else if (!isComplete()) {
      return setTimeout(continueWhenComplete, 1000);
    }

    // Perform switchover tasks here...

    callback();
  });
   
  continueWhenComplete();
};

cloudformationDeploy.deploy(configA, templateA, function (error, results) {
  if (error) {
    failed = true;
    // More cleanup here...
  }
});
cloudformationDeploy.deploy(configB, templateB, function (error, results) {
  if (error) {
    failed = true;
    // More cleanup here...
  }
});