Automated end to end tests against a running instance of your web application are a necessary part of the development process. It is arguably more efficient to lean more in favor of end to end testing than unit testing. Both have their place, but automated end to end tests generally provide greater code coverage for less effort, and further liberate dedicated QA people from much of the drudgery inherent in their work, allowing them to focus on higher value activities. Ultimately, comprehensive automated end to end testing is required if heading down the road towards continuous integration and then continuous deployment of a web application.

The standard for running automated end to end tests these days is to use Selenium. The open and freely available Java Selenium standalone server - and its various per-browser drivers - can be accessed via a WebDriver network interface. Installed on a machine, Selenium follows the WebDriver instructions that you provide to launch and control a local browser instance, inspect the DOM on loaded pages, and so forth.

One reason that groups shy from fully embracing automated end to end testing to the degree needed for continuous deployment is that it is a major undertaking to maintain a Selenium farm diverse enough for your average major web application. If you are not selling into enterprise - where you can arbitrarily decide you are supporting only a few browser versions - then "diverse enough" might mean twenty different combinations of browser and OS, and possibly more once mobile devices are included in the mix. Even if you decide to create images and spin up suitable Selenium servers as needed in a cloud server, someone still has to build that infrastructure, keep it running, and make it possible to run Selenium tests in the development environment.

Fortunately there are services that do this for you, such as SauceLabs, BrowserStack, and TestingBot. You pay a monthly rate to gain access to Selenium servers with a wide variety of browser/OS combinations. Each of these services provides an SSH tunnel tool that allows testing against a local web server or other web servers isolated from the outside world inside your network. There are numerous ways in which you can configure your test setup with a Selenium service, but this is the most practical:

• Launch the web server you will be running end to end tests against.
• Launch an SSH tunnel, providing an unique tunnel identifier.
• Run end to end tests in parallel, flagging them with the tunnel identifier.
• Shut down the SSH tunnel.
• Shut down the web server.

Running tests in parallel is necessary because (a) any major application will have tests that run for hours in series, and (b) the nature of real world networks means that leaving SSH tunnels open for long periods of time is asking for trouble. The more tests you want to run in parallel the more that a Selenium service will cost you, but if you are willing to spend the money for much faster testing then it is well within the bounds of possibility to run hundreds of end to end tests in parallel through a single SSH tunnel on a single average developer machine. Much of the time taken by a test thread is spent waiting for responses from a remote Selenium server - it is not a computationally intensive activity.

A Demonstration Package

If you head on over to GitHub you'll find an example Node.js package showing parallel testing of a local Express server via either BrowserStack, SauceLabs, or TestingBot, specifying tests to run in Firefox 26 on Windows 8. You can install it via NPM:

npm install selenium-service-example

To use this you will have to register a free account with one of the three services mentioned above. This is quick and painless, and gives you the ability to run two parallel test threads at once, for a small number of total minutes of testing per month. It is enough for exploration and little more, but sufficient for our purposes here. Once registered you will need to take note of your account name and key for BrowserStack or SauceLabs, or key and secret for TestingBot. Those items must be added to config/index.js in order to enable testing. You will also need to set the service property in that file to the name of the service that you are using. For example:

module.exports = {
  // ------------------------------------------------------------------------
  // Specify the cloud Selenium service used.
  // ------------------------------------------------------------------------

  // Values: browserstack|saucelabs|testingbot
  service: 'browserstack',

  browserstack: {
    // Copy in the user and key from the Automate section of your BrowserStack
    // account.
    user: 'myuser',
    key: 'longrandomkey',

The package comes with a Vagrant Ubuntu 12.04 VM definition to ensure that the tunnel can actually run - TestingBot uses Java, but SauceLabs and BrowserStack native binaries are a little finicky about the environment as of the time of writing. They won't work on CentOS, for example. So launch and provision the VM, and then access it via SSH:

vagrant up
vagrant ssh

You can now run the examples in the Vagrant VM

cd /vagrant
node runTests

If all goes well, then this will:

• Launch a simple Express server.
• Launch the SSH tunnel for the configured Selenium service.
• Run parallel test processes, each running its own set of end to end tests in Firefox 26 on Windows 8.
• Shut down the SSH tunnel.
• Shut down the Express server.
• Display the results.

If errors occur, then the test runner will abort and shut down all of its child processes before exiting.

Sample Code

The important parts of the example are firstly how to launch and monitor the tunnel process, and secondly how to spawn and manage parallel test threads. They are by no means the only ways to do this, but err on the side of trying to be as straightforward as possible.

Launching a BrowserStack tunnel, for example, which requires monitoring the output to determine when it is ready:

/**
 * Launch a BrowserStack SSH tunnel instance.
 *
 * @param {Function} callback
 */
exports.launchBrowserStackTunnel = function (callback) {
  var self = this;
  var callbackInvoked = false;

  if(!config.browserstack.user) {
    return callback(new Error('Missing config.browserstack.user.'));
  }
  if(!config.browserstack.key) {
    return callback(new Error('Missing config.browserstack.key.'));
  }

  // BrowserStack requires specification of all servers that will be accessed
  // through the tunnel. If you don't list yours, then it won't work.
  //
  // Here we only care about the localhost server, but you can list more than
  // one server:
  // host1,port1,ssl1,host2,port2,ssl2,etc ...
  var hostString = [
    config.server.host,
    config.server.port,
    // Indicating that this is not an SSL connection.
    0
  ].join(',');

  var args = [
    config.browserstack.key,
    hostString,
    // Uncomment for verbose logging - not a great deal of difference with this
    // tunnel, however.
    // '-v',
    '-tunnelIdentifier',
    config.browserstack.tunnel.identifier,
    // Disable Live Testing and Screenshots, just test with Automate.
    '-onlyAutomate',
    // Skip checking for the validity of the folder/hosts parameters. This is
    // usually advisable, as you might want to fire up the tunnel before the
    // thing you are testing is ready, so as to save time.
    '-skipCheck'
  ];

  this.tunnelProcess = childProcess.spawn(
    config.browserstack.tunnel.path,
    args, {
      cwd: path.join(__dirname, '..')
    }
  );

  // Pipe both stdout and stderr to the specified log file.
  var logFile = path.join(__dirname, '..', config.browserstack.tunnel.log);
  var writer = fs.createWriteStream(logFile);
  this.tunnelProcess.stderr.pipe(writer);
  this.tunnelProcess.stdout.pipe(writer);

  // Set up a timeout watch to shut down if the tunnel setup hangs - which does
  // happen, though not often.
  var timeoutId = setTimeout(function () {
    self.tunnelProcess.removeListener('data', readyListener);
    callbackInvoked = true;
    callback(new Error('Timed out waiting for SSH tunnel to initialize.'));
  }, config.browserstack.tunnel.timeout);

  /**
   * A listener function for the tunnel. Callback when the tunnel outputs a
   * ready message.
   *
   * @param {Buffer} data
   */
  function readyListener (data) {
    if (data.toString().match(/You can now access your local server/)) {
      console.log('BrowserStack SSH tunnel launched and ready.');
      // Get rid of this listener and the timeout function.
      self.tunnelProcess.removeListener('data', readyListener);
      clearTimeout(timeoutId);
      // And onwards.
      callbackInvoked = true;
      callback();
    }
  }
  this.tunnelProcess.stdout.on('data', readyListener);

  // If the tunnel exits for any reason before the stopTunnel() method is
  // called than that is an error condition.
  this.tunnelProcess.on('exit', function (code) {
    // This might happen before readiness, so get rid of this listener and the
    // timeout function.
    self.tunnelProcess.removeListener('data', readyListener);
    clearTimeout(timeoutId);
    delete self.tunnelProcess;

    // And if it does happen before readiness, then we should make sure the
    // callback still happens.
    var error = new Error('SSH tunnel exited unexpectedly with code: ' + code);
    if (callbackInvoked) {
      self.handleError(error);
    } else {
      callbackInvoked = true;
      callback(error);
    }
  });
};

Running parallel test subprocesses requires a way for each subprocess to inform the master as to the outcome. At the very minimum the master must adjust its exit code to reflect the fact that tests passed or failed, and will normally need to do more than that. There are any number of ways to communicate between processes but here the messaging system built in to process.fork() is used.

/**
 * Launch the test runner subprocesses.
 */
exports.runTestProcesses = function (callback) {
  var self = this;

  function onExit(wrapper, code) {
    console.log('Test process ' + wrapper.index + ' exited with code ' + code);
    wrapper.code = code;
    delete wrapper.process;

    var complete = self.testProcesses.every(function (processWrapper) {
      return !processWrapper.process;
    });
    if (complete) {
      callback();
    }
  }

  this.testProcesses = config.workers.map(function (paths, index) {
    console.log('Launching test process ' + index);
    // Spawn a worker process, sending over the index and the base64 encoded
    // configuration to work with.
    var testProcess = childProcess.fork('./src/worker', [
      index,
      new Buffer(JSON.stringify(config)).toString('base64')
    ], {
      cwd: path.join(__dirname, '..'),
      silent: true
    });

    var wrapper = {
      complete: false,
      index: index,
      process: testProcess
    };

    testProcess.on('exit', function (code) {
      onExit(wrapper, code);
    });

    testProcess.on('message', function (message) {
      wrapper.complete = true;
      wrapper.failureCount = message.failureCount;
    });

    var logFile = path.join(__dirname, '..', 'logs', 'test-proc-' + index + '.log');
    var writer = fs.createWriteStream(logFile);
    testProcess.stderr.pipe(writer);
    testProcess.stdout.pipe(writer);

    return wrapper;
  });
};

The child process sends a message the parent when it is done, assuming that it makes it that far. If won't send a message if it errors out or throws, but that can be handled as a failure case by the parent.

mocha.run(function (failureCount) {
  // Close off the browser whatever happened.
  global.browser.quit();

  // Message the parent process to tell it what happened.
  process.send({
    failureCount: failureCount
  });

  if (failureCount) {
    process.exit(1);
  } else {
    process.exit(0);
  }
});