Browser Level Testing - Martin Kleppmann

Martin's talk was about what you can do when you want to go beyond unit testing. In particular, this is a talk about how you can test JavaScript.

Why can't you rely on your JavaScript library to guarantee that all browsers will treat your JavaScript the same way? Here are a couple of examples.

IE isn't so clever when working out the length of arrays with trailing commas:

var x = [1, ];

x.length  1  // Firefox 3.5
x.lenth = 2  // IE7

IE also gets a bit confused about what an HTML id really is. Given this HTML:

<div id="hello"></div>

you see non standard behaviour in IE:

typeof(hello) = 'undefined' // FF 3.5
typeof(hello) == 'object'  // IE7

In other words, if you want to guarantee that your application works in a specific browser you really need to test the behaviour in that browser. There are so many browsers in common use today that it needs automating, but how do you go about it?

Driving the browser with simulated events

Selenium is a good example of this technique. Selenium itself is loaded from localhost, while the site you're testing is often running on a remote host. This violates the browser security model, so you need to load the site that you're testing through a proxy.

Problems:

  • You can't test sites that redirect between different domains.
  • You can't test things that occur outside of the DOM (pop-ups flash, etc.).
  • It's slow (particularly in IE).

Browser specific automation APIs

As an alternative to simulating events, you can control browsers through their programmable APIs. It's a lot more work to implement, but it works a much better than event simulation. Projects taking this approach include WebDriver and Watir.

WebDriver is an open source project that is supported by Google. It has implemntations of browser specific APIs and supports IE, Firefox, Chrome and Opera (but not Safari). It also drives HTMLUnit and supports Celerity, which emulates the behaviour of a browser in a headless manner. As a result it's faster than testing in a real browser.

Watir supports IE, Firefox, Safari, Chrome and HTMLUnit.

Martin started a startup company that runs Selenium in the cloud, running on a bunch of virtual machines. He's come up with a DSL for describing tests called Go-Test.

Interestingly, the Selenium and WebDriver projects are joining forces. WebDriver will be used to control the browsers, while Selenium will provide the programming API. Martin thinks that Watir and Selenium should move towards providing front ends to WebDriver, to create a superb open source browser level testing framework.

Here's some example RSpec code for controlling WebDriver via Selenium:

Controlling WebDriver with Selenium

Writing robust tests

The tests you can run with these frameworks do tend to be quite brittle -- you don't need to make many changes to your app before you can expect your tests to fail. You can mitigate the problem somewhat by writing flexible locators (or selectors). If you're not sure what a locator is, they're used within tests to identify tags in your HTML. For example:

id = searchInput
name = search
xpath = //form[@id='searchform']/input[2]

Here are a couple of examples of how you can select a headline on a BBC article:

xpath = //td[@class=text']/div[1]/a
css = a.tsh

The xpath example is a little brittle; if the BBC stopped using tables to lay out this page then the <td> tag would disappear and a test that relied on the xpath locator would break. The CSS selector is (in this case) a much neater way of doing it, and would probably survive updates to the HTML (the tsh class name has some semantic meaning, though I forget what Martin said it meant).

So there are 2 approaches that you can take when writing these front end tests:

  1. Disposable tests; created automatically with record/playback software. They might not survive for very long when your application gets refactored.
  2. Domain-specific abstractions; carefully crafted tests which are more effort to write, but which should be less effort to maintain.

Which approach is best? Which do you prefer? Martin would be interested in hearing your thoughts.

It was a great overview of the current state of browser testing, and (as a long term Selenium user) I'll definitely be having a look at WebDriver on Monday.

You can no doubt find out more at Martin's blog, or find him on Twitter as @martinkl.

More talks from Ruby Manor