QUnit or Jasmine?
Over the last few months I've been spiking a few user interface ideas
for the Agile Planner, which has meant that I've been writing a
lot of CoffeeScript. While I've been experimenting, I've been content to
test it my work with acceptance tests written in Ruby with
capybara-webkit. Last week my CoffeeScript became a little more
complicated, and I could feel my progress slowing down as I grappled to
keep everything in my head. It was time to go back to TDD.
It's been a while since I've unit tested JavaScript in anger, so I spent half an hour investigating the most obvious choices. There seem to be two main contenders:
- Jasmine - a BDD framework that's clearly influenced by RSpec, and
- QUnit - a more traditional test framework from the jQuery project.
I compared them on three criteria:
- Speed of setup. I tend to avoid testing tools that take me more than hour to get up and running.
- Simplicity. Test frameworks don't need to be complicated; they just need to let me specify how my code should behave and keep out of my way. All things being equal, I'll choose an xUnit style testing framework to an RSpec style framework. If you're thinking "but I prefer BDD!" don't forget that the "B" in BDD is just language, and you can use that language in any framework you like.
- Support for Continuous Integration. If you can't run your test suite automatically then sooner or later you'll forget to run it manually, and at some point ship a broken app to your users. Running tests in your CI build also enables you to automate continuous deployment, which is always a giggle.
Evaluating Jasmine
I looked at Jasmine first. A few weeks ago I stumbled across a project called Jasminerice that makes it really easy to unit test CoffeeScript in a Rails 3 project, using the Rails asset pipeline.
While I'm not generally a fan of investing time in technologies whose
use is limited to a single framework (Rails), Jasminerice really
appealed. It looks like drop-in CoffeeScript testing for Rails. I'm
using the asset pipeline, so in theory all I'd need to do to start
testing my code is to install the jasminerice gem and start creating
files in spec/javascripts.
With such simplicity comes reduced flexibility. I don't use RSpec (for
me, it can't hold a candle to MiniTest). I keep all my tests in a test
folder. My CoffeeScript/JavaScript tests are going to live in
test/javascript. I opened the Jasminerice source to see whether the
path was configurable. It wasn't. I could have monkey patched it, but I
was starting to feel a bit of friction.
What about running tests on the command line? The Jasminerice README
pointed me to guard-jasmine, which adds a bunch of extra features.
As you might expect, it's a plugin for Guard that will run your tests
when you save your .coffee files, but you can also run it outside of
Guard. It uses the PhantomJS headless browser, so running tests from
the command line or on a CI server is no problem.
Taking a quick look back at my three criteria, how do Jasmine, Jasminerice and guard-jasmine stack up?
Speed of setup
Jasmine looked innocently straightforward to begin with, but a thorough read through of the three projects' README files took a while. guard-jasmine has a lot of options. That kind of control is often just what you need, but all I need to do here is write a few unit tests.
Simplicity
I didn't feel very clear on the relative responsibilities of the three projects (Jasmine, Jasminerice and guard-jasmine). It felt as though I'd end up reading the code in order to be confident that I'd set it up correctly. That's fine, but I didn't fancy doing it for three separate projects.
Then I noticed that guard-jasmine depended on RSpec, which it had added
to my project. That's 10,000 lines of Ruby code that I don't need to
install, just to test some JavaScript. I felt as though I'd be swimming
against the current with MiniTest; I don't want a spec directory, and
I don't want RSpec in my bundled gems.
Support for Continuous Integration
guard-jasmine clearly has this aspect well and truly sorted. It looks fantastic.
Thoughts on Jasmine
If you're an RSpec fan, have time to read through all the docs and play around with it, guard-jasmine looks brilliant.
Evaluating QUnit
I'm using MooTools rather than jQuery on Agile Planner, so my first question was whether or not QUnit relies on jQuery. It doesn't; if jQuery is loaded it'll use it internally, otherwise it falls back to standard DOM manipulation techniques. This discovery was a great start; it's clearly been designed to be framework agnostic, and that means that any time I invest in learning it can be re-used on other projects, using different libraries.
I was also rather chuffed that it took me 30 seconds to find and search the source code in order to check this out; it's easy to read, and there was only one file to look in. A simple and transparent project. Great start.
There only seemed to be one obvious downside to QUnit. It's designed to run in a browser, while I was really hoping for something that I could launch from Ruby. I use TConsole to run my Ruby tests and I was really hoping that I'd be able to find a way to launch my CoffeeScript tests from within TConsole. QUnit's dependency on a browser made this look rather unlikely out of the box.
Incidentally, TConsole is the best test runner I've used, in any language, and its high time I blogged about it.
So how does QUnit stand up against my three criteria?
Speed of setup
Setting up QUnit is easy, though not that succinctly documented. You
need to make an HTML file with a few pre-defined tags in, and then load
qunit.css and qunit.js in the file's <head> tag.
You also need to load your CoffeeScript libraries and test files (which is fairly easy) and then load the web page in your browser.
That's all there is to it. Once you've got hold of the files, it's fast.
Simplicity
I can't imagine how it could be simpler. There are no gems to install, just three files to download and a snippet of code to edit.
There is, of course, a downside...
Support for Continuous Integration
UPDATE: When I first wrote this article there weren't many options for running QUnit from the command line. Now (July 2013) there seem to be a few options.
As Knalli and Rhys Evans commented (below) QUnit can now be run from the terminal via node.js and phantomJs installed, using grunt. Eric Mann gave a demonstration of how to use grunt with Qunit in his talk from the 2013 jQuery conference. Coverage of grunt starts around 15 minutes into the video.
You can also use Karma. Toran Billups has produced an screencast on integration testing with Karma showing you to do it on an Ember app (but the Karma config would be transferrable to any JavaScript app you happen to be building).
Thoughts on QUnit
I love the small size, simplicity of installation (and the lack of external dependencies), and the fact that I can start writing tests without even involving Bundler. Seriously, trying it out without involving Bundler was noticeably refreshing.
What I picked...
It was an easy decision in the end; I went with QUnit.
I doubt I'll be running my CoffeeScript tests from the console, and will
just use JsTestDriver for continuous integration. I'll use
guard-livereload to automatically refresh my in-browser tests when my
code/tests change.
In another post I'll show you exactly what I've got in my QUnit HTML file, and how I'm loading my CoffeeScript.
If you can't wait for that post either nag me on Twitter and I'll
make you a gist, or do some Googling. There are already a few gists out
there, and a few threads on Stack Overflow. And if you go and find
those, just remember that if you compile your CoffeeScript with -b
(the "bare" option), you're doing it wrong!