Our 3 Laws (of Cucumber)

If you use Cucumber, you need to read Jonas Nicklas' post, You're Cuking it Wrong

It sums up nicely everything I've hated about how most folks write Cucumber.

Seriously - go and read it now if you haven't already… At the risk of ruining my reputation of being witty and charming by using a blogging cliché, "don't worry, I'll wait right here for you to come back".

My eyes were first opened at #rc5 where I saw how others were writing Cucumber scenarios: regexen, CSS and XPath in their features, and overcomplicated Features bogged down in needless repetition of boring steps. Not surprising, these people were unhappy with Cucumber as a solution.

We've been using Cucumber for well over a year on all of our client projects, and we love it to bits (heck, Rik even presented on it at Perth RORO way back in 2009); obviously we use it to exercise our code as our users will, but we also use it to test our APIs, and for testing legacy code that we've just inherited (gak!).

We use a kind of "3 Laws of Cucumber" when writing our features, and this has stood us in good stead:

  1. Does it make complete sense to someone who has never cut code before;
  2. Can it be made shorter;
  3. Can we make it run faster.

(Apologies to Asimov, but you get the idea.)

1) Does it make complete sense to someone who has never cut code before

Our first, and most important goal of writing Cucumber is that the client MUST be able to understand it. Otherwise, why would we bother writing it?

If we want tests which only we can read, we may as well just use rSpec and test everything individually.

However, we consider that tests that the client can read (and more importantly, can help us refine) are essential to our development process, for more reasons than you might think.

Firstly, (and most obviously) they provide a way to confirm new features with stakeholders. They can read the story and mentally make sure that it matches their expectation of what the feature will do. It's at this point where a client can look at the Scenarios and remember "You know what? We need to capture the width of the sprocket here too…". That is, the understandable nature of well written Scenarios allows our customers to imagine what the solution will look like, allowing them to provide us more useful feedback.

Next, well written Scenarios provide a great starting point when we discover there is an error in business logic, because we can put them in front of the stakeholders and say "Right, this is what this process does at the moment. Let's go through it and you yell when something's not right". Sure, you could write up that documentation at the time (or trace the code with them there… gak!), but by virtue of writing our Scenarios in readable English, we already have a high level resource available.

We look at the Scenario, the client can usually spot the problem quickly, we write a new Scenario and get on with development quickly and efficiently.

Finally, by having very little in the way of specialist syntax, it's easier to work with stakeholders to confirm the existing functionality of legacy or inherited code. Often, inherited code comes with all sorts of quirks and convoluted UI pathways which the Customer is already aware of. Rather than us tracing every single link and feature, we can work with our clients to quickly document existing functionality as it sits, before we start making changes. This saves us time and saves the customer money, but most importantly it brings the Customer in to the development process - something which they've probably not had before (almost a guarantee if it's a rescue project!).

In fact, we have found that often, semi-technical stakeholders will quickly take to writing the first cut of these Features themselves, which we then just need to massage to suit steps already available.

To keep our steps readable, we will go out of our way to ensure that there is no XPath, CSS, or URLs in our Features. At the same time though, we are not prepared to trade precision for readability - Then I should see "Welcome Back, Jason" (matching anywhere in the page) is not a suitable substitute for Then I should see "Welcome Back, Jason" in #notice...

Our step would look like this :

To do this, we provide our own selector_for helper (similar to that provided by default for path_to) to map a string to CSS or XPath, and we have steps like :

(You can see the selector_for helper method here )

Obviously, the complexity has to be somewhere, but we will gladly trade easy to read Features for slightly tricky step definitions. (That said, "complex" steps never really seem that complex if you are comfortable with regex - we are).

Update 20Sep2010: Related to this, check out Bodaniel's post about exactly this : Selector-Free Cucumber Scenarios )

We also use complex path matchers when appropriate to keep URLs out of our steps.

To do this, we ensure that every time we refer to a page, it is done so using a human readable name - whether it be a generic page, or a specific one. For example :

In this way, we can easily generate a complex URL given a very simple human description, testing not only the page itself, but also that our routing is working as we expect.

(It also makes it trivial to update our Scenarios if we need to change our routes…)

2) Can it be made shorter

The second rule ensures that we don't create huge, boring scenarios which set up the same state every time; we make heavy use of Background steps, as well as writing "condensed" steps to make the story more succinct.

For example, whilst it may be important for the Authentication process to be clearly defined once (like in an authentication Feature):

When we are doing every other Scenario where login is just something that needs to happen (but isn't the focus of the feature - you know, like most cases) we simply have one step which does this:

This is - as far as I see - one of the major benefits of Cucumber; the ability to create steps of arbitrary complexity to represent complex processes quickly when they don't add any real value to the feature being tested.

3) Can we make it run faster

Finally, we realise that, as Cucumber is testing our entire stack, we need to make performance tweaks wherever we can so as that they are usable during development.

Thankfully, we find that most of our entire Cucumber suites take less than 4 minutes to run on my MBP's 2.4GHz Core 2 Duo. But still, that's long enough for us to avoid running the full suite every cycle, and so instead we only run the features we need (or use autotest) and run the full suite before commit.

(Granted, this is a rather quick suite comparatively… A colleague told me the other day how their test suite took 20 minutes to run… I do not envy them that… That said, their project is easily 10x the complexity of those we deal with day-to-day)

To help reduce the time to test, we make sure that we limit how often we fire up in-browser tests like Selenium, and let Capybara and Webrat do as much of the heavy lifting as we can. Selenium is obviously essential to test AJAX and other JS, but we find that most of the time we can get away without using it if we structure our tests a little more carefully.

In Closing

Cucumber is an awesome piece of software (as are Webrat, Capybara, Selenium, et al) - both as a development tool, and as a tool to help engage and involve non-technical stakeholders in their project.

However, when used incorrectly, Cucumber becomes a truly terrible solution to both.

A poorly written set of Features is verbose, slow to write, difficult to maintain and slow to run. Even worse, poorly written, Cucumber looses the benefits which make it so valuable - it becomes hard for stakeholders to understand, and useless as a resource for facilitating meaningful and clear discussion between technical and non-technical stakeholders.

If you drive a car only in first gear, it's not the car that's useless, it's the driver.

If you're writing garbage Features, it's not Cucumber that sucks - it's the developer.