Donnerstag, 8. November 2012

symfony2, Behat, Mink & Zombie.js: Headless browser tests

I wanted to work with Behat for a while now. I remember hearing a talk by @everzet at the Symfony live Paris this year which got me interested in Behaviour Driven Development (BDD) and Behat. Now I finally came around and looked into the whole topic.

I decided to go straight to browser tests which also work for javascript heavy sites as this is a real pain for me right now, using symfony and the TestClient it provides which of course cannot evaluate any javascript.

So much "stuff"

I agree that the headline of this post contains many libraries/frameworks, so let me add some background information on all of them and how they will work together.

symfony2 is my php framework of choice. It's where the application will live and where everything is developed. Behat will serve as the BDD testing framework, meaning that tests are written as feature descriptions and Behat is taking care to execute those text as actual tests against your code. The basics are described in the docs.

Mink is an acceptance testing framework for web applications which encapsulates different browser simulators, so one API can be used and the simulator (selenium for example) can be exchanges. Some of those simulators (such as sahi or selenium) need to run actual browsers to make requests and evaluate responses. Others (like goutte) are headless but cannot evaluate javascript.

Zombie.js is a headless browser emulator which also can evaluate javascript (because it is written in javascript), so it combines the speed of goutte with the usage on javascript heavy sites from sahi or selenium. Right now Zombie.js only evaluates javascript with the chromium javascript engine.

Bonus tip: If you are really fancy, you can use goutte for your non-javascript tests and Zombie.js for javascript tests locally. On a special testing server, you run selenium (or sahi) and execute the same tests against real browsers. That way tests are very fast locally where they are executed very often and very reliable in the testing environment.


Beside the stuff you need to run symfony2 (I am using symfony 2.1 with PHP 5.4.7 on Ubuntu 10.04) you need Zombie.js. The installation docs on their side are a bit fuzzy and didn't work for me. I got it running by installing node.js and npm from source and then running sudo npm install -g zombie. One word of caution: Zombie.js doesn't seem to work flawlessly. I spent much time to figure out why some of my features failed about half the time. I could eliminate this problem by testing against the production environment but I actually don't know what caused the problems.

Afterwards you can add your dependencies to your composer.json:

"require-dev": {
    "behat/behat": "2.4.*@stable",
    "behat/mink": "1.4.*@stable",
    "behat/mink-extension": "*",
    "behat/mink-zombie-driver": "*",
    "behat/symfony2-extension": "*"

Running composer (php composer.phar update --dev) will install Behat, Mink and the Zombie.js driver as well as the required extensions. You should have a behat file in your bin folderafterwards and should be able to execute it by running ./bin/behat -h. If this is not the case, check for composer errors.

Configuring Behat 

Next you need a behat.yml file. I placed it inside my app folder as there are also files for phpunit, phing and so on. This has the drawback that Behat doesn't find the file automatically and I need to pass the config file everytime. In the behat.yml you need to configure the Zombie extension and activate the symfony2 extension for Mink:

        Behat\Symfony2Extension\Extension: ~
            base_url: http://yourproject.local
            default_session: zombie
            javascript_session: zombie
            zombie: ~

The only thing you need to change is the base_url which of course should be your local project url. Now if you execute ./bin/behat --config=app/behat.yml -dl you should see a few definitions like "Given /^(?:|I )am on homepage$/" and so on.

To initialize Behat run ./bin/behat --init @YourBundleName command. This should create a Features folder inside your Bundle in which you place your feature files and which contains a "Context" Folder with your FeatureContext. You need to change the FeatureContext a bit to work. Replace the BehatContext the class extends with the MinkContext. Your class signature should look like this:

class FeatureContext extends MinkContext implements KernelAwareInterface

Now you can go ahead and implement your first feature.

Let's visit the homepage

As a very basic example I want to implement the following feature:

In order to get some information about the project
As a website user
I need to see content on the homepage

That should not be very hard. Let's create a file called homepage.feature inside your Features folder and fill in the following content:

Feature: Homepage
  In order to get some information about the project
  As a website user
  I need to see content on the homepage

  Scenario: Showing the homepage
    Given I am on homepage
    Then I should see "MyProject"

Of course you should replace "MyProject" with some text from your homepage. Now we can run Behat and see if our "feature" works:
./bin/behat --config=app/behat.yml @YourBundleName
Specificing your bundle like this (with the @YourBundleName) is crucial to Behat recognizing the file structure which is a bit different from normal Behat setups. It took me some time to figure that out, so be careful!

If this one runs successfully, you are ready to define "real" features and test them with Zombie.js . Congratulations!

Mind the gap(s)!

When setting this up, I had some problems I want to share so you don't waste your time.

As mentioned above, the moment you use the SymfonyBundle for symfony extensions, thinks work different. Don't try to use the normal Behat docs only to wonder why your features or the FeatureContext are not found. The docs are not perfect on that end, but by following the docs on the symfony extension carefully, it should all work out. And you got this post as well!

I already said it, but I have to repeat it: Zombie.js is not bug-free, so expect tests to fail even if the feature works fine. I couldn't figure out what exactly was wrong with my setup, but for some reason Zombie.js kept failing most of the time for one special test. Sometimes, out of the blue, it would work. I also couldn't get the sleep function to work.

Using the dev environment (by calling the app_dev.php) didn't work for me at all with Zombie.js. It kept failing here and there. I would guess this is because my dev environment is not the fastest (using vagrant it means it's a virtual machine inside my not-so-powerfull laptop). I switched to the normal app.php for now which works way better. Of course I have to remember to clear the cache. After I clear the cache, all tests fail one or two times, then they work again. It's strange, but that's the way it works for me right now. Maybe someone knows some tweaks he can share with me - I would appreciate it!

I also fought a bit with understanding Behat (and Mink) and now I would change the way I approached the whole topic. I thought it would be best to start with Behat, Mink and Zombie.js to get results fast. I would rather start with Behat alone to see how BDD works and if I'm used to it add Mink and Goutte to write some acceptence tests. Only if everything works fine, I would add Zombie.js to the setup. That way I guess I would not run into that many errors, where many simply occured to the fact that I didn't know how Behat (or Mink) expected things to be.

Keine Kommentare:

Kommentar veröffentlichen