Here at Compendium, we're devoting a substantial amount of effort into automating our testing process. It started last summer, with the introduction of some basic functional testing with
Selenium IDE. Shortly thereafter, we started using
PHPUnit to verify that our web service endpoints were functioning properly. We've also added some tests for business logic classes. The creation of unit tests for new features became a mandatory process.
In the fall, a server using
CruiseControl was introduced to execute the PHPUnit tests whenever new code was committed. This helped us keep an eye on whether changes were breaking existing functionality, and it reduced the amount of manual testing we would need to do on our testing days.
Lacking in this picture was automation for the functional tests. We had some spare developer time the last couple of weeks, so it became my task to figure out how we could automate functional tests from CruiseControl.
Selenium has grown into a suite of tools that not only allow you to create and execute user interface tests, it also has tools for automating and distributing the effort. As a first pass, we decided to go with
Selenium RC, which is a standalone Java application that runs on a machine where the browser tests will be conducted. The application listens for incoming HTTP requests expressed in a protocol referred to as
"Selenese". Each request corresponds to a Selenium test script operation which is submitted from another host on the network.
Fortunately, developers don't have to learn Selenese to communicate with the Selenium RC server. The
Selenium test API has been mapped into several popular programming languages, which means that you can create a functional test expressed in terms of your favorite language and then run it within the context of a unit testing program on that language.
Moreover, you don't even have to learn the raw API in depth because Selenium IDE has code generation export that allows you to convert a test you just created in the IDE into the target language.
The code generation for PHP just happened to be targeted toward PHPUnit. This was very nice because it allowed us to create the test in Selenium IDE, export into PHP, drop it into our source code tree, and then add another target for running the tests with our build automation tool,
Phing, which is the PHP analogue of
Apache Ant for Java. Once that was done, Cruise Control would be automating the tests.
Unfortunately, there were some bumps on the road to functional testing Nirvana. First, the code generation for Selenium IDE is not perfect. The store operation, which caches a computed value for later use, generated references to PHP variables sans the obligatory dollar sign prefixes. So some of the tests had to be hand edited afterward for syntactical correctness.
More problematic was hooking the tests into Phing. The tests we had created ran fine when invoked from the command line using PHPUnit. But the build rule in Phing resulted in failures. For some reason, the operation to connect to the Selenium RC server was failing. Some debugging revealed that this was because it was attempting to connect to localhost, rather than the machine where the browser tests were to be run.
This issue seemed odd, because we had used a
convention for specifying multiple server parameters that was described in the PHPUnit handbook. According to the documentation, you could define a public static variable named
$browsers to specify an array of settings. Why was this being ignored when run under Phing?
After a lot more spelunking in the source code, I figured out the cause. It turns out that Phing uses a test runner class that is completely independent of the test runner class that comes with PHPUnit. The PHPUnit runner is aware of the $browser variable, while the Phing one is not. Phing simply ignores the variable, leaving the developer with no other option other than to add Selenium API calls in the set up method of the test class.
This didn't make me a happy camper, because that meant that for a given test, we would have to create a separate class for each browser we wanted to support (e.g. one for Firefox on Windows, and another for Internet Explorer). After some grumbling and sketching of class hierarchies, I came up with a workaround.
- Define base classes for each browser type. The set up method would use the Selenium
setBrowser() API to select the browser. - Define subclasses that inhert off the browser base classes for each platform. This would allow us to have different flavors of tests for each platform for browsers that run on multiple OSes, like Safari and Firefox.
- For each functional test, create a common class that has the generated Selenium test exposed as a public static method called
runTest(). - Also for each functional test, create a class for each browser to inherit from the base classes and then call the common
runTest() method.
In essence, this allowed us to get close to multiple inheritance using a language that doesn't support multiple inheritance. Having been a C++ programmer in a former life with the strongly held viewpoint that
multiple inheritance was an abomination, I didn't feel too dirty about this solution.