At Compendium we have a RESTful web service API. We have been evolving our testing over time to give us better coverage as well as utility. I thought I would share a couple of our 'best practices' in the hope that other developers may be able to benefit.
Before I get started, a bit of nomenclature. I refer to a service endpoint to imply a URI. For example, /user/dowork might be a service endpoint. Also, I refer to a resource to imply a group of service endpoints. That is, /user might be a resource since there are several operations/endpoints available for it.
1. Create an endpoint runner for each resource
Each resource should have an endpoint runner. The responsibility of an endpoint runner is to implement a client for each endpoint at that resource. The endpoint runner does no testing of its own. Continuing on with our user example, we would have a class called User that had a dowork method. The dowork method signature has all of the required/optional attributes for the service endpoint.
By having an endpoint runner, you can very easily use endpoints within different tests as well as putting basic required functionality in your test fixtures. An example be putting a user create call in your test setUp method and then deleting that user with a user delete call in your tearDown method.
You can also repackage these endpoint runners for distribution or use for sample code in documentation.
2. Create a test case for each endpoint
Now that you have your endpoint runner in place you can create your test case without having to worry about most of the details of the service itself. Each endpoint should get its own test case that will contain all of the tests appropriate to that endpoint.
3. Create Positive and Negative tests for each endpoint test case
A positive test ensures that under the specified conditions the test succeeds (and should succeed). A negative test ensures that under the specified conditions the test fails (and should fail).
When I start writing my tests, I start out by just creating all of the methods, all with empty bodies. Each parameter that can be specified gets a minimum of 3 tests. A positive test that ensures that if provided with an appropriate value the endpoint returns successfully. A negative test that ensures that if provided with invalid data (that is, data of the correct format but nonsense) the service fails appropriately. And a negative test the ensures that if provided with bad data (that is, data not of the correct format) the service fails appropriately.
This starts us out with basic coverage.
4. Test security conditions
This is crucial. Test for authentication AND authorization failures. A canonical example would be a /user/edit endpoint (or probably just POST to /user/username). A user can modify themself. An administrator can modify themself and people in their 'group'. And a 'root' user can modify themself and all other users.
This requires 4 tests. One to ensure that you have to be authenticated (you are logged in) and at least 3 to check the authorization cases. In our case, using a RESTful interface, I would ensure that I got back a 403 (or other appropriate) HTTP code for each test.
5. Create test suites
Your test cases should be included in test suites. Each resource should have a suite containing the test cases for all endpoints. This will help you quickly run all tests for an applicable resource.
6. Use Continuous Integration (and build for it)
You want to make sure that your web service endpoints are always operational. Use your continuous integration solution of choice to run 'builds' on a periodic basis. In this case a build just triggers all tests. We run all tests anytime there is a checkin to the repository. However you can also run scheduled builds.
Before I get started, a bit of nomenclature. I refer to a service endpoint to imply a URI. For example, /user/dowork might be a service endpoint. Also, I refer to a resource to imply a group of service endpoints. That is, /user might be a resource since there are several operations/endpoints available for it.
1. Create an endpoint runner for each resource
Each resource should have an endpoint runner. The responsibility of an endpoint runner is to implement a client for each endpoint at that resource. The endpoint runner does no testing of its own. Continuing on with our user example, we would have a class called User that had a dowork method. The dowork method signature has all of the required/optional attributes for the service endpoint.
By having an endpoint runner, you can very easily use endpoints within different tests as well as putting basic required functionality in your test fixtures. An example be putting a user create call in your test setUp method and then deleting that user with a user delete call in your tearDown method.
You can also repackage these endpoint runners for distribution or use for sample code in documentation.
2. Create a test case for each endpoint
Now that you have your endpoint runner in place you can create your test case without having to worry about most of the details of the service itself. Each endpoint should get its own test case that will contain all of the tests appropriate to that endpoint.
3. Create Positive and Negative tests for each endpoint test case
A positive test ensures that under the specified conditions the test succeeds (and should succeed). A negative test ensures that under the specified conditions the test fails (and should fail).
When I start writing my tests, I start out by just creating all of the methods, all with empty bodies. Each parameter that can be specified gets a minimum of 3 tests. A positive test that ensures that if provided with an appropriate value the endpoint returns successfully. A negative test that ensures that if provided with invalid data (that is, data of the correct format but nonsense) the service fails appropriately. And a negative test the ensures that if provided with bad data (that is, data not of the correct format) the service fails appropriately.
This starts us out with basic coverage.
4. Test security conditions
This is crucial. Test for authentication AND authorization failures. A canonical example would be a /user/edit endpoint (or probably just POST to /user/username). A user can modify themself. An administrator can modify themself and people in their 'group'. And a 'root' user can modify themself and all other users.
This requires 4 tests. One to ensure that you have to be authenticated (you are logged in) and at least 3 to check the authorization cases. In our case, using a RESTful interface, I would ensure that I got back a 403 (or other appropriate) HTTP code for each test.
5. Create test suites
Your test cases should be included in test suites. Each resource should have a suite containing the test cases for all endpoints. This will help you quickly run all tests for an applicable resource.
6. Use Continuous Integration (and build for it)
You want to make sure that your web service endpoints are always operational. Use your continuous integration solution of choice to run 'builds' on a periodic basis. In this case a build just triggers all tests. We run all tests anytime there is a checkin to the repository. However you can also run scheduled builds.
Comments for Effective Web Service Test Coverage
Leave a comment