If your application is conversing with a variety of external systems through a series of different protocols, or if your application itself consists of a series of loosely coupled modules with complex interactions between them, then integration tests become very important. The line between integration testing and component testing is blurry (not least because integration testing is a somewhat overloaded term). We use the term integration testing to refer to tests which ensure that each independent part of your application works correctly with the services it depends on.
Integration tests can be written in the same way as you write normal acceptance tests. Normally, integration tests should run in two contexts: firstly with the system under test running against the real external systems it depends on, or against their replicas controlled by the service provider, and secondly against a test harness which you create as part of your codebase.
It is essential to ensure that you don’t hit a real external system unless you are in production, or you have some way of telling the service that you are sending it dummy transactions for testing purposes. There are two common ways to ensure that you can safely test your application without hitting a real external system, and generally you will need to employ both of them:
• Isolate access to the external system in your testing environment with a firewall, which you probably want to do in any case early on in your devel-opment process. This is also a useful technique to test the behavior of your application when the external service is unavailable.
ptg
• Have a configuration setting in your application that makes it talk to a simulated version of the external system.
In an ideal situation, the service provider will have a replica test service that behaves exactly like the production service, except in terms of its performance characteristics. You can develop your tests against this. However, in the real world, you will often need to develop a test harness of your own. This is the case when:
• The external system is under development but the interface has been defined ahead of time (in these situations, be prepared for the interface to change).
• The external system is developed already but you don’t have a test instance of that system available for your testing, or the test system is too slow or buggy to act as a service for regular automated test runs.
• The test system exists, but responses are not deterministic, and so make validation of tests results impossible for automated tests (for example, a stock market feed).
• The external system takes the form of another application that is difficult to install or requires manual intervention via a UI.
• You need to write standard automated acceptance tests for functionality involving external services. These should almost always run against test doubles.
• The load that your automated continuous integration system imposes, and the service level that it requires, overwhelms the lightweight test environment that is only set up to cope with a few manual exploratory interactions.
Test harnesses can be quite sophisticated, depending, in particular, on whether the service it doubles up for remembers state or not. If the external system remem-bers state, your harness will behave differently according to the requests that you send. The highest-value tests that you can write in this situation are black box tests, in which you consider all the possible responses your external system can give and write a test for each of these responses. Your mock external system needs some way of identifying your request and sending back the appropriate response, or an exception if it gets a request it’s not expecting.
It is essential that your test harness replicates not only the expected responses to service calls, but also unexpected ones. In Release It!, Michael Nygard discusses creating a test harness which simulates the kinds of pernicious behavior you can expect from remote systems that go wrong or from infrastructural problems.3
3. Section 5.7, pp. 136–140.
ptg These behaviors could be due to network transport problems, network protocol
problems, application protocol problems, and application logic problems. Exam-ples include such pathological phenomena as refusing network connections, ac-cepting them and then dropping them, acac-cepting connections but never replying, responding extremely slowly to requests, sending back unexpectedly large amounts of data, replying with garbage, refusing credentials, sending back exceptions, or replying with a well-formed response that is invalid given the state of the appli-cation. Your test harness should be able to simulate each of these conditions, perhaps by listening on several different ports, each of which corresponds to some failure mode.
You should test your application against as many pathological situations as you can simulate to make sure it can handle them. That other patterns the Nygard describes, such as Circuit Breaker and Bulkheads, can then be used to harden your application against the kinds of unexpected events that are bound to occur in production.
Automated integration tests can be reused as smoke tests during deployment of your system into production. They can also be used as diagnostics to monitor the production system. If you identify integration problems as a risk during de-velopment, which they almost inevitably are, developing automated integration tests should be an early priority.
It is essential to incorporate activities concerning integration into your release plan. Integrating with external services is complex and requires time and planning.
Every time you have to integrate with an external system, you add risks to your project:
• Will a test service be available, and will it perform well?
• Do the providers of the service have bandwidth to answer questions, fix bugs, and add custom functionality?
• Will I have access to a production version of the system that I can test against to diagnose capacity or availability problems?
• Is the service API accessible easily using the technology my application is developed with, or will we need specialist skills on the team?
• Are we going to have to write and maintain our own test service?
• How will my application perform when the external service doesn’t behave as expected?
In addition, you will have to add scope for building and maintaining the inte-gration layer and the associated runtime configuration, as well as any test services required and testing strategies such as capacity testing.
ptg
Process
The production of acceptance tests can be an expensive and even laborious task if communication between the team members isn’t effective. Many projects rely on testers examining upcoming requirements in detail, going through all possible scenarios, and designing complex test scripts they will follow later. The results of this process might be sent to the customer for approval, following which the tests are implemented.
There are several points at which this process can be very simply optimized.
We find that the best solution is to have a single meeting with all of the stakehold-ers at the beginning of each iteration, or about a week before a story will start development if you’re not using iterations. We get customers, analysts, and testers in a room together and come up with the highest-priority scenarios to test. Tools like Cucumber, JBehave, Concordion, and Twist allow you to write acceptance criteria down in natural language in a text editor and then write code to make these tests executable. Refactorings to the test code also update the test specifica-tions. Another approach is to use a domain-specific language (DSL) for testing.
This allows acceptance criteria to be entered in the DSL. As a minimum, we will ask the customers to write the simplest possible acceptance tests covering the happy paths of these scenarios there and then. Later, after this meeting, people will often add more sets of data to use to improve the coverage of the tests.
These acceptance tests, and the short descriptions of their objectives, then be-come the starting point for developers working on the stories concerned. Testers and developers should get together as early as possible to discuss the acceptance tests before starting development. This allows developers to get a good overview of the story and understand what the most important scenarios are. This reduces the feedback cycle between developers and testers that can otherwise occur at the end of development of a story and helps reduce both missed functionality and the number of bugs.
The handover process between developers and testers at the end of the story can easily become a bottleneck. In the worst case, developers can finish a story, begin on another story, and be interrupted halfway through the new story by a tester who has raised bugs on the previous story (or even a story that was completed some time ago). This is very inefficient.
Close collaboration between developers and testers throughout the development of a story is essential to a smooth path to the release. Whenever developers finish some functionality, they should call over the testers to review it. The testers should take over the developers’ machine to do this testing. During this time, developers might continue work on an adjacent terminal or laptop, perhaps fixing some outstanding regression bugs. This way they’re still occupied (since testing can take some time), but are easily available in case the tester needs to discuss anything.
ptg