Recommended Mindset for Evaluation

Evaluation is often associated with the end of predictive life cycles (e.g., user acceptance testing and release of a solution). For iterative or adaptive life cycles, evaluation may be associated with the end of a time segment (e.g., an iteration or sprint), or when a user story is delivered or completed for an adaptive approach that has no fixed time segments, such as Kanban. When evaluation criteria are defined early, they can be applied to a project in progress and provide input into the development of test strategies, test plans, and test cases. Early uses of lightweight evaluation techniques identify which areas of a solution need the most testing. Early articulation of evaluation criteria is an excellent way to specify or confirm both functional and nonfunctional requirements.

6.3.2 Treat Requirements Analysis, Traceability, Testing, and Evaluation as

Complementary Activities

Adaptive life cycles explicitly define acceptance criteria with concrete examples as part of the elaboration of a user story. The acceptance criteria and the user story definition support each other. Together, these establish mutual agreement between business stakeholders and those responsible for developing the solution for what is required and how to know that the requirement has been met. Predictive and iterative life cycles also recognize the value of early testing in a project.

Early test specifications provide concrete examples that add clarity to requirements, regardless of how they are specified.

Formal traceability matrices provide verification that requirements support business goals and objectives and that testing sufficiently covers the requirements. Goals and objectives and the traces between them and the requirements that support them provide insight into candidate metrics for evaluation.

Collaboration Point—Business analysts, project managers, and testers should work together early on to create consistent and complementary approaches to analysis, testing, and evaluation activities. Testers can help check for completeness and consistency of all forms of requirements and evaluation criteria, and business analysts can help check for sufficient test coverage for areas of the solution that have the highest business priority.

6

6.3.3 Evaluate with the Context of Usage and Value in Mind

Validating a solution is more complex than validating to determine whether individual requirements have been met. Validation not only ensures that the solution is working as designed, but also confirms that it enables the usage and value that the business expected. No matter how requirements are elicited and specified, there is some intended usage associated with each requirement or group of requirements. The names of epics, user stories, and use case scenarios that focus on functionality provide direct identification of the intended usage. For epics and user stories that are not about functionality, usage is inferred from the “so I can...” part of the story, which provides its rationale.

Example—The following epic is written for the insurance company discussed earlier in Section 2. It describes a need for information regarding the adjudication of claims. The “so I can” statement alludes to the usage of the data:

“As an operations manager, I need data about the percentage of claims that were automatically adjudicated and the average time for a manual adjudication, so that I can assess how the revised adjudication practices are performing.”

Nonfunctional requirements that define expectations for the performance or operation of a solution also have an implied context of usage, because a solution is expected to perform or be available, scalable, or usable so that work can be accomplished.

6.3.4 Confirm Expected Values for Software Solutions

For automated solutions, it is essential to validate functionality that manipulates data by using the solution’s data retrieval functionality. However, the sufficiency of the solution also needs to be evaluated by confirming that the actual data is accurately stored, because that data may potentially be retrieved in other ways for other users and uses.

Example—Consider the insurance company example, where the business need is to provide a software solution so that a claim can be submitted via a mobile device. Part of the evaluation of the solution is to look at the results of tests where a claimant submits a claim using the mobile device and also look at the results of tests where the claim is retrieved using the mobile device to ensure that the actual claim is presented as expected. However in an insurance company, a claim can be retrieved in other ways, such as through a reporting mechanism or another interface such as a claim adjuster’s workstation. When those usages are not part of the scope of the solution, it is still part of the scope to accurately store the claim data so that it is available for any usage.

To verify that the data is accurately stored and available for other forms of retrieval, the results that were obtained by accessing the data directly from its storage location should be reviewed to confirm that the actual stored values match the expected stored values. Even when direct confirmation of stored data is included during testing, it is worthwhile to confirm it again as part of evaluating the most-used

portions of the functionality. Evaluation of a software solution is usually conducted against a system that is either about to be released or has already been released; therefore, testing during evaluation usually encompasses a larger body of data than the initial testing. As a result, new and different kinds of data anomalies may be uncovered by looking at the results of tests used to directly confirm the data, and these anomalies will need to be analyzed.