Capturing specifications with property-based testing

Unit tests can be used as artifacts to document and capture the runtime specification of a function. In the case of computeAverageGrade, for example

QUnit.test('Compute Average Grade', function (assert) { assert.equal(computeAverageGrade([80, 90, 100]),'A');

assert.equal(computeAverageGrade([80, 85, 89]), 'B');

assert.equal(computeAverageGrade([70, 75, 79]), 'C');

assert.equal(computeAverageGrade([60, 65, 69]), 'D');

assert.equal(computeAverageGrade([50, 55, 59]), 'F');

assert.equal(computeAverageGrade([-10]), 'F');

});

you can come up with a simple document that states the following:

■ “If the student’s average is 90 or above, the student is awarded an A.”

■ “If the student’s average is between 80 and 89, the student is awarded a B.”

■ … And so on

Natural language is often used as a means to capture the requirements a system shall fulfill; but natural languages express meaning in a certain context, often not known by all parties, and this generates ambiguity when you try to translate requirements to code. This is why you have to constantly bug product owners or team leads to clarify ambiguities present in task specifications. One of the main causes of ambiguity is a result of adopting an imperative style of documentation when using if-then cases: if case A, then the system should do B. The downside of this approach is that it doesn’t describe the totality of the task to account for all boundary conditions. What if case A doesn’t occur? What is the system expected to do then?

Good specifications shouldn’t be case-based; they should be generic and universal.

Look at the slight difference in wording in these two statements:

■ “If the student’s average is 90 or above, the student is awarded an A.”

■ “Only an average of 90 or above will award the student an A.”

By removing the imperative-case clauses, the second statement is much more com-plete. Not only does it express what happens when the student reaches 90 or above, but it also places the restriction that no other numerical range will result in an A.

You can derive from the second statement that, at the least, any other computed

average won’t result in the student being awarded an A, which you couldn’t intuit from the first.

Universal requirements are much easier to work with, because they aren’t depen-dent on the status of the system at any point in time. For this reason, like unit tests, good specifications don’t have side effects or make assumptions about their surround-ing context.

Referentially transparent specifications increase our understanding of what func-tions are supposed to do and give us a clear picture of the input condifunc-tions they must satisfy. Because referentially transparent functions are consistent and have clear input parameters, they lend themselves to being easily tested with automated mechanisms that can push them to the limit. This brings us into a much more compelling testing modality called property-based testing. A property-based test makes a statement about what the output of a function should be when executed against a definite set of inputs.

The canonical framework or reference implementation is Haskell’s QuickCheck.

By the same token, JavaScript emulates QuickCheck with a library called JSCheck (see the appendix for setup information), by none other than Douglas Crockford,¹ author of JavaScript: The Good Parts (O’Reilly, 2008). JSCheck can be used to create a technical response to a matching referentially transparent specification of a function or pro-gram. Hence, proving the properties of a function is done by generating a large num-ber of random test cases aimed at rigorously exercising all possible output paths of your function.

Also, property-based tests control and manage the evolution of your program as it’s being refactored to ensure that new code doesn’t introduce unintentional bugs into the system. The main advantage of using a tool like JSCheck is that its algorithm gener-ates abnormal datasets to test with. Some of the edge cases it genergener-ates would most likely be overlooked if you had to manually write them.

The JSCheck module is nicely encapsulated into a global JSC object:

JSC.claim(name, predicate, specifiers, classifier)

QuickCheck: Property-based test for Haskell

QuickCheck is a Haskell library for randomized property-based testing of a program’s specification or properties. You design a specification of a pure program in the form of properties the program should fulfill, and QuickCheck generates a large permuta-tion of test cases against your program and produces a report. You can find more information at https://hackage.haskell.org/package/QuickCheck.

1 Douglas Crockford is a popular computer programmer, writer, and speaker best known for his ongoing involvement in the evolution of the JavaScript language, popularizing JSON, and creating several JavaScript libraries like JSLint, JSMin, and JSCheck, among others. He’s also the author of the must-read JavaScript: The Good Parts.

At the heart of this library is the creation of claims and verdicts. A claim is made up of the following:

■ Name—Description of the claim (similar to QUnit’s test description).

■ Predicate—Function that returns a verdict of true when the claim is satisfied or false otherwise.

■ Specifiers—Array describing the type of the input parameters and the specifica-tion with which to generate random datasets.

■ Classifier (optional)—Function associated with each test case that can be used to reject non-applicable cases

Claims are passed into JSCheck.check to run random test cases. This library wraps creating a claim and feeding it into the engine in a single call to JSCheck.test, so you’ll use this shortcut method in the example tests. Let’s look at an example of writ-ing a simple JSCheck specification for computeAverageGrade that captures the follow-ing specification: “Only an average of 90 or above will award the student an A.”

JSC.clear();

JSC.on_report((str) => console.log(str));

JSC.test(

'Compute Average Grade',

function (verdict, grades, grade) {

return verdict(computeAverageGrade(grades) === grade);

As you can see in listing 6.5, you use declarative specifiers to capture the properties of this program:

■ JSC.array—Describes that the function expects inputs of Array type.

■ JSC.integer(20)—Indicates the maximum length this function is expected to work with. In this case, it’s arbitrary, so any number from 1 to 20 will suffice.

Listing 6.4 Property-based test for computeAverageGrade

I always like to start with JSC.clear to initialize and start a fresh testing context.

Name of the claim

Passes the predicate function the verdict object that defines the condition to verify

Signature or specifier array describing the contract for generating averages that deserve an A

Classifier function runs on each test, so you can use it to append data to the report

■ JSC.number(90, 100)—Describes the types of elements in the input array. In this case, they’re numeric (including integers and floating-point numbers) in the range from 90 to 100.

The predicate function is a bit tricky to understand. The predicate returns a true verdict when a claim holds, but what happens in the body of the predicate is for you to determine depending on your specific program and what you want it to verify. In addition to the verdict function used to announce the result of the test case, you’re also given the generated random input and the expected output. In this case, the result you want to announce is the check to validate that computeAverageGrade returns the expected grade: A. This example uses a few specifiers, but there are many more you can read about on the project’s website, and you can also create your own.

Now that you understand the main pieces of the program, let’s go ahead and run it. The report can be lengthy, because JSCheck will generate by default 100 random test cases based on the specification provided. I’ve trimmed it, but you can still follow what’s happening:

Compute Average Grade: 100 classifications, 100 cases tested, 100 pass Testing for an A on grades:

90.042,98.828,99.359,90.309,99.175,95.569,97.101,92.24 pass 1 Testing for an A on grades:

90.084,93.199, pass 1 // and so on 98 more times Total pass 100, fail 0

JSCheck programs are self-documented; you can easily describe the contract for your function’s inputs and outputs to a level regular unit tests can’t. You can also see the significant level of detail that a JSCheck report contains. JSCheck programs can run as standalone scripts or embedded into QUnit tests; that way, they can be included as part of your test suites. The interaction between these libraries is shown in figure 6.8.

In the next example, you’ll use JSCheck to test the checkLengthSsn program, which has the following specification:

■ A valid Social Security number must satisfy these conditions:

– Contains no spaces – Contains no dashes – Is nine characters long

– Follows the format outlined by ssa.gov, composed of three parts:

1 The first set of three digits is called the Area Number.

2 The second set of two digits is called the Group Number.

3 The final set of four digits is called the Serial Number.

The following listing shows the code; then I explain the relevant parts.

QUnit.test('JSCheck Custom Specifier for SSN', function (assert) { JSC.clear();

JSC.on_report((report) trace('Report'+ str));

JSC.on_pass((object) => assert.ok(object.pass));

JSC.on_fail((object) =>

assert.ok(object.pass || object.args.length === 9,