6.5 Verification
7.1.1 Partial Specification Refinement
We have implemented a verification system with Objective Caml to evaluate our ap- proach to partial specification refinement. Our experimental results were achieved with an Intel Core 2 CPU 2.66GHz with 8Gb RAM. The four columns inTable 7.1 andTable 7.2 describe respectively the analysed programs, the analysis time in sec- onds, and the methods’ (given and inferred) preconditions and postconditions. All formulae with a grey background are inferred by our analysis. For some programs,
7.1. Experimental Results
Prog. Time Pre Post
create∗ 0.379 emp ∧ n≥0 res::llBhSi ∧ n=|S| ∧ ∀v∈S·1≤v≤n 1.752 emp ∧ n≥0 res::dllBhr,Si∧ n=|S|∧∀v∈S·1≤v≤n 0.954 emp ∧ n≥0 res::sllB2hSi∧ n=|S|∧∀v∈S·1≤v≤n sort ∗ insert 0.591 x::llhni ∧ n≥1 x::llhmi ∧ m=n+1 0.789 x::dllhp, ni ∧ n≥1 x::dllhq, mi ∧ n≥1 ∧ m=n+1 ∧ p=q 0.504 x::sllhn, xs, xli∧ v≥xs x::sllhm, mn, mxi ∧ xs=mn ∧ mx=max(xl,v)∧m=n+1 tail insert 0.566 x::llhni ∧ n≥1 x::llhmi ∧ m=n+1 0.628 x::sllhn, xs, xli∧ v≥xl x::sllhm, mn, mxi ∧ v=mx∧ mn=xs ∧ m=n+1 rand ∗ insert 0.522 x::llhni ∧ n≥1 x::llhmi ∧ m=n+1 0.830 x::dllhp, ni ∧ n≥1 x::dllhq, mi ∧ m=n+1 ∧ p=q — x::sllhn, xs, xli∧ (fail) x::sllhm, mn, mxi∧ (fail)
delete 0.630 x::llBhSi ∧ |S|≥2 x::llBhTi ∧ ∃a.S=Tt{a} 1.024 x::sllBhSi ∧ |S|≥2 x::sllBhTi ∧ ∃a.S=Tt{a}
delete 1.252 x::dllBhp, Si ∧ |S|≥2 x::dllBhq, Ti ∧ ∃a.S=Tt{a} ∧ p=q
travrs
0.296 x::llhmi∧ n≥0∧m≥n x::lshp, ki ∗ res::llhri ∧ p=res∧ k=n ∧ m=n+r
2.205 x::sllBhSi ∧ n≥0∧|S|≥n x::slsBhp, Ti ∗ res::sllBhS2i∧ p=res
∧|T|=n∧S=TtS2∧∀u∈T,v∈S2·u≤v
append∗
0.512 x::llhxni∗y::llhyni∧ xn≥1 x::llhmi ∧ m=xn+yn
0.660 x::dllhxp, xni ∗
y::dllhyp, yni ∧ xn≥1 x::dllhq, mi ∧ m=xn+yn ∧ q=xp
0.948 x::sllhxn, xs, xli∧ xl≤ys
∗ y::sllhyn, ys, yli
x::sllhm, rs, rli ∧ yl=rl ∧ m≥1+yn ∧ m=xn+yn dispatch3 0.786 lst::ll2hn, si gtl 0::ll2hn 1,s1i∗ltl0::ll2hn2,s2i∧ n= n1+n2∧s=s1+s2∧s1≥3n1∧s2<3n2+1
7.1. Experimental Results
Prog. Time Pre Post
travrs 0.532 x::bthS, hi x::bthT, ki ∧ S=T ∧ h=k
count 0.709 x::bthS, hi x::bthT, ki ∧ res=|S| ∧ S=T ∧ h=k height 0.913 x::bthS, hi x::bthT, ki ∧ res=h=k ∧ S=T insert 1.276 x::bthS, hi ∧ |S|≥1 ∧ h≥1 x::bthT, ki ∧ T=St{v} ∧ h≤k≤h+1 delete 0.970 x::bthS, hi ∧ |S|≥2 ∧ h≥2 x::bthT, ki∧ ∃a·S=Tt{a}∧h−1≤k≤h
search 1.583 x::bsthsm, lgi x::bsthmn, mxi ∧ sm=mn ∧ lg=mx ∧ 0≤res≤1 bst insert 1.720 x::bsthsm, lgi x::bsthmn, mxi ∧ (v<sm∧v=mn ∧ lg=mx ∨ lg<v∧v=mx∧sm=mn ∨ sm=mn∧lg=mx)
avl ins 11.12 x::avlhS, hi res::avlhT, ki∧ T=St{v}∧h≤k≤h+1
sdl2nbt 5.826 x::sdlBhp,q,Si∧ |S|≥1∧
p=null ∧ q=tail res::nbthTi ∧ T=S Table 7.2: Experimental results for tree-processing programs.
we have verified them with different pre/post shape templates. Programs with star
∗ have different versions for various data structures.
The results highlight the refinement of both pre- and postconditions based on user- provided shape specifications, even for complicated data structure such as AVL trees. Firstly, our approach can compute non-trivial pure constraints for postcondition. For example, the program create’s code is listed as follows:
Node create(int n) {
if (n == 0) return null; else {
Node r = create(n - 1); return new Node(n, r); }
7.1. Experimental Results
}
After the refinement we obtain the value range in the created list (between one and n). For delete which removes a random node from the list, we know the content of the result list is subsumed by that of the input list. For a sample program dispatch3 fromBouajjani et al. (2010) which divides a list into two lists where one only con- tain values no less than three and the other less than three, we obtain a detailed quantitative relationship n=n1+n2∧s=s1+s2∧s1≥3n1∧s2<3n2+1. For list-sorting
algorithms, we confirm the content of the output is the same as that of the input. For tree-processing programs (insert, delete and avl ins), we obtain that the height difference between the input and output trees is at most one. Meanwhile, we can calculate non-trivial requirements in precondition for memory safety or func- tional correctness. For many programs we can get the constraints for precondition such that the input list/tree should have at least one (or two) nodes for the sake of memory safety. A more interesting example is the travrs method which takes in a list with length m and an integer n and then traverses towards the tail of the list for n steps:
Node travrs(Node x, int n) { if (n == 0) return x;
else return travrs(x.next, n - 1); }
For this program, the analysis discovers m≥n in the precondition to ensure memory safety. Another example is the append method concatenating two sorted lists into one:
7.1. Experimental Results
Node w = x.next;
if (w == null) x.next = y; else append(w, y);
}
To ensure that the result list is sorted, the analysis figures out that the minimum value in the second list must be no less than the maximum value in the first list. Without those discovered constraints the program will either cause memory violation or cannot meet the specification.
A second highlight is our flexibility by supporting multiple predicates. Our analysis tries to refine different specifications for the same program at various correctness lev- els (with different predicates), for instance sort insert, tail insert and append. For rand insert, which inserts a node into a random place (after the head) of a list, we confirm that the list’s length is increased by one, but cannot verify the list is kept sorted if it was before the insertion, as the result indicates.
We have also tried our approach over part of the FreeRTOS kernel (Barry,2006). For its list processing programs list.h and list.c (472 lines with intensive manipulation over composite sorted doubly-linked lists) it took 2.85 seconds for our system to refine all the specifications given for the main functions, which further confirms the viability of our approach.
7.1. Experimental Results
Prog. Time Pre Post
create (main) emp ∧ n≥0 res::sllB2hSi∧ n=|S|∧∀v∈S·1≤v≤n
alloc 0.214 emp res::Nodehn, ri
travrs (main) x::sllBhSi ∧ n≥0∧|S|≥n x::slsBhp, Ti ∗ res::sllBhS2i∧ p=res
∧|T|=n∧S=TtS2∧∀u∈T,v∈S2·u≤v
next 0.135 x::sllBhSi ∧ |S|≥0 x::Nodehv, resi ∗ res::sllBhS1i ∧ S={v}tS1
append (main) x::llhxni∗y::llhyni∧ xn≥1 x::llhmi ∧ m=xn+yn
travrs 0.619 x::llhni ∧ n≥1 x::lshres, mi ∗ res::Nodehv, nulli Sorting (main) x::llBhSi ∧ |S|≥1 res::sllBhTi ∧ T=S (‡) merge 4.107 x::sllBhSxi ∗ y::sllBhSyi res::sllBhTi ∧ T=SxtSy
flatten 2.693 x::bstBhSi res::sllBhTi ∧ T=S insert 0.824 r::sllBhSi ∗ x::Nodehv, i res::sllBhTi ∧ T=St{v}
Table 7.3: Experimental results for list-processing programs.
7.1.2
Specification Synthesis for Auxiliary Methods
and Loops
For specification synthesis, we have extended the system in the previous section for evaluation purpose. Our experimental results were achieved with the same hard- ware environment. The four columns inTable 7.3describe respectively the analysed programs, the analysis time in seconds, and the methods’ (given and inferred) pre- conditions and postconditions. The programs whose names have (main) denote the primary procedures (without a timing report as most of them are already shown in the previous section). The programs with time information exhibit the auxil- iary procedures whose specifications are wholly synthesised and inferred (as their background is totally grey).
7.1. Experimental Results
Prog. Time Pre Post
bst
insert (main) x::bsthsm, lgi
x::bsthmn, mxi ∧ (v<sm∧v=mn ∧ lg=mx ∨ lg<v∧v=mx∧sm=mn ∨ sm=mn∧lg=mx)
alloc 0.255 emp res::Node2hv, null, nulli
avl ins (main) x::avlhS, hi res::avlhT, ki∧ T=St{v}∧h≤k≤h+1 height 0.775 x::avlhS, hi x::avlhT, ki ∧ T=S ∧ h=k=res
sdl2nbt (main) x::sdlBhp,q,Si∧ |S|≥1∧
p=null ∧ q=tail res::nbthTi ∧ T=S
loop1 0.496 head::sdlBhp,q,Si∧ p=null ∧ q=tail∧ head=root=end head::sdlBhnull, q, Shi ∧ end=tail∗ root::sdlBhp, tail, Sri ∧ S=ShtSr∧ (∀x∈Sh, y∈Sr·x≤y) ∧ 0≤|Sh|−|Sr|≤1
Table 7.4: Experimental results for tree-processing programs.
user annotations by synthesising specifications for auxiliary methods, given raw specifications of primary methods. For example, we have manually set some of the instructions in the programs in the previous section as auxiliary procedures and tried to generate their specifications, such as alloc, next and travrs in list-processing programs and alloc for bst insert. For new test suites, we have analysed a number of list-sorting algorithms with at least one auxiliary method each. Note that all the list-sorting algorithms have the same specification for their primary methods (line ‡), while the annotations found by our approach for auxiliary methods are diverse (the grey lines below ‡). From the automatically derived specifications, we can see that the auxiliary method merge for merge sort merges two sorted lists into one, and the auxiliary method flatten for tree sort flattens a binary search tree into a sorted list. As another example, avl ins also has some auxiliary (recursive) methods such as calculation of tree’s height and double rotation, which are automatically analysed as well.
7.1. Experimental Results
tions are quite complex if written by hand, for example loop1. If we require users to provide these specifications for auxiliary methods (even shape-only), it is still quite tedious and error-prone. On the contrary, the users now have the option to leave such work for our system, which witnesses the value of our improvement of the verification process.
7.1.3
Verification of Programs with Unknown Components
For the verification of program with unknown components, we have implemented the verification algorithms and the abstract semantics with Objective Caml and eval- uated them over some heap-manipulating programs. The results are in Table 7.5, Table 7.6, Table 7.7, Table 7.8 and Table 7.9. In each table, the first and sec- ond columns denote the programs used for evaluation and their time consumption, respectively. During the experiments, we manually hide some instructions in the original programs as calls to unknown procedures, whose specifications we try to discover during the verification process. Accordingly, the third column in the first two tables contain both the specifications of the programs to be verified (upper line), and the derived specifications for the unknown procedure (lower line). For the third table, as we use the same specification x::llBhSi ∗→ res::sllBhTi ∧ T=S to verify all the sorting algorithms, the third column (from the second line on) states the discov- ered specification for the unknown call only. Similar as previous examples, some of the programs with the same name have different versions, say, the ones processing (sorted) singly-linked lists (ll and sll) are different from their counterparts for doubly-linked lists (dll).
It can be seen that all programs are successfully verified, with some obligations on the unknown calls discovered. We note down two observations on the experimental results. The first is that the discovered specifications for the unknown procedures
7.1. Experimental Results
Prog. Time Main spec. (Φpr∗→ Φpo) & Derived unknown spec. (Φupr∗→ Φupo)
create∗
0.405 emp ∧ n≥0 ∗→ res::llBhSi ∧ n=|S| ∧ ∀v∈S·1≤v≤n emp ∧ a≥1 ∗→ res::Nodehc, bi ∧ 1≤c≤n
1.895 emp ∧ n≥0 ∗→ res::dllBhrp, Si ∧ n=|S| ∧ ∀v∈S·1≤v≤n emp ∧ a≥1 ∗→ res::Node2hc, d, bi ∧ 1≤c≤n
1.020 emp ∧ n≥0 ∗→ res::sllB2hSi ∧ n=|S| ∧ ∀v∈S·1≤v≤n emp ∧ a≥1 ∗→ res::Nodehc, bi ∧ a−1≤c≤a
sort ∗
insert
0.667 x::llhni ∧ n≥1 ∗→ x::llhmi ∧ m=n+1
a::Nodehb, ci ∗ c::llhdi ∗→ a::Nodehb, ei ∗ e::llhd+1i
0.842 x::dllhp, ni ∧ n≥1 ∗→ x::dllhq, mi ∧ n≥1 ∧ m=n+1 ∧ p=q
a::Node2hb, g, ci ∗ c::dllha, di ∗→ a::Node2hb, g, ei ∗ e::dllha, d+1i
0.764 x::sllhn,xs,xli ∧ v≥xs ∗→ x::sllhn+1,mn,mxi ∧ mn=xs ∧ mx=max(xl,v) a::Nodehb,ci∗c::sllhd,g,hi∧b≤f≤g ∗→ a::Nodehb,ei∗e::sllhd+1,f,hi tail insert 0.498 x::llhni ∧ n≥1 ∗→ x::llhmi ∧ m=n+1 a::Nodehb, nulli ∗→ a::llh2i
0.627
x::sllhn, xs, xli ∧ v≥xl ∗→
x::sllhm, mn, mxi ∧ v=mx ∧ mn=xs ∧ m=n+1 a::Nodehb, nulli ∧ b≤c ∗→ a::sllh2, b, ci
rand ∗
insert
0.514 x::llhni ∧ n≥1 ∗→ x::llhmi ∧ m=n+1
a::Nodehb, ci ∗ c::llhdi ∗→ a::Nodehb, ei ∗ e::llhd+1i
0.697 x::dllhp, ni ∧ n≥1 ∗→ x::dllhq, mi ∧ m=n+1 ∧ p=q
a::Node2hb, g, ci ∗ c::dllha, di ∗→ a::Node2hb, g, ei ∗ e::dllha, d+1i Table 7.5: Experimental results (lists, part 1).
7.1. Experimental Results
Prog. Time Main spec. (Φpr∗→ Φpo) and Derived unknown spec. (Φupr∗→ Φupo)
delete∗
0.646 x::llBhSi ∧ |S|≥2 ∗→ x::llBhTi ∧ ∃a·S=Tt{a}
a::Nodehb, ci ∗ c::Nodehd, ei ∗ e::llBhEi ∗→ a::Nodehb, ei ∗ e::llBhEi
0.916
x::sllBhSi ∧ |S|≥2 ∗→ x::sllBhTi ∧ ∃a·S=Tt{a}
a::Nodehb, ci ∗ c::Nodehd, ei ∗ e::sllBhEi ∧ ∀f∈E·b≤d≤f ∗→ a::Nodehb, ei ∗ e::sllBhEi ∧ ∀f∈E·b≤f
1.430
x::dllBhp, Si ∧ |S|≥2 ∗→ x::dllBhq, Ti ∧ ∃a·S=Tt{a} ∧ p=q a::Node2hb, f, ci ∗ c::Nodehd, a, ei ∗ e::dllBhc, Ei ∗→
a::Nodehb, f, ei ∗ e::dllBha, Ei
append
0.523 x::llhxni ∗ y::llhyni ∧ xn≥1 ∗→ x::llhmi ∧ m=xn+yn a::llhbi ∗→ a::lshres, ci ∗ res::llhdi ∧ b=c+d
0.861
x::sllhxn, xs, xli ∗ y::sllhyn, ys, yli ∧ xl≤ys ∗→ x::sllhm, rs, rli ∧ yl=rl ∧ m≥1+yn ∧ m=xn+yn
a::sllBhAi ∗→ a::slsBhA1i ∗ res::sllBhRi ∧ A=A1tR ∧ ∀b∈A1,c∈R·b≤c
Table 7.6: Experimental results (lists, part 2).
are usually more general than what we expect. Bear in mind that we have replaced some instructions from those programs with unknown calls. We have compared the inferred specifications for those unknown calls with the original instructions. The re- sults show that the specifications derived by our algorithm not only fully capture the behaviours of those instructions, but also suggest other possible implementations. A case in point is list’s append:
void append(Node x, Node y) { Node w = unknown(x);
if (w.next == null) w.next = y; else append(w.next, y);
}
7.1. Experimental Results
Prog. Time Main spec. (Φpr∗→ Φpo) and Derived unknown spec. (Φupr∗→ Φupo)
Binary tree processing programs
travrs 0.417
x::bthS, hi ∗→ x::bthT, ki ∧ S=T ∧ h=k
a::bthA, bi ∧ a6=null ∗→ a::Node2hc, d0, e0i ∗ d0::bthD, fi ∗ e0::bthE, gi ∧
A={c}tDtE ∧ b=max(f,g)+1, or
a::bthA, bi ∧ a6=null ∗→ a::Node2hc, e0, d0i ∗ d0::bthD, fi ∗ e0::bthE, gi ∧
A={c}tDtE ∧ b=max(f,g)+1
count 0.705
x::bthS, hi ∗→ x::bthT, ki ∧ res=|S| ∧ S=T ∧ h=k
a::bthA, bi ∧ a6=null ∗→ a::Node2hc, d0, e0i ∗ d0::bthD, fi ∗ e0::bthE, gi ∧
A={c}tDtE ∧ b=max(f,g)+1, or
a::bthA, bi ∧ a6=null ∗→ a::Node2hc, e0, d0i ∗ d0::bthD, fi ∗ e0::bthE, gi ∧
A={c}tDtE ∧ b=max(f,g)+1
height 0.821
x::bthS, hi ∗→ x::bthT, ki ∧ res=h=k ∧ S=T
a::bthA, bi ∧ a6=null ∗→ a::Node2hc, d0, e0i ∗ d0::bthD, fi ∗ e0::bthE, gi ∧
A={c}tDtE ∧ b=max(f,g)+1, or
a::bthA, bi ∧ a6=null ∗→ a::Node2hc, e0, d0i ∗ d0::bthD, fi ∗ e0::bthE, gi ∧
A={c}tDtE ∧ b=max(f,g)+1
insert 1.354 x::bthS, hi ∧ |S|≥1 ∧ h≥1 ∗→ x::bthT, ki ∧ T=St{v} ∧ h≤k≤h+1 a::Node2hb, null, nulli ∗→ a::bthA, 2i ∧ A={b, c}
delete 1.019
x::bthS, hi ∧ |S|≥2 ∧ h≥2 ∗→ x::bthT, ki ∧ ∃a · S=Tt{a} ∧ h−1≤k≤h a::Node2hb,c,nulli∗c::Node2hd,null,nulli ∗→ a::Node2hb,null,nulli, &
a::Node2hb,null,ci∗c::Node2hd,null,nulli ∗→ a::Node2hb,null,nulli Table 7.7: Experimental results (trees, part 1).
7.1. Experimental Results
Prog. Time Main spec. (Φpr∗→ Φpo) and Derived unknown spec. (Φupr∗→ Φupo)
Binary search tree and AVL tree processing programs
search 1.851 x::bsthsm, lgi ∗→ x::bsthmn, mxi ∧ sm=mn ∧ lg=mx ∧ 0≤res≤1
a::bsthb, ci∧a6=null ∗→ a::Node2hd, e0, f0i∗e0::bsthb, gi∗f0::bsthh, ci∧g≤d≤h
bst
insert 1.822
x::bsthsm, lgi ∗→ x::bsthmn, mxi ∧ (v<sm∧v=mn∧lg=mx ∨ lg<v∧v=mx∧sm=mn ∨ sm=mn∧lg=mx) a::Node2hb, null, ci ∗ c::bsthd, ei ∧ f<b<d ∗→ a::bsthf, ei, and a::Node2hb, c, nulli ∗ c::bsthd, ei ∧ e<b<f ∗→ a::bsthd, fi
avl ins 5.202 x::avlhS, hi ∗→ res::avlhT, ki ∧ T=St{v} ∧ h≤k≤h+1 a::avlhA, bi ∗→ a::avlhA, bi ∧ res=b
sdl2nbt 5.238
x::sdlBhp,q,Si ∧ |S|≥1 ∧ p=null ∧ q=tail ∗→ res::nbthTi ∧ T=S
a::sdlBhnull, c, Ai ∧ a=b=d ∗→ a::sdlBhnull, b, A1i ∗ b::sdlBhe, c, Bi ∧
d=c ∧ A=A1tB ∧ (∀f∈A1, g∈B·f≤g) ∧ 0≤|B|−|A1|≤1
Table 7.8: Experimental results (trees, part 2).
Armed with list segment predicates and corresponding lemmas, we are able to infer that the unknown call may actually traverse the list for arbitrary number of nodes, provided it does not go beyond the list’s tail. To conclude, our verification always tries its best to find a sound (with respect to the program being verified) and general (with respect to the unknown call) specification for unknown procedures invoked.
The second observation is that the precision of specifications discovered for unknown procedures depends on their callers’ given specification. As can be seen we have verified several list-processing programs where each one has various specifications. Within these programs we want to point out that the ones with specifications of both normal lists and sorted lists share the same code (but just with two different specifications). Such examples include create, sort insert, delete, and so on. For create, it creates a list containing numbers from 1 to n in descending order: