3.2 Testing
3.2.2 Strengthening the Property
EENI While EENIInit,Halted,«mem is the property we ultimately care about, it does
not lend itself to efficient testing. Much like when proving noninterference we would come up with stronger, potentially inductive, specifications that imply EENI and are easier to prove, the same is true for testing: stronger properties can be much better at finding bugs.
A first observation is that information flows often appear early in counterexam-
ples in the form of a low variation in the stack or the pc, but then it is necessary
to store the leak into the memory to make it observable by our property. By re- defining indistinguishability to take into account the entire machine state, we can obtain shorter counterexamples (that are easier to find):
Definition 3.2.2.1. Machine states S1 = pc1 s1 m1 i1 , S2 = pc2 s2 m2 i2
are indistinguishable with respect to entire low states, writtenS1 «low S2, if either
`pc1 =`pc2 =H or else`pc1 =`pc2 =L, m1 «m2, i1 «i2, s1 «s2, and pc1 «pc2.
A second, dual observation is that all counterexamples begin by pushing ele- ments onto the stack before exploiting some information-flow bug to leak a secret. This is necessary since initial states in our stack machine begin with an empty
stack! By generating quasi-initial states, containing arbitrary (but indistinguish-
able with respect to «low) stacks in addition to memories, we can significantly
improve the average MTTF. However, this approach doesn’t come without a price. When generating such quasi-initial states, there is no guarantee that such a state is
actually reachable from an initial state. In principle, that means that QuickChick
could report spurious problems that cannot actually arise in any real situation. In
general, we can address such problems by carefully formulatinginvariantsof reach-
able states and ensuring that we generate quasi-initial states satisfying them. In practice, though, for this extremely simple machine we did not encounter any spu- rious counterexamples (that was not the case for an extended, more complicated register machine that we tried afterwards!).
Thus, by instantiating EENI appropriately, we obtain a stronger property EENIQInit,HaltedXLow,«low that finds all bugs much faster, with a geometric average
MTTF of 46.48 seconds, an order of magnitude less than the baseline of 334.6.
LLNI Making the full state observable and starting from quasi-initial states sig-
nificantly improved testing performance. However, we can get even better results by moving to a yet stronger noninterference property. The intuition is that EENI generates machines and runs them for a long time, but it only compares the final states, and only when both machines successfully halt; these preconditions lead
to rather large discard rates. On the other hand, comparing intermediate states
as well and reporting a bug as soon as intermediate states are distinguishable can
lead to yet shorter and easier-to-find counterexamples. While the pc is high, the
two machines may be executing different instructions, so their states will natu- rally differ; we ignore these states and require only that low execution states are
pointwise indistinguishable. We call this new property low-lockstep noninterfer-
ence (or LLNI). Benchmarking LLNI in the same bugs reveals a further increase
in performance, with an impressive MTTF of only 7.69 seconds!
SSNI Still, there is more room for improvement! LLNI essentially checks the
following invariant: if two low indistinguishable machines take a step and remain low, then they stay indistinguishable. The drawback is when machines are in a high state they are allowed to differ arbitrarily. In a noninterference proof, an inductive invariant would have to guarantee that after after the machines go back to a low state, they would be low indistinguishable. In our development, this
stronger invariant gives rise to the single-step noninterference property (SSNI).
Definition 3.2.2.2. A machine semantics is single-step noninterfering with re-
spect to an indistinguishability relation « (written SSNI«) if the following condi-
tions (often called unwinding conditions) are satisfied:
1. For all S1, S2 PLow, ifS1 «S2, S1 ñS11, and S2 ñS21, then S11 «S21;
3. For all S1, S2 R Low, if S1 « S2, S1 ñS11, S2 ñS21, and S11, S21 P Low, then
S1 1 «S21.
Note that SSNI talks about completely arbitrary states, not just initial or quasi- initial ones.
The definition above is parametric in the indistinguishability relation used. Finding the right relation can take some work, just like in proofs! Fortunately,
QuickChick can help with this process as well. Low indistinguishability («low)
is too weak and QuickChick can easily find counterexamples to condition 3, e.g.,
by choosing two indistinguishable machine states with i = rReturns, pc = 0, and
s =
”
Rp01,0q@L
ı
; after a single step the two machines have distinguishable pcs 0
and 1, respectively. On the other hand, treating high states exactly like low states in the indistinguishability relation is too strong. In this case QuickChick finds
counterexamples to condition 2, e.g., a single machine state with i =
”
Pop
ı
,
pc = 0, and s=r0@Ls steps to a state withs=rs, which would not be considered
indistinguishable.
These counterexamples show that indistinguishable high states can have differ-
ent pcs and can have completely different stack frames at the top of the stack. So
all we can require for two high states to be equivalent is that their memories and instruction memories agree and that the parts of the stacks below the topmost low return address are equivalent. This is strong enough to ensure condition 3.
Definition 3.2.2.3. Machine states S1 = pc1 s1 m1 i1 , S2 = pc2 s2 m2 i2
are indistinguishable with respect to whole machine states, written S1 «full S2, if
m1 «m2,i1 «i2, `pc1 =`pc2, and additionally
if `pc1 =Lthen s1 «s2 and pc1 «pc2, and
if `pc1 =H then cropStack s1 «cropStack s2.
The cropStack helper function takes a stack and removes elements from the top until it reaches the first low return address (or until all elements are removed).
Using SSNI«full for testing, even with arbitrary starting states, performs very
executed only for one step, we could get away with very small states. Indeed, after fine-tuning the resulting distribution we got a MTTF average of less than 0.5 sec- onds! Once again however, messing with the generation (this time to only produce tiny states) is not without risks. For example, we originally only created instruc- tion memories of size 1 (the single instruction to be executed). Unfortunately, that is not enough to exhibit bugs where the secrets leaked concern instruction memory pointers: without two different instruction memory locations all valid pointers are equal!
MSNI When optimizing the generation for SSNI, we must be extremely cautious
to avoid ruling out useful parts of the state space. Since SSNI operates by executing
a machine state for a single step to check the invariant, generating the complete
state space of pairs of indistinguishable machines becomes very important.
Comparing LLNI and SSNI with respect to their efficiency in testing, we identify an interesting tradeoff. On the one hand, a significant limitation of LLNI is that
bugs that appear when thepc is high are not detected immediately, but only after
the pc goes back low, if ever. One example is the Store-4 buggy rule presented
earlier, where we do not check whether the pc label flows to the label of the
memory cell. On such bugs LLNI demonstrates orders of magnitude worse bug- finding efficiency. On the other hand, SSNI is significantly less robust with respect to starting state generation. If we do not generate every valid starting state, then SSNI will not test executions starting in the missing states, since it only executes one instruction. LLNI avoids this problem as long as all valid states are potentially reachable from the generated starting states.
These observations lead us to formulate one final property that combines the
advantages of both LLNI and SSNI: : multi-step noninterference (MSNI). The for-
mal definition of MSNI is given in the journal version of the paper [66]. Informally, we start from an arbitrary pair of indistinguishable machine states and we check the SSNI invariant along a whole execution trace. Using generation by execution with fine-tuned instruction frequencies for this property leads to an efficiency that is on par with the better of SSNI or LLNI, uncovering IFC violations as soon as they appear; at the same time, unlike SSNI, MSNI is robust against faulty generation.