Details of parallel verification algorithm sub-procedures

We now describe in further detail the sub-procedures used byParallelVerifyand that are shown in Figure 5.5. Recall that the parent procedureParallelVerifyspawnsNumWorkers+ 2child threads which run the proceduresTrainingSelector,NodeSchedulerandVerifyWorker. We omit the details of

TrainingSelector_{here, it is described in Chapter 4.}

Management of nodes inNodeScheduler

ParallelVerifyspawns a single thread to run the procedureNodeScheduler, shown starting on Line 400 of Figure 5.5. This procedure manages the selection of nodes to execute next and maintains the flow of nodes between worker threads. There are two queues of nodes, a “ready” queueQRand an “added” queueQA. These queues are shared between the worker threads and theNodeScheduler thread. Worker threads pull nodes fromQR and push new nodes onto QA. There is only one scheduler thread and one or more worker threads producing and consuming nodes from the queues

QRandQA; hence,QRis a single-producer-multi-consumer queue andQAis a multi-producer- single-consumer queue. The goal ofNodeScheduleris to keepQAempty andQR“full,” and we will define what we mean by this below. Nodes are in one of four possible states, either actively being explored insideVerifyWorker, stored inQR, stored inQAor stored inLive. A node at the front of

QRis the highest priority node not currently being explored. The nodes inQAare “infant” nodes that have been created byVerifyWorkerthreads and need to be added toLive. The remaining nodes stored inLiveare the candidate nodes.

Upon initialization, the procedure NodeSchedulercreates a root node and adds it to a set of nodes calledLive(401–403). After initialization, the procedureNodeSchedulerhas three cases of execution. First, the condition on line 405 checks ifLiveis non-empty and if there are more worker threads waiting to execute than nodes inQR, i.e., the queueQRis not “full”; if this condition is true, we callSelectNode(see Chapter 4 for details) and atomically append the result toQR. Second, if

QAis non-empty, its members are dequeued and added toLive(408–409). Finally, the condition on line 410 is only true if no worker threads are active, both queues are empty and there are no remaining states to execute; when this condition is met, all paths of exploration rooted atσn−1have been exhaustively explored and a termination condition has been met. The booleanFinishedwill

be set totrue, forcing all threads to exit. The parent thread executingParallelVerifywill return⊥in this case.

Building execution fragments inVerifyWorker

Shown starting on Line 413 of Figure 5.5, the procedureVerifyWorkerdoes the main work of client verification: stepping execution forward in the stateσof each node. LikeNodeScheduler, the procedureVerifyWorkerruns inside of a while loop until the value ofFinishedis no longer equal to

false(414). Recall that the parent procedureParallelVerifyspawns multiple instances ofVerifyWorker. Whenever there is a node on the queueQR, the condition on line 415 will be true and the procedure callsdequeueatomically. Note that even if|QR| = 0, multiple instances ofVerifyWorkermay call

dequeue, but only one will return a node, the rest will retrieve undefined (⊥) fromdequeue.

If ndis not undefined, the algorithm proceeds to execute the state nd.stateand extend the associated pathnd.pathup to either the next network instruction (send or recv) or the next symbolic branch (a branch instruction that is conditioned on a symbolic variable). The first case, stepping execution on non-network / non-symbolic-branch instructions, executes in a while loop on lines 420–422. The current instruction is appended to the path and the procedureexecStepis called, which symbolically executes the next instruction in stateσ. These lines, are where the majority of the computation work is done by the verifier (see Figure 4.5 in Chapter 4) and viewed as the “hot” path of the verification algorithm. The ability to concurrently step execution on multiple states is where the largest performance benefits of parallelization are achieved. Note that calls to

execStepmay invoke branch instructions, but these are non-symbolic branches. In the second case, if the next instruction is send or recv and if the constraintsσ.constraintsaccumulated so far with the symbolic stateσdo not contradict the possibility that the network I/O messageσ.next.msgin the next instructionσ.nextismsgn(i.e.,(σ.constraints∧σ.next.msg=msgn)6⇒false, line 423), then the algorithm sets the termination value (Finished=true) and sets the return value of the parent function (Valid← π k hσ.nexti). All other threads of execution now exit becauseFinished= true and the parent procedureParallelVerifywill returnValid, which is now an execution fragment that meets the verifier’s goals successfully.

In the final case, (isSymbolicBranch(σ.next) = true), the algorithm is at a symbolic branch. Thus, the branch condition contains symbolic variables and cannot be evaluated as true or false

in isolation. Using symbolic execution, the algorithm evaluates both the true branch and the false branch by executingσ.nextconditioned on the condition evaluating tofalse(denoted[execStep(σ)| σ.next.cond7→false]in line 428) and conditioned on the branch condition evaluating totrue(432). In each case, the constraints of the resulting state are checked for consistency (430, 434). If either state is consistent, it is atomically placed ontoQA(431, 435).