Performance Analysis of the WFSC Method 76

3.2.4.1

Sensitivity to number of cut-off velocity samples

To investigate the effect of the number of cut-off velocity samples on the performance of the WFSC method, synthetic WFSC data were produced by sampling the theoretical WFSC curve (Eqn. 3-2) at equal intervals from 10 to 200 samples in

increments of 10 samples for vc = 0 to 18 mm/s. Highly oversampled curves containing 1000 samples were also produced. Each sampled curve was processed using the automatic WFSC algorithm [2] to determine the locations along the vc axis of the beginning and end of the characteristic interval. For each of the sampled curves, two characteristic interval detection errors were defined as the differences between the cut-off velocities at the start and end of the detected intervals and their corresponding cut-off velocities for curves of 1000 samples. The mean interval detection error over the 2080 curves times 2 errors/curve was plotted as a function of the number of samples. The minimum sufficient number of samples was identified as the first point at which the mean error curve changes its slope from negative to positive, at which point the curve has approximately converged to a steady error value.

3.2.4.2

Sensitivity to frame-to-frame variability of color pixel density

To investigate the sensitivity of the WFSC method to variability in CPD, we define the color pixel signal-to-noise ratio (cpSNR) as the ratio of the mean CPD at a point along a WFSC to the standard deviation of the CPD when a sequence of frames are acquired with all conditions held constant. In the following simulations, we used the error in fitting Eqn. 3-2 to the 20 experimental flow-phantom WFSCs from [2, 4] to compute a reference cpSNR. The mean value of cpSNR for the 20 experimental WFSCs was designated (cpSNR) phantom.

Synthetic WFSCs were produced using the minimum sufficient number of samples identified in the previous simulations. Synthetic color pixel noise was added to each sample as a zero-mean Gaussian random variable with σ = m (CPD(vc) ⁄ (cpSNR) phantom) for integer m ≥1. For each value of m, the relative vascular quantification error

was computed as the mean difference between the CPD at the cut-off velocity selected by our automated WFSC method [2] and the CPD at the optimum cut-off velocity defined by the cost function divided by the CPD at the optimum cut-off. Simulations were repeated for increasing m to search for the threshold in cpSNR = (cpSNR) phantom ⁄ m at which the

mean vascular quantification error exceeds 5%. The 5% error goal was chosen to maintain the performance of the automated WFSC method [2] within no worse than half of the 10% error achieved by the original WFSC method [4].

3.2.4.3

Comparison of methods for selecting the operating cut-off

velocity

We computed and compared the relative vascular quantification errors achieved when using four different methods for selecting the operating cut-off velocity. The first method, referred to as closest, selects the sampled cut-off velocity closest to the optimum cut-off velocity defined by the cost function. The second selection method, which represents an idealized version of our automated WFSC method [2], was a perfect binary

decision (i.e., whichever of the middle or end points of the characteristic interval is closest to the optimum cut-off). The vascular quantification errors using these two methods were compared using the two one-sided test (TOST) procedure presented by Schuirmann [14]. The TOST is the most basic form of equivalence testing used to establish that the means of two data sets differ by less than a user selected tolerance. In a TOST, two one-sided t-tests are applied on user-defined lower (negative) and upper (positive) bounds on the difference in means. The two tests yield two p-values of which the greatest is taken as the p-value of the equivalence test. In this chapter, the user- defined “equivalence interval” [14] was set to [-2.5, 2.5] to allow for a variation equal to the 5% error goal.

In some simulated cases, we observed that the optimum cut-off velocity was located towards the beginning (i.e., the left end) of the characteristic interval. Therefore, the third selection method considered was a perfect ternary decision that chooses whichever of the beginning, middle, or end points of the characteristic interval is closest to the optimum cut-off velocity. The vascular quantification error of a perfect ternary decision was compared to the vascular quantification error of a perfect binary decision using the TOST procedure.

The vascular quantification error using the fourth selection method, our previously published automatic algorithm [2], was computed and compared to the error using the perfect binary decision method using a TOST procedure. In addition, the correspondence between cut-off velocities selected using the automatic and perfect binary methods was evaluated by determining the number of cases in which both methods selected the middle of the characteristic interval, the number of cases in which both

methods selected the end of characteristic interval, and the number of cases in which the two methods selected different operating points. Finally, to evaluate the accuracy of the automatic algorithm at different vascular volume fractions, the vascular quantification error was plotted as a function of the CPD value at the optimum cut-off velocity.

3.2.4.4

Properties of reliable characteristic intervals

The previous WFSC simulations reported in [4] showed that the characteristic interval length and the minimum cut-off velocity along the interval are correlated with the accuracy of the CPD as an estimate of the vascular volume fraction. It was concluded in [4] that selection curves with intervals shorter than 2.0 mm/s or with intervals that begin at high cut-off velocities (> 2.0 mm/s) provide unreliable estimates of vascular volume fractions. Therefore, the original WFSC method was not recommended to be used if these conditions are not satisfied. To revisit these recommendations for the improved WFSC method presented in [2], a similar analysis was performed in which the vascular quantification error for all 2080 simulated WFSCs was plotted as a function of the interval length and interval minimum cut-off velocity. The vascular quantification error was computed as the difference between the CPD at the optimum cut-off velocity and the automatically selected cut-off velocity. Using these curves, we defined the minimum detectable interval length, the minimum interval length for reliable quantification (the interval length at which the quantification error is below the 5% target error), and the threshold for the starting cut-off velocity of an interval. The analysis was performed for theoretical WFSCs with no color pixel noise and synthetic WFSCs with the minimum acceptable cpSNR determined in Section. 3.2.4.2.