Wavelet Decomposition

fTCD data were analyzed using 5-level wavelet decomposition that utilized Daubachies 4 mother wavelet. Such analysis was performed since it was used before with fTCD data and it yielded the most efficient fTCD-based BCI in literature [137] as explained in chapter 3.0. To reduce the dimensionality of the fTCD feature vector, 4 features were computed for each wavelet band instead of using all wavelet coefficients as features. The 4 features included mean, variance, skewness, and kurtosis. Feature vector corresponding to each trial included 24 features for each channel and 48 features in total.

6.1.4 Feature Reduction and Classification

The Wilcoxon signed rank test [113] was used to select the significant features from fTCD feature vectors. p-values of 0.001, 0.005, 0.01, and 0.05 were used. As for EEG, the feature vector of each trial contained 2 features obtained by projecting the trial data using = 1, 2, 3, …., and 8 eigenvectors from both ends of W. To assess the performance of single-modal BCIs (EEG only and fTCD only BCIs), selected features from each modality were classified solely using the SVM classifier. In particular, the performance of fTCD only system was evaluated at p-values of 0.001, 0.005, 0.01, and 0.05. Also, the performance of EEG only system was evaluated using 2 (2, 4, 6, …., and 16) CSP features. The best set of performance measures for each modality were reported in the results section below.

To evaluate the performance of the hybrid system, the EEG feature vector of each trial containing features was projected into one scalar SVM score (EEG evidence). Moreover, the selected features from the fTCD feature vector were also projected into one scalar SVM score

(fTCD evidence). In particular, the EEG and fTCD feature vectors corresponding to training trials were used to learn 2 SVM classifiers separately. The 2 classifiers were used to obtain 2 SVM scalar scores representing projected EEG and fTCD feature vectors of each trial under test. EEG and fTCD SVM scores were evaluated at p-values of 0.001, 0.005, 0.01, and 0.05 and 2 (2, 4, 6, …., and 16) CSP features. The best set of performance measures were reported in the results section.

6.1.5 Bayesian Fusion and Decision Making

We designed a Bayesian framework to fuse EEG and fTCD evidences (SVM scores described in Section 6.1.4) under 3 different assumptions including joint EEG-fTCD distributions ( 1), independent EEG and fTCD distributions ( 2) as well as weighted independent EEG and fTCD distributions ( 3). For each binary selection problem, EEG and fTCD evidences corresponding to trials of that problem were partitioned randomly into training and testing sets using 10-fold cross validation scheme. Assume is the number of trials presented to a certain BCI user. Given a set of EEG and fTCD evidences Y = {y , … y } where y = {e , f }, e and f are EEG and fTCD evidences respectively. As shown in Fig. 24, for a test trial , inference of the unknown user intent x will be achieved through state estimation using the EEG and fTCD evidences jointly. Here, since we solve the binary classification problem through state estimation, we assume that takes only two distinct values for each binary classification problem. Fig. 24 (b) assumes independence between the EEG and fTCD evidences conditioned on the unknown

x = arg max p(x |Y = y ) (10)

In (10), p(x |Y) is the posterior distribution of the state x conditioned on the observations Y. From Bayes rule and noting that the conditional distribution of , ( ) does not depend on the state variable , (10) is equivalent to:

x = arg max p(Y = y |x ) p(x ) (11)

where p(Y|x ) is the state conditional distribution of the evidences Y and p(x ) is the prior distribution defined over the values of state. Since the number of trials belonging to each class is randomized in our study, we assume that p(x ) is uniformly distributed. Accordingly, we rewrite (11) as:

x = arg max p(Y = y |x ) (12)

The distribution p(Y|x ) is computed from the training data. In particular, p(Y|x = c ) and

p(Y|x = c ) are state conditional distributions of the evidences Y belonging to class c (x = c )

and class c (x = c ) respectively. The conditional distribution representing each class were estimated using kernel density estimation with a Gaussian kernel. Silverman's rule of thumb [138] was used to calculate the kernel bandwidth. User intent at trial is inferred by solving eqn. (12) at the evidences Y = y . In this study, the distribution p(Y|x ) was evaluated under 3 different assumptions.

6.1.5.1Assumption 1: Joint Distribution

Since the k evidences are y = {e , f }, (11) can be written as:

where p(e, f|x ) is the state conditional joint distribution of e and f. This is represented graphically in Fig. 24 (a). To compute p(e, f|x ) for N-10 training trials, the joint distribution of the EEG and fTCD scores was assumed to follow a multivariate Gaussian distribution. Kernel density estimation with a Gaussian kernel was employed to compute the distributions p( , f|x ), = 1, 2. For evidences under test y = {e , f }, e and f are plugged in (13) and the user intent x that yields the maximum likelihood is selected.

6.1.5.2Assumption 2: Independent Distributions

In order to compute p(Y|x ), we will assume that conditioned on the latent state, the EEG and fTCD evidences are independent from each other as seen in Fig. 24 (b), then accordingly also considering the uniform prior over the states, we rewrite (13) as:

x = arg, max p( = e |x )p(f = f |x ) (14)

a) b)

Figure 24 Probabilistic graphical model illustrating the state and measurement relationships assuming EEG and fTCD evidences are jointly distributed (a) and independent (b).

SVM EEG and fTCD scores of the N-10 training trials are used separately to compute 2 distributions [(e|x ) and p(f|x )] using kernel density estimation with Gaussian kernel. For evidences under test y = {e , f }, e and f are plugged in (14) and the user intent x that yields the maximum likelihood is selected.

6.1.5.3Assumption 3: Weighted Independent Distributions

Since the contribution of EEG and fTCD evidences towards making a correct decision might be unequal, we suggest weighting the distributions p(e|x ) and p(f|x ), and thus rewriting (13) as:

x = arg, max p( = e |x ) p(f = f |x ) (15)

where and 1 represent the optimum contribution of the EEG and fTCD modalities, respectively. ranges from 0 to 1. p(e|x ) and p(f|x ) were computed as mentioned in section 6.1.5.2. To simplify (15), natural logarithm was employed. In particular, since natural logarithm is a monotonically increasing function, maximization of the right-hand side of (15) is equivalent to maximization of the natural logarithm of this right-hand side. Therefore, (15) is equivalent to the convex combination of the log likelihoods given by (16).

x = arg, max [ ln p( = e |x ) + (1 ) ln p(f = f |x )] (16)

The training of our hybrid system requires the identification of the optimum task period for each individual. For assumption 3, unlike 1 and 2, we performed 2D optimization since we seek optimizing both the task period and the

search over a