Results Discussion

We present our findings in Tables3.5and3.6. As expected, training times for Logistic Regression and Naïve Bayes far exceed kNN, which requires no dedi-cated training procedure. Instead, during testing, kNN uses an exhaustive search through every member of the training set to find its k closest neighbours. As such, we expected kNN to yield the reverse effect, i.e. becoming dramatically slower in testing. This is compared to NB where, once priors and conditionals are computed, computing the class is a series of joint probability multiplications, which performs in near-constant time. LR simply computes the logit function for a given feature vector and the parameters acquired by minimising the least-squares cost function (see Section3.4.2). In general, LR was the costliest to train, while kNN was the costliest to test. These trends were consistent across all days worth of data.

The results of training and testing drop slightly between 14 and 21 days, which we found was due to the variance in device usage between users. While the dataset is sampled at 5-minute intervals, this does not account for periods when the device is disabled. Thusly, Figure 3.4 depicts the mean number of samples captured per day for each participant. Only three users had data lasting 21-days – users 1, 2 and 5 – constituting 198 samples per day per user on average,

FIGURE3.4: Mean samples per day per participant in the GCU dataset [135].

FIGURE3.5: TEE versus non-TEE overhead averaged across train-ing and testtrain-ing. (a) traintrain-ing and (b) testtrain-ing overhead of OP-TEE versus untrusted world; and relative SGX speed-up of (c) training

and (d) testing versus non-SGX implementation.

while, for 14 days, this rises to 245 samples per day. The differences in sample

‘density’, due to the usage variance between users, is the primary driver behind the cause for similar timings between 14-day and 21-day training and testing.

This lower training times of 21-days worth of data is hence biased heavily by the unusual sparsity of User 1. Evidently, device usage is a significant influence in determining CA training and testing times.

The comparative performance of each platform is illustrated in Figure3.5.

As expected, OP-TEE yields a small overhead during testing and training. In training, this yields a small overhead of approximately 1.5–4.25% depending on method. In absolute terms (Table3.5), this ranges between a 12ms overhead (one-day training average with NB) to 404ms (14-days training average with LR).

While certainly noticeable in isolation, a maximum overhead of 404ms is unlikely to be burdensome, as training/enrollment is either a one-off or occasional task.

Occasional re-training may occur, for example, once every week or month to refresh a model that no longer reflects the user’s current behaviour, say, due to natural changes in the user’s (permanent) location, e.g. residency, or application usage (after installing new applications or deleting unused ones). During testing, it seems that OP-TEE yields a high relative overhead for NB and LR of approxi-mately 110% versus 10% for kNN. However, this relative overhead is somewhat misleading as the absolute kNN testing time is the highest of all (at most 12,154µs for kNN versus 528.7µs for LR with 21-days of training data). While kNN has a substantially lower training time, its exhaustive search during testing yields a 20-24 times overhead relative to the absolute testing times NB and LR. One reason for the high relative TEE overhead of NB and LR is the greater impact that context switching plays relative to the actual testing time.

To our knowledge, no literature exists regarding the precise performance overhead incurred by OP-TEE and ARM TrustZone versus a suite of untrusted applications. Benchmarking the latency incurred by SMC calls and OP-TEE OS internals in isolation is out-of-scope in this paper. However, we believe that such an investigation would be worthwhile for the benefit of TEE applications in general, e.g. DRM and payment tokenisation, as a separate research contribution.

Most surprising is the substantial difference between SGX and non-SGX performance. At first, we believed SGX would incur some overhead, with the additional memory enforcement, e.g. encryption of enclave pages between the CPU and DRAM, and context switching. Yet, it exceeds the REE by approximately 60-100 times for training and 4–11 times for testing, depending on the algorithm.

Upon investigation, while SGX and non-SGX implementations were identical, enclave applications are linked with Intel’s own C (sgx_tstdc.lib) and C++

standard libraries (sgx_tstdcxx.lib), which must be compulsorily linked to any SGX enclave application. These ‘trusted’ libraries are used to provide

functions that can only be used within enclaves. At present, enclaves do not support I/O operations or functions that rely upon OS system calls. Indeed, attempting to link non-Intel C/C++ standard libraries will “either fail the enclave signing process or cause a runtime failure due to the use of restricted instructions” [153].

Intel’s trusted libraries provide an optimised subset of compatible functions based on Intel’s Integrated Performance Primitives (IPP) [154]. This suite offers parallelised, multi-threaded optimisations of standard routines based on the most recent AVX and SSE instruction sets, which “significantly increases performance”

over unoptimised implementations [154]. Comparatively, the non-SGX portion uses the Microsoft Visual C++ standard library with no such enhancements.

Interestingly, training times benefit most, due likely to the computationally intensive methods involved, namely computing prior and conditional proba-bilities for Naïve Bayes, and linear optimisation for LR. Further investigations revealed that IPP does favour such arithmetical¹⁴and string-based¹⁵operations strongly. Unfortunately, the fact that SGX’s C/C++ standard libraries must be compulsorily linked is an unwelcome distortion in gauging the overhead of SGX versus non-TEE performance directly.

What we can conclude, however, is that the observed performance of SGX raises security concerns in itself: there is a clear incentive for developers to move non-critical components to enclaves if some perceived security benefits can be gained with little performance overhead. This also applies to OP-TEE as the timing overhead, in real terms, is not dramatically greater than the untrusted world. Yet, recklessly shifting code to a TEE can dangerously increase the TCB of the trusted application, which may expose security-critical components to logical flaws or programming bugs from non-security related code sections. The essence of a secure application in any TEE is to minimise its TCB to that which is functionally necessary, and any incentive to increase the TCB for performance gains may undermine the application’s overall security in reality. A second observation is that, for both TEEs, algorithm choice has a far more significant effect on training and testing times than whether to use a TEE.