Design of the testing protocol

The testing protocol must define the activities and the conditions under which the evalu- ation is carried out. It must match the requirements mentioned above and fulfill the fol- lowing criteria:

1. Standardization

Refers to which activities are defined to be performed, how accurately, and their duration.

2. Repeatability

Refers to the reproducibility of the target use-case scenario and the defined ac- tivities and conditions.

3. Representativeness

Refers to definition of usage conditions and possible impact of external factors.

Controlled or non-controlled protocol execution

Many of the trials used controlled protocols with continuous supervision, typically speci- fied indoor gym or lab activities. Although controlled conditions are more standardized, repeatable and well-annotated, the external validity of controlled protocols is more diffi- cult to generalize for common usage conditions and typical user behavior, e.g. supervi- sion of the test subjects during execution of the test reduces the number of uncontrolled hand movements in order to decrease the level of possible artifacts and measurement errors. Non-controlled protocols, typically everyday life, sleep or outdoor sports activities, without continuous supervision and without any detailed specification of the activities is closer to the actual target use-case conditions. However, although the external validity of such protocols is higher as they are more representative of the target use-case con- ditions, they are weaker in terms of repeatability and standardization. Any interpretation of the results of non-controlled protocols relies on user-subjective annotations of possible unexpected events during the test.

External environment factors that can affect the signal quality and the accuracy of the HR estimation, such as the outdoor temperature or ambient light interference, also need to be specified and controlled as closely as possible.

Testing scenario specification

The activities in the testing protocol must directly reflect, or at least closely simulate, real usage conditions. A typical testing protocol, which describes a protocol derived from

64

sport physiology testing guidelines (Winter et al. 2006; Nieman 2011), is presented in Table 6. The main advantages of indoor testing are the fact that it is relatively simple to standardize the test conditions (e.g. temperature and ambient light), the researcher has full control of how the test is executed, and the tests can easily be repeated. However, such protocols control several factors which cannot be controlled during typical real-user scenarios (e.g. temperature variations and related variations in skin blood perfusion, var- ying ambient light level, clothing, activity pattern such as step rate and variations in style when exercising, e.g. on a treadmill or an outdoor path, etc.). Although outdoor testing better represents real user conditions (outdoor jogging, cross-country skiing, inline-skat- ing) many external factors may not be controlled and these may vary widely, so such evaluation studies require much larger samples and it is more challenging to define the practical limitations of the test conditions. Therefore, such evaluations are scarcer. Daily activities represent long-term scenarios aimed at revealing any potential inaccuracy in the device being tested during random unspecified wrist movements (e.g. washing dishes or typing on a keyboard). These scenarios can be performed within defined pro- tocols in specialized laboratories; unspecified hand movements can be simulated with a Rubik’s cube game, for example. Motionless (e.g. sleep or awake rest) protocols are used for testing beat-to-beat detection accuracy and the long-term resting HR level with an unobtrusively-worn test device.

It is important to include two special activities at the beginning of the protocol: a warm-

up activity and a synchronization activity. The former aims to standardize the subject’s

blood perfusion and body temperature, while the latter (e.g. several squats) will help synchronize signals between the tested and reference devices during any subsequent signal processing.

The activities in the protocol should be performed in a predefined order to ensure that the measurements are standardized and repeatabable. Some activities, typically exer- cises such as walking or ergo-cycling, might improve blood perfusion, so subsequent test activities must take account of this in order to achieve more accurate HR estimation. The results estimated according to the example protocol proposed in Table 6 might be biased towards a lower measurement error. This is caused by the correlation of human HR and running speed while systematically increasing the running speed in the protocol. Moreover, no breaks between running speeds are included; these are typical for interval training which represents the most difficult testing scenario.

65

Table 6: Example of testing protocol for indoor testing scenario

Activity Duration

Synchronization – 5 squats and rest 1 min

Warm-up exercise – 6 km∙h-1 _{running 5 min}

Still standing 1 min

Walking on a treadmill at 3 km∙h-1_{, 0% inclination 3 min}

Walking on a treadmill at 3 km∙h-1_{, 10% inclination 3 min}

Walking on a treadmill at 5 km∙h-1_{, 0% inclination 3 min}

Walking on a treadmill at 5 km∙h-1_{, 10% inclination 3 min}

Running on a treadmill at 9 km∙h-1_{, 0% inclination 3 min}

Running on a treadmill at 11 km∙h-1_{, 0% inclination 3 min}

Rest sitting 4 min

Cycling 60 rpm, 75 Watts resistance 3 min

Cycling 90 rpm, 75 Watts resistance 3 min

Rest sitting 4 min

The order in which the activities are performed in the protocol may affect the results of the evaluation. This variation is hard to control unless the order of the activities is ran- domized. However, randomizing the order of the activities can cause problems for the practical execution of the evaluation campaign. The order of the activities in the protocol needs to be taken into account in order to reduce the output error, but this can system- atically and deliberately bias the results. For example, activities designed for testing the sensitivity of hand movements’ artifacts performed after a warm-up exercise might in- crease accuracy of the HR estimation, but decrease the external validity of the protocol.