KinecTracker (KiT) Application

The main requirements that were defined for the KiT application are the following:

1.1. Online visualization of data provided by a Kinect sensor, including the possibility of viewing two or more data types simultaneously;

1.2. Acquisition of one or more data types provided by the Kinect, in a synchronous way; 1.3. Possibility of saving a “label” associated with a given instant, during an acquisition; 1.4. Management of different acquisition sessions;

Subject Portable computer KiMA KiT Preview and acquire data Review and manage data Motion analysis RGB-D camera

(Kinect v1 or v2) _{Color and/or} infrared, depth

and 3-D body joint data

1.5. Possibility of discarding the oldest data of the current acquisition, according to the available disk space or a given time duration indicated by the user;

1.6. Selection of the sensor tilt angle (Kv1 only).

Most of these requirements correspond the use cases illustrated in Figure 4.2. In a scenario where the data acquisition takes place at a clinical environment, the clinician (e.g., physician, nurse) is responsible for choosing the appropriate preferences, managing the acquisition sessions and car- rying out the data acquisitions. The data are provided by a single Kinect camera, which is connected to the computer where KiT is running through a USB cable. The subject being assessed is involved in the data acquisitions by being in the field of view of the camera. In other scenarios, such as am- bulatory monitoring, the patient itself, a relative or a caregiver performs the role of the clinician.

Figure 4.2. Use case diagram associated with the KiT application.

The graphical user interface (GUI) of the main window of KiT for Kv2 is shown in Figure 4.3 (KiT for Kv1 is similar). There are two visualization areas corresponding to the primary and

4.1 KinecTracker (KiT) Application

secondary data sources. For each visualization area, it is possible to choose one of the available data types. For colour, IR, depth and body index data, it is also possible to have the body joint data drawn over the image (colour + body, depth + body, etc.). Therefore, KiT allows viewing up to three data types simultaneously (requirement 1.1).

Figure 4.3. KiT’s main window GUI (Kv2), including the display of depth, body index and body data.

Some preferences regarding data acquisition or the Kinect itself can be indicated in the “Pref- erences” window, shown in Figure 4.4 for Kv1 and Kv2, which is opened when choosing the Prefer- ences option in the Edit menu. The available options depend on the used Kinect version. For both versions, it is possible to choose the folder where the different sessions/acquisitions will be stored and define a storage buffer. The latter is defined by indicating the percentage of the disk space that can be used, or the acquisition duration in minutes/hours/days, until the oldest data starts being erased (requirement 1.5). This functionality is useful when the total duration of an acquisition is unknown (e.g., continuous monitoring of epileptic patients, where the relevant movements correspond to un- predictable sporadic seizures).

In Preferences, it is also possible to choose any combination of data types to be saved (re- quirement 1.2). In the case of the Kv1, choosing not to save the body joint data (skeleton) corresponds to disabling joint tracking. Moreover, the user can select the resolution and frame rate of the colour

and depth data, choose to track only the upper body joints and define the camera tilt angle (last option corresponds to requirement 1.6). For the Kv2, the IR image brightness can be adjusted (for online preview only).

(a)

(b)

Figure 4.4. Preferences window of KiT for Kinect v1 (a) and v2 (b).

In the Options panel of the main window (below the Secondary Source drop down menu), the user can manage different acquisition sessions (requirement 1.4), facilitating the organization of the acquisitions according to the different subjects or days. The current session can be indicated by creating a new session or by selecting an existing session. These options are also presented to the

4.1 KinecTracker (KiT) Application

In the Controls panel of the main window (below the Options panel), there are three options related to the acquisition of data. The first one (“Background” button) allows the acquisition of a single frame, which can be useful if we wish to carry out motion analysis by using a background subtraction technique over the colour data. The “Acquire” button starts a data acquisition (require- ment 1.2). During the acquisition, the name button’s name changes to “Stop” and can be used to stop the current acquisition.

The third option in the (“Label” button) is included in the Actions panel within the Controls panel, and is only available during an acquisition. When selected, it creates a label for the current instant (requirement 1.3), which consists of saving the timestamp and frame number associated with that instant. The user can indicate a name and an optional description through a dialog window. This option can be useful to indicate the beginning of a relevant movement (e.g., epileptic seizure), so it can be easily found when reviewing the acquisition in KiMA.

Some additional features are available in the menu and/or tool bar: reset image display; flip image horizontally and vertically; show colour/IR data in full screen; open the last acquisition in KiMA; and quit application. Furthermore, the status bar presents information regarding subject track- ing, application status, and current frame rate for each data type.

KiT was implemented based on an existing application developed for Kv1, by performing the following main improvements:

 Elimination of unnecessary data frame processing for visualization/acquisition (i.e., only the data types chosen for visualization and/or acquisition are processed);

 Possibility of visualizing IR/depth and body data simultaneously (only body over colour was displayed in the original application);

 Possibility of indicating the data types to be acquired, which in the case of the body joint data also corresponds to enabling/disabling subject tracking (Kv1 only);

 Management of acquisition sessions through the application’s GUI;

 Acquisition of data in a synchronized way (using an event provided by the Kinect SDK);  Acquisition of body joint data corresponding to one of the detected subjects as default (previously, body joint data were only acquired if a specific subject detected by Kinect was chosen by the user, which led to data loss if the subject left the camera’s depth range, since the ID attributed to the subject by the Kinect changes when this happens);

 Use of a queue for storing the frames while they are not written to file (this prevents data frames loss in computers with less processing power and facilitates the creation of differ- ent files every minute);

 Increase of flexibility in the definition of the storage buffer, by adding the possibility of indicating any size (within the valid values) and choosing from different types (percent- age of used disk, minutes, hours or days);

 Modification of file format to facilitate parsing during video review and motion analysis, and addition of a header with relevant information (e.g., resolution, tilt angle, application version);

 Addition of an option for saving a label, which replaced the option for saving an event corresponding to the last 3 minutes of data;

 Possibility of changing the position of the overlay text (date, time, session folder path) vertically, and flipping the images both vertically and horizontally;

 Option for opening the last acquisition (if any) in KiMA.

Regarding the Kinect data, the SDK (v1.8 [213] and v2.0 [214] for Kv1 and Kv2, respec- tively, in C#) provides the data frames as they become available. In KiT, for each frame and data type, the data itself and associated information (e.g., image size in the case of image frames) are extracted and then stored together with the associated timestamp (date and time when it became available).

For the colour, IR, depth and body index, the data corresponds to image pixel information. For the body, the data includes the 3-D position of the tracked joints (i.e., body), as well as the asso- ciated mapped colour and depth points. The latter information is used for displaying the body joints over the colour and depth images, respectively. The mapping is performed by relying on the “Coor- dinateMapper” class provided by the Kinect SDK [176].

If a given data type is to be displayed, the corresponding stored data are firstly processed and then the corresponding image/body is drawn in the GUI. When the user starts an acquisition, a thread is started for each data type and the data of each new data frame are inserted in a FIFO (first-in-first- out) queue (one queue per data type). The thread then gets the frames from the queue and saves the associated data to file. The same thread is responsible for creating a new file each minute.

The format of the files used to store each data type is presented in Appendix A.1. The header of each file includes the tilt angle and height of the Kinect. These values are computed based on the clipping floor plane detected by the Kinect itself [213-215]. For each body joint data frame, the