Section II Material and Methods
6. Design of a Tool for Data Acquisition and Analysis
The development of reliable software tools to enhance early diagnosis, especially in home environments, is highly desirable particularly for the neurobehavioral assessment of the newborn. Such tools should provide objective measures to complement clinicians’ qualitative analysis that is based on subjective skills. Acoustical analysis of infant’s cry and automatic methods for movements analysis may provide objective parameters indicative of neurological pathologies.
The software AVIR described in Chapter 3 was structured to become a module of a larger system named Audio-Video Infant Monitoring (AVIM) whose architecture is shown in figure 6.1
During the PhD period, the analysis of such integrated system was developed reported in a publication recently submitted to a journal.
AVIM is conceived for managing data of a large set of patients. Its most innovative aspect is the ability of merging in a single tool the management of medical records and reports, audio/video data acquisition, handling and analysis, editing and filling out customized tests. Moreover, AVIM supports the clinical perceptual evaluation of cry and spontaneous movements of newborns and infants. The user can manage through a single system all the steps, from the recording to clinical data management and the results of analysis with a significant simplification of the whole procedure.
Figure 6.1 AVIM system structure. The system consists of three main sections: “Patient Data Management”, “Recorder Tool” and “Analysis Tool”.
Besides becoming a computer-aided diagnosis tool, AVIM was developed with the aim of guiding the clinician throughout the process of contact-less assessment and early diagnosis of neurological disorders through a user-friendly environment. Personal and clinical data, markers and notes, as well as acoustical and kinematical parameters of clinical interest are provided and stored with a minimum effort required to the operator. To highlight its capabilities AVIM is applied to the management of data of a newborn recorded five
Patient Data Management
Recorder Tool (AVIR)
Analysis Tool
AVIM DB AVIM
92 times from 10 days after birth until the 24th week of age according to the protocol described in Chapter 3. Results are reported in Sect. III devoted to Experimental Results.
The AVIM system is a flexible monitoring system equipped with webcam and microphone that allows setting up specific tests and adding notes during the recording. It is structured so as to eventually make use of modules devoted to the automatic analysis of audio and video recordings obtained according to the protocol described in Chapter 4 and 5.
Thanks to its user-friendly structure and the simple devices used, the system is well suited for performing audio and video recordings both in the hospital and at patient's home. To this aim, AVIM is equipped with two different access levels: a restricted one for nurses or paramedics for data acquisition and an unrestricted one for experienced clinicians to manage audio and video recordings and analysis. It can be arranged for transferring data to a server provided with a centralized database.
The system is developed in C# language using the OpenCv image processing library for video acquisition and recording. The first release of AVIM is designed to run under Windows OS (Windows 7 and Windows 8).
The four AVIM tools are described in the following sections.
Patient Data Management
An integrated system to manage the patient clinical storyboard and share data and medical history through a devoted interface. It allows importing and exporting signals and images from external sources: clinicians can automatically import data from already existing medical records depending on the system used in the hospital. The user can enter patient’s data for a specific analysis and then proceed using the tool devoted to audio/video recording. The storage of patient’s data is managed through a centralized database structured in order to guarantee privacy and personal data protection.
Moreover this tool allows loading patient’s data and patient analysis and tests. The user can eventually view the patient’s report and carry out the diagnosis through the storyboard and the medical data.
Recorder Tool
The user interface includes a main window for selecting the recording devices and an acquisition window. The system allows recording from up to two microphones and two webcams at the same time. The acquisition window is equipped with single- or double-window to display video streams. The use of both pieces of equipment and windows may be useful to display both the subject and the surrounding environment (i.e. to assess mother-child interaction, or analyse the subject from two different viewpoints; see Figure 6.2). Dual microphones play a similar task, allowing to record both infant cry and e.g. the mother's voice. This module is based on the AVIR recording tool described in Chapter 3.
93 Figure 6.2. AVIM Recorder tool: acquisition window with a double view of the newborn. The user can select markers to point out the status of the patient (such as sleep, restlessness, etc.), ambient noise, interfering events and enter notes
during the recording. Analysis Tool
This tool allows playing/cutting/copying/assessing sequences of interest using markers placed during the recordings process. The tool also allows entering clinical scores, notes and reference markers (e.g. point out a relevant event) on single crying or motion sequences without the need to resort to the use of other software or to pen and paper. Audio and video editing is managed through specific interfaces. Within each interface the selection and extraction of relevant signal segments is obtained positioning a cursor at their starting and ending points: the software automatically merges into a single file all the selected segments one after the other and saves it pressing the button “Extract”. This option proved to be particularly useful in the assessment of GMs from video recordings. Figure 6.3 shows the interface for video recordings and figure 6.4 the corresponding one for audio recordings.
Figure 6.6 shows an example concerning a customized test for GMs set up through button “Test Maker”. The clinical tests (fig. 6.6) for the infant GMs assessment were set up by Stella Maris Institute (Pisa, Italy). Data sheets (active Excel sheets) are provided that allow saving the results of the test as well as personal and clinical data: name, birth, recording time, postmenstrual age, birth weight, recording date, age. All sheets allow the automatic computation of the global score.
The “Start Tracking” button allows the analysis of movements and cry which will be described in a next section.
94 Figure 6.3. The AVIM Analyser tool: interface for video analysis.
Figure 6.4. The AVIM Analyzer tool: interface for audio (newborn cry) analysis.
95 Figure 6.6. Customized test. AVIM allows recalling a specific test for the analysis of GMs
Audio Analysis
AVIM allows adding modules for the analysis of crying (NeCA) to extract the following acoustic parameters (and related statistics): fundamental frequency (F0), first two resonance frequencies of the vocal tract (F1 and F2), curtosis, skewness and time duration of each cry unit (CU). Details about NeCA are reported in Sect.IV.
If the analysis module is available, the acoustical analysis can be performed for the 5 recording
sessions defined in the protocol.
Video Analysis
This tool provides a support to the semi-automatic analysis of movements from video clips. It implements the method described in Chapter 5 to extract movements’ features such as speed and acceleration. The tracking of movements is launched with the "Start Tracking" button (figure 6).
All results obtained by the infant crying and GMs analysis are saved as a patient report. Thus the clinician can easily get the storyboard for the neuro-behavioural assessment of the newborn. Specifically, the clinician gets a report with a clinical evaluation for each session as shown in figure 6.6.
96