DESIGN OF A WEB GRAPHIC USER INTERFACE FOR REAL-TIME EEG INTERPRETATION

(1)

DESIGN OF A WEB GRAPHIC USER INTERFACE FOR REAL-TIME EEG INTERPRETATION

A Degree Thesis

Submitted to the Faculty of

Escola Tècnica d'Enginyeria de Telecomunicació de Barcelona

Universitat Politècnica de Catalunya by

Víctor Peña Romero

In partial fulfilment

of the requirements for the degree in

TELECOMMUNICATIONS TECHNOLOGIES AND

SERVICES ENGINEERING

(2)

Abstract

This project proposes a method for easier interpretation of Electroencephalogram (EEG) data acquired from newborn babies so that it can be understood by a wider range of medical professionals. The equipment currently used requires highly qualified experts in order to detect possible seizures in newborns.

The project is divided into two blocks. The main block is the implementation of a system that enables the communication between the device recording the EEG data from the baby and a server. The second block consists of the design of a front-end user web interface that plots a graph of the EEG data.

The communication part is based on an Arduino board. The board processes the acquired EEG data and sends it via a Wi-Fi connection to a server creating the backbone of the system. When the information is received, it is sent to the client side for real-time plotting.

(3)

Resum

Aquest projecte proposa una tècnica per facilitar la interpretació de les dades de electroencefalograma (EEG) adquirides de nounats perquè puguin ser interpretades per un grup de professionals mèdics més ampli. L’equip utilitzat avui dia requereix experts altament qualificats per a detectar possibles convulsions en nadons.

El projecte es divideix en dos blocs. El principal és la implementació del sistema que permet que el protocol de comunicació enviï les dades EEG adquirides del nadó a un servidor. El segon bloc consisteix en el disseny de la interfície web d'usuari que traça el gràfic amb les dades EEG.

La part de comunicació es basa en una placa Arduino. La placa processa les dades EEG ja adquirides i les envia a través d'una connexió Wi-Fi a un servidor creant la base del nostre sistema. Quan la informació és rebuda, s'envia a la part del client per a la representació del gràfic en temps real.

(4)

Resumen

Este proyecto propone un método para facilitar la interpretación de los datos de electroencefalogramas (EEG) adquiridos de recién nacidos, para que puedan ser interpretados por un grupo de profesionales médicos más amplio. El equipo utilizado hoy en día requiere expertos altamente calificados para detectar posibles convulsiones en bebés.

El proyecto se divide en dos bloques. El principal, es la implementación del sistema que permite que el protocolo de comunicación envíe los datos EEG adquiridos del bebé a un servidor. El segundo bloque consiste en el diseño de la interfaz web de usuario que traza el gráfico con los datos EEG.

La parte de comunicación se basa en una placa Arduino. La placa procesa los datos EEG ya adquiridos y los envía a través de una conexión Wi-Fi a un servidor creando la base del sistema. Cuando la información es recibida, se envía a la parte del cliente para la representación del gráfico en tiempo real.

(5)

Acknowledgements

I would like to thank Dr. Andriy Temko and Dr. Climent Nadeu for giving me the opportunity to work in INFANT Research Center and EmbeddedSystems@UCC, University College Cork. I would also like to thank Dr. Emanuel Popovici for his supervision during the project.

I wish to express my most sincere thanks to Sergi Gomez, PhD candidate of the Electrical and Electronic Engineering Department in UCC for the unconditional help and support throughout the project and my stay in Ireland. I would also like to thank Mark O’Sullivan for his patience and help with the project.

Last but not least, I would also like to thank all my family, especially Raúl Escalante for all their support during my stay abroad.

(6)

Revision history and approval record

Revision Date Purpose

0 29/11/2019 Document creation

1 20/01/2020 Document revision

DOCUMENT DISTRIBUTION LIST

Name e-mail

Victor Peña Romero [email protected]

Emanuel Popovici [email protected]

Climent Nadeu Camprubí [email protected]

Written by: Reviewed and approved by:

Date 29/11/2019 Date 20/01/2020

Name Victor Peña Romero Name Emanuel Popovici

Position Project Author Position Project Supervisor

(7)

List of Figures

Figure 1: Scheme of the whole system………...………11

Figure 2: Block diagram………..…………..12

Figure 3: Gantt diagram………...………….13

Figure 4: Duplicated broadcast data………...……14

Figure 5: Neonatal Brain Stethoscope……….……..15

Figure 6: Sparkfun ESP32 Board.Taken from https://www.sparkfun.com/..…...…….……16

Figure 7: UDP Datagram………..17

Figure 8: Comparison between UDP and TCP. Taken from https://www.colocationamerica.com/blog/tcp-ip-vs-udp...18

Figure 9: Client – Server scheme communication. Taken from https://docs.citrix.com/en- us/citrix-sd-wan-wanop/10-2/how-html5 works.html ……….………..19

Figure 10: D3.js logo……….……20

Figure 11: Plotly.js logo……….20

Figure 12: Parsed values loop……….21

Figure 13: WS code connection at the server side……….……….………….22

Figure 14: WS code connection at the client side……….22

Figure 15: Whole file scheme of the project………..23

Figure 16: Time split axis……….……….24

Figure 17: Dropdown menu………..………24

Figure 18: First implementation of two buttons………..25

Figure 19: Performance of the main button……….……….……….25

Figure 20: Android APK version ……….………25

Figure 21: Window resize event………..…………26

Figure 22: Comparison of data received in the server side……….……….27

Figure 23: 1Hz Sine wave as input value..………27

Figure 24: Final design web page……….…………..28

(9)

List of Tables:

Table 1: Parse values HEX to Floats………...21

Table 2: General project budget………..29

Table 3: Components list…..………29

Table 4: Personnel table………...29

(10)

1. Introduction

1.1. Project Background

The real-time EEG interpretation project is a small piece of the whole NeuroBell project.

Neurobell is one of the many projects carried out at the INFANT Research Centre. This project began with the aim of increasing the number of medical professionals capable of diagnosing abnormalities in neonatal brain signals (i.e. EEG).

Accurate interpretation of EEG data is a difficult task. Therefore, a high level of expertise and training is required. Despite the fact that EEG data is the most commonly used data for interpreting neonatal brain function, there is still a lack of experts in the field to interpret the EEG data in a timely manner.

The research team hosted at UCC was challenged with implementing a solution for facilitating EEG monitoring using sonification and deep learning AI via an Android app to provide all the information in a more user-friendly interface so that it would expand the number of medical professionals capable of interpreting EEG data.

The main focus of this project is the android app aspect, which provides visualization of a single-channel of EEG, a seizures detector via traffic-light indication through trained deep learning convolutional neural networks (CNNs) and the data sonification which is useful to detect changes in the EEG morphology in the event of neonatal seizures.

This project will contribute to the research of multi-platform systems to be compatible with a wide range of devices.

Figure 1: Scheme of the whole system

(11)

1.2. Project overview and goals

The project was carried out at Embedded Systems Research Group in the School of Engineering and with the Irish Centre for Fetal and Neonatal Translational Research (INFANT) at University College of Cork (UCC). The purpose of this project is to implement a communication system between an Electroencephalogram (EEG) data acquisition block and a web-client block, which enables user-friendly interpretation of EEG data in task the of neonatal seizure detection.

The whole system overview consists of three blocks as illustrated in Figure 1. The first block is the EEG data acquisition system, where the data is acquired through electrodes and sent it to the Arduino board. This block has already been developed by the research team at UCC. Therefore, this project will use EEG data already acquired and stored in text files. However, it will be sent by simulating the real-time acquisition.

The second and main block consists of the communication protocol to push the EEG data to a server using a NodeJS application. The Arduino board acts as a Wi-Fi transceiver and sends the data to a private access point where the server is connected in order to receive and store the EEG data.

The final block starts with the communication between the server and the client side using the WebSocket protocol so that each client that connects to the private access point would be able to access and send data to the real-time EEG graph.

The project’s main goals are:

a. Implement an Arduino script that processes the EEG data acquired and converts the board into a Wi-Fi transceiver.

b. Implement a server capable of receiving and processes the data.

c. Configure the communication protocol between server and client side.

d. Design the front-end GUI for the HTML web-page.

e. Ensure reliable, real-time plotting of the EEG data.

(12)

1.3. Project requirements and specifications Project requirements:

- A fully functional communication between the acquisition block and client side without losing data packets.

- Build a user web interface similar to the actual android app in order to display the graph.

Project specifications:

- Non-stop EEG data-flow communication to the server when request is done.

- Real-time graph on the client side.

- High-speed data plotting, with a minimum of 256 samples per second.

- Chart window length of up to 20 seconds of EEG data

1.4. Work Plan (Tasks and Gantt diagram)

The work plan has been divided into 4 different work packages as follow in figure 2:

Figure 2: Block diagram

(13)

The final Gantt diagram of the whole project is presented in the Figure 3:

Figure 3: Gantt diagram

(14)

1.5. Incidences and deviations

Initial time expectations were not measured correctly as some parts of the project suffered from some unexpected incidences, which required more time. In the final work plan, the signal processing blocks referred as “Running Algorithms” was removed and proposed as future development.

A similar situation has happened with the initial block of “Scalability” which was supposed to be done once the final webpage was correctly performing. This block was supposed to implement more than one input signal. Unfortunately, during the first few months of development, it was decided that there was not enough time for such an ambitious final product, since the product development time was reduced by one month. Therefore, a decision to remove the less relevant features was taken.

With regard to the Gantt diagram, as the initial time-frame for the project development was 6 months, it was decided to reduce this by one month to allow for time to prepare the final presentation of the project, which was due in the beginning of the 6th month, as seen in Figure 3.

Despite the previous time adjustments, the project has been in constant development until the implementation of the WebSocket protocol, where almost 3 weeks was dedicated to solving the problem where only a constant amount of values (around 300 values) were transferred from the server to the client. After a long period of debugging, the problem was fixed by controlling the generation and closing of new client connections.

After all the incidences were fixed, the system still had a drawback. In the last blocks of the project, in the broadcast connection, when more than one connection is required, the application does not work as expected.

When the first user connects to the webpage, the system is able to plot the data correctly as it is sending the data for the first time. However, when a second user tries to connect to the web page, it will receive the data duplicated affecting the first user’s plotting speed.

The duplicated data received by the second client connection is shown in the Figure 4 with a sine wave as an input signal, which has been zoomed into. In the event of more than one simultaneous user, the data is multiplied by the numbers of new connections.

Figure 4: Duplicated broadcast data

(15)

2. State of the art of the technology used or applied in this thesis:

2.1. Background

Seizures are a disorder that many people have to face daily and can be really challenging, especially in the case of newborn babies. The most common tool to detect this disorder is electroencephalography (EEG), which enables real-time recording of the electrical activity of the brain.

This method usually requires special and expensive equipment, the main elements consist of electrodes placed on the along the scalp and an amplifier box to acquire the data. The most commonly used electrodes require abrasive creams and conductive gels to reduce skin impedance which makes this process quite invasive and time-consuming [1]. Studies have shown that without the usage of EEG interpretation, less than 9% of seizures would be detected [2].

The use of the EEG for newborn babies requires a highly-qualified healthcare experts who are very limited. The lack of resources has prompted research in the area of automated seizure detection [3]. There is a simpler form of EEG which can be interpreted by neonatal intensive care unit staff, called amplitude integrated EEG (aEEG). Although it is simpler to interpret, it has several limitations as there are not still enough experts able to interpret the data.

In recent years the demand for improved brain health monitoring has increased. Therefore, research is being conducted in order to implement more portable equipment that requires less expensive tools and tries to avoid the time-consuming patient preparation by using dry electrodes without a reduction in EEG signal accuracy. [4].

The whole scheme of the system consists of dry electrodes attached along the scalp of the newborn, a low-cost portable EEG acquisition board that process the data and pushes it via Bluetooth Low Energy (BLE) connection to an Android device where a friendly GUI is implemented.

(16)

Previous studies have explored EEG sonification to facilitate EEG interpretation in an easier way, so that non-experts can make use of this feature. It has been demonstrated that the visual sense is less accurate than human hearing in assessing the spatial and temporal evolution of frequency [5].

The algorithm implemented in [6] reflects the higher sensitivity of the human hearing system to the presence of the rhythm. A phase vocoder is developed in order to detect the frequency changes over time in the audio mapping the spectrum by preserving the phase coherence [7].

For long-term usage of raw EEG data, a deep learning convolutional neural network (CNN) was implemented in [8] for the task of seizure detection. The architecture implemented enabled an optimal feature extraction and classification stages in one routine [1]. Different layer depths were tested and compared in [8].

2.2. Data Acquire – ESP Wi-Fi Board

On the communication side, research has been done in order to select the most efficient board to meet the system requirements.

Arduino has a wide variety of microcontrollers that enables connection to internet. Many years ago, Arduino released the first version of a “Wi-Fi Shield”, a basic board that enabled users to plug it onto other modules so they could connect it with the network.

A more powerful board was released by Arduino MKR1000 combining different features from past Arduino boards including the Wi-Fi Shield.

There are several variants of the MKR1000 board in the market and different companies have released different microcontrollers based on the original one compatible with the Arduino IDE. It prompted research into the most convenient board for a data communication with the lowest power consumption and price.

Sparkfun electronics developed a low-consumption board called ESP8266 module which is an integrated chip with Wi-Fi connection. The main goal is to allow the microcontroller to access a Wi-Fi network. Sparkfun also released an advanced version of the previous model, the ESP32 module which still works with the same specifications as the 8266 with Wi-Fi compatibility, but it adds support for Bluetooth low-energy which is really useful as BLE is used in the previously completed research [1].

(17)

The decision to use the ESP32 was taken due to its low power consumption, meaning it can operate with a voltage range of 2.2 to 3.6V [9]. It also supports IPv4 and TCP/UDP/HTTP/FTP protocols that are really useful for the EEG data communication to the server.

2.3. Data transmission

The communication between the Wi-Fi transceiver board and the server, requires a communication network protocol in order to send the EEG data in small packets. At the early stage of the project, some research was conducted in order to decide which protocol was the most suitable for our system requirements.

The two main network protocols are the User Datagram Protocol (UDP) and the Transmission Control Protocol (TCP) which both run on the top of the internet protocol (IP).

2.3.1 UDP

UDP it is a non-connection-oriented protocol, in other words, it enables process-to-process communication. It does not establish a connection or check if the destination computer is ready to receive the data, it only sends the data directly. UDP is used to transfer data at fast rate as it does not wait for any reception confirmation. It does not guarantee data delivery or re-transmit lost packets, so it is less reliable. It is commonly used to transmit data such as audio and video files.

It provides port numbers to identify the different user requests and can send a value that represents the number of bits in a transmission message, also called checksum. The UDP messages are encapsulated and sent in IP datagram as seen in the following example:

Figure 7: UDP Datagram

2.3.2 TCP

TCP establishes a connection between the source and the destination computer before initiating the communication. One of its characteristics is that it is highly reliable, since it

(18)

In Figure 8 a comparison between both protocols is made. It is easy to identify the fast sending response from the sender to the receiver.

Many client-server applications that have a request and response use UDP instead of taking the trouble to establish and then release a connection [10].

Regarding the project’s specifications, a higher priority has been given to a high-speed communication, leading us to a UDP implementation on the ESP32 board. Hence, in this project, the board will send EEG data segments in packets to the server at high speed and the server will receive and process them without sending any acknowledge back to the sender.

Figure 8: Comparison between UDP and TCP.

(19)

2.4. Real-time communications

Real-time communications have been growing enormously. Our daily activities involve real- time web applications. There are many ways of implementing such web applications, the first model of client connections is the HTTP model that uses a long-polling technique.

Long-polling keeps a HTTP connection open until the server has data to be pushed to the client, this process increases the latency as it leads a bunch of data being transferred to the server every time a HTTP request is done.

The second most common tool for real-time applications is called WebSocket (WS). This method provides a continuous and low-latency connection that enables the server and the client to send messages to each other at any time. This technique is also called full-duplex connection over a TCP single connection. Whereas, HTTP provides a half-duplex communication with a slower speed connection, as tested in [11].

The WS procedure begins with the handshake between the client and the server. Despite the HTTP latency, the client sends a regular HTTP request to the server by pushing a header to inform the server side that the client wants to establish a WebSocket connection.

The are some libraries that implements WS technology to prove the interface. The WS interface is usually divided into two parts: the client side that runs inside the browser and the server side that works with Node.js.

Socket.IO is a library that carries all the WS functionalities and adds multiple improved functions such as providing the event-based communication between the browser and the

Figure 9: Client – Server scheme communication

(20)

2.5. Charting libraries

In order to visualise the EEG data, the project implements a graph on the front-end user interface to plot the received data on a clear and simple graph.

The chart generation is the crucial part for a user-friendly GUI, as the graph is the main tool for doctors to interpret the data. Different libraries dominated by SVG (Scalable Vector Graphics) have been implemented throughout the project in order to reach the performance, speed and resolutions required. Plotly was the library used for in the final implementation of this project:

D3.js

D3 is one of the most common libraries used as it is open source and free to use. Its emphasis on web standards provides compatibility with most browsers and it contains extensive and powerful tools that enable you to implement almost any kind of chart with really useful APIs and documentation to guide you in changing the layout with CSS.

Chart.js

Chart.js is a simple, yet flexible open source JavaScript library. It plots the data smoothly with some predefined animations that let the graph look modern. However, the major drawbacks are the limited number of chart types and CSS customization.

Plotly.js

Plotly differs from the others as it is an online library for data analysis and visualization. It provides very accessible documentation and tutorials. Each of the graphics can be exported in .png, PDF, SVG and EPS formats, and you can choose the width and height measurements of the visualization to suit the requirements for EEG visualisation.

Figure 10: D3.js logo

Figure 11: Plotly.js logo

(21)

3. Methodology / project development:

As previously stated, this project is divided into three different blocks. The first block develops the data acquisition, which had already been implemented by the time this project began. Therefore, text files of previously acquired EEG data was used.

3.1. Arduino (sender side)

Regarding the connection block, a member of the team (Mark O’Sullivan) implemented the UDP protocol, establishing the connection from the board to a private WLAN using C programming language in the Arduino IDE.

The sender implementation on the Arduino board consists of firstly reading the EEG values, where each float value is parsed into four bytes. The decision of parsing the data into four bytes was taken after some previous implementations with sending float values directly resulted in only HEX values being received and many packets were lost.

Therefore, by sending four bytes at the sender part and receiving the HEX values at the receiver, it is easier to assign an array of length four for each value received and parse it back into floats.

The 32 bits value is sent to an array of 4 values where they are sent one by one consecutively inside the following loop:

*(float*)(byteV) = buff[k];

for (int b=0; b<4; b++){

bytes[4*k+b] = byteV[b];

Udp.write(byteV[b]);

}

Figure 12: Parsed values loop

As it can be seen in the following example, in order to parse the HEX value ‘4’ into bytes, an array of length 4 is needed as it converts to “0100”.

HEX

(22)

3.2. Server (receiver side)

Moving back to the receiver creation, the project is mainly implemented with JavaScript for the whole connection block shown in table 1.

A proper connection from the server with the private access point has to be done which uses the UDP protocol in order to establish the communication from the board to the server by assigning the port and IP.

Once the connection is implemented, some tests were completed to check that the data received has no data loss and the hex to float parsing is done correctly for each value.

A client side must be implemented in order to display data on a HTML web page implementation.

There are different protocols which establish connections between a server and a client.

Research was conducted to find the most efficient and suitable protocol that suits the system requirements. As a result, the system uses the WebSocket (WS) protocol [mentioned in 2.4].

The WS protocol requires a basic code structure as seen in Figure 13. It enables the sender to send or broadcast data to one or more clients at the same time.

function onSocketConnect(ws) {

sender.on('message', function(EEG) { //EEG = data received from the sender (Arduino) for (let client of clients) {

client.send(JSON.stringify(f32));

};

});

};

Figure 13: WS code connection at the server side

Regarding the client side, the following code is the basic scheme for the reception and storage of the data in the variable “EEG” which was used in the code.

socket.onmessage = function(event) { var EEG = JSON.parse(event.data);

};

Figure 14: WS code connection at the client side

(23)

3.3. Client

The main purpose of the client is to receive and plot the data in a HTML file that is usually called index.html, where a chart displays the values. A lot of time was invested in the implementation of different graphs until finding that plotly [mentioned in 2.5] was best for plotting the EEG values smoothly.

The firsts steps of the client development were the reception of values as seen previously, pushing the data received to the html file, and the creation of the html file in order to start displaying the values in the browser.

The CDN (Content Delivery Network) is used to include the plotly features into the HTML file, it enables the client to use these features for further chart customization aside from the CSS settings.

A basic representation of the whole file scheme is shown in Figure 15, which gives a better understanding of the system flow.

The Arduino file runs a function that sends a chunk of data every 100ms to the server. The WS protocol establishes the connection between the server and the client, enabling them to send messages to each other at any time without an explicit request. Therefore, the client is able to refresh or update the chart synchronously, and packet delivery latency only falls on the sender Arduino part.

3.4. Charting

The main feature implemented has been the time-scale representation. The chart is supposed to contain 20 seconds of EEG data within the chart.

The system sends one packet that contains 25 EEG values every 100ms, which implies that the client side should be able to update the chart at an average rate of 250 values per

Figure 15: Whole file scheme of the project

(24)

The procedure to change the timescale is determined by equation 1, which re-computes the segment each time the user selects a new value. Speed is the time value selected on the GUI and Fs is the sampling frequency. A maximum buffer value is defined in order to prevent overlap of the window size, which is set to 30. The result of the equation S is the number of samples that need to be cropped in order to reduce the window size.

𝑆 = 𝐹𝑠 × (30 − 𝑆𝑝𝑒𝑒𝑑) (1)

Once the S value is known, the charting script will only show the values of the buffer B from S to length(B) (i.e. the last values in the array). Note that when Speed = 30, S = 0, which means that all values on the buffer are shown (from 0 to length(B)).

A similar implementation was completed for the amplitude range. The unit of EEG amplitude data is 𝜇𝑉, the system is predefined to plot at 100 𝜇𝑉. However, a second dropdown menu has also been implemented with different amplitude ranges of 10, 20, 50, 100, 200 and 500 𝜇𝑉.

Regarding the graph settings, a third option has been implemented in order to make the graph more efficient at the time the browser calls the chart for the first time.

The three settings mentioned were placed on the right side of the web page where a pink section wraps them to display it in a simple but user-friendly interface.

The chart is displayed on the left side of the page, taking approximately 70% of the whole horizontal length screen, while the graph settings occupy the remaining 30%.

At the top-right side of the chart, a small option menu which is implemented by default in the plotly library. This enables the user to take a screenshot of the data plotted, zoom in-

Figure 17: Dropdown menu Figure 16: Time split axis

(25)

3.5. Usability

Two buttons were implemented to enable the user to start and stop the reception and plotting of the EEG data in the chart. Despite both buttons working correctly, a usability study was completed to optimize the performance of the buttons. As a result, the individual start and stop buttons were removed and merged into only one push button that does both tasks and occupies less space.

The first time the graph runs, by default it starts at amplitude 0, with no signal, the user has to click on the “START” button in order to let the client side know when it has to start pushing the data into the web page and updating the chart with new incoming values.

In case it is required to stop the plot, the user would have to click again on the button that stops updating the graph with EEG values and pushes ‘0’ values to the graph as no input.

The decision of implementing dropdown menus for the selection of the Amplitude and Time was also carefully studied in order to ensure the most efficient usage.

Several types of menus and buttons were studied. Radio buttons are really intuitive as the user is able to see at glance all the options available. However, despite its simplicity, it occupies a lot of unnecessary space in the screen.

Another type of button studied was the slider button which gives the user the possibility to choose from a wide range of values. This button could have been useful for selecting the time measured in seconds. However, it has a lack of precision when selecting a specific value.

For this reason, this GUI contains dropdown menus where the user is able to click and choose between a long range of values, while occupying very little space.

Figure 18: First implementation of two buttons

Figure 19: Performance of the main button

(26)

3.6. GUI

The last stage implemented was the design of the final GUI. Many designs were completed throughout the project, with the several implementations and tests of the different charts.

Each chart commonly uses its own design by default. As already mentioned, some libraries allowed us to change its design while others do not. Therefore, a draft design had to be done considering the default chart setting driving the web design into a different idea from what was originally expected.

Since the final implementation of the chart was completed using the plotly library, it was possible to change the predefined graph settings, such as colours, fonts, grid options and even the plotting line directly in the JavaScript function.

In terms of the web page layout, the system has been designed as similar as possible to the android version in order to establish a multi-platform/device system.

A basic colour pallet used in the previously designed android app version can be seen in Figure 20. In the image below, the patient ID can be seen, this project did not implement this feature due to the magnitude of the development.

In both cases, the design has been implemented using RGB colour codes, so it was easy to use same reference colours for the dark grey background and the pink colour that represents the INFANT Research Centre brand.

In comparison to the Android version, this project uses a white background colour and a pink plotting line. This simple change was done in order to ease future work on the project, as more than one trace of EEG is often plotted within one graph, therefore, future implementations will be able to add more traces which can be easily identified by assigning different line colours to each of them.

A basic header has been added to the web page where the INFANT logo can be found on the right side while the project and app name are shown on the left. It is a fixed header for aesthetic reasons, the header stays in the same place even if the page is scrolled down, the element would not leave a gap as it is displayed above the background content.

Figure 20: Android APK version

(27)

3.7. Window resize event

Regarding the two sections on the webpage, the chart and the graph settings block, an implementation was done in order to detect when the user is readjusting the window size.

In the case which the user shrinks the window, a threshold has been set to detect when both sections are about to overlap and then the right section goes to the bottom part so that the chart can keep a proportional scale size for a proper interpretation.

Figure 21: Window resize event

The same implementation was done for the opposite case where the user enlarges the window so that both sections have enough space to be displayed together, then the blocks come back to inline section as the original layout.

(28)

4. Results

At the end of the project, the initial goal was successfully achieved. Despite several tests with different libraries for plotting charts, the system is able to send the EEG data from the text file to a server through a Wi-Fi transceiver and plot the data on a user-friendly web GUI.

First tests were implemented in order to check the correct reception of parsed values on the server side. In the following figure it can be seen the data that is received in hexadecimal format directly from the Arduino board and the latter is the data parsed into floats, specifically the 25-length array with EEG values that is then sent to the client side.

On the client side, in order to check that there was no data lost through the communication, a 1Hz sine wave was generated so that it was easy to identify any lost packets in the signal reception.

Figure 22: Comparison of data received in the server side.

Figure 23: 1Hz sine wave as input value

(29)

The final design is shown in Figure 24. The three main blocks can be easily identified. The header with the project name and the Infant logo is at the top, while the two big sections are in-line: the chart on the left side and the graph settings on the right side.

It can be seen in the chart that the EEG data is being plotted with an amplitude of 100 𝜇𝑉 and a time representation of 5 seconds on the x-axis.

Figure 24: Final design web page

(30)

5. Budget

The project has required free licence software*, one Arduino board and one Junior engineer.

The total budget of the project is:

Items Cost (€)

Components 510

Personnel 12000

Free licence Software 0

Total 12510

Table 2: General Project Budget

Components:

Component Units Cost

Arduino ESP32 Board 1 10€

Computer 1 500€

Table 3: Components list

Personnel:

Employee Annual Salary Months Cost

Research Assistant/Junior

engineer

24000

6 12000€

Table 4: Personnel table

*The Software required for this project has been the Arduino IDE and a source-code editor that enables JavaScript and HTML coding.

(31)

6. Conclusions and future development:

In conclusion, the project’s requirements and goals have been successfully achieved. The development of a new platform for EEG interpretation has been completed, allowing the end-user to connect (with a wide browser compatibility) to a web page in order to interpret the real-time EEG data received in a user-friendly GUI in order to increase the demographic of clinical staff capable of understanding and assessing the plotted EEG data.

Despite the functional performance of the system, the project did not merge the fully functional acquisition block, therefore, a recommendation for future development would be to start working on the early stages of this project in order to develop a connection directly with the dry-electrodes connected to the newborn’s scalp. Another interesting improvement would be an option on the GUI to connect to a newborn in real-time or instead to interpret EEG data already stored on the server with off-line communication.

Meanwhile, more research and development could be done to explore and implement new features on the web page. For instance, an interesting feature would be the option to add more EEG signals using different traces on the chart, enabling comparisons between the signals or even between different patients.

New projects could be proposed based on this system. Regarding the raw signal, some process could be done in order to work with different blocks of signal processing as developments of algorithms for EEG robustness, filtering and resampling or even the addition of sonification algorithms.

(32)

Bibliography:

[1] O’Sullivan, M., Gomez, S., O’Shea, A., Salgado, E., Huillca, K., Mathieson, S., Boylan, G., Popovici, E., Temko, A. (2018) Neonatal EEG Interpretation and Decision Support Framework for Mobile Platforms. In Proceedings of the 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Honolulu, HI, USA.

[2] Murray, D.M., Boylan, G.B., Ali, I., Ryan, C.A., Murphy, B.P., Connolly, S. (2008). Defining the gap between electrographic seizure burden, clinical expression and staff recognition of neonatal seizures. Arch. Dis. Child. Fetal Neonatal Ed. 2008, v. 93, pp 187–191.

[3] Temko, A., Thomas, E., Marnane, W., Lightbody, G., Boylan, G. (2011). EEG-based neonatal seizure detection with Support Vector Machines, Clinical neurophysiology: official journal of the International Federation of Clinical Neurophysiology. v. 122, pp. 464–473.

[4] O’Sullivan, M., Pena, J.P., Bocchino, A., O’Mahony, C., Costello, D., Popovici, E., Temko, A.

(2017). Comparison of electrode technologies for dry and portable EEG acquisition. In Proceedings of the 2017 7th IEEE International Workshop on Advances in Sensors and Interfaces (IWASI), Vieste, Italy, pp. 15–20.

[5] Khamis, H., Mohamed, A., Simpson, S., Mcewan, A. (2012). Detection of temporal lobe seizures and identification of lateralisation from audified EEG. Clinical neurophysiology: official journal of the International Federation of Clinical Neurophysiology. v. 123. pp. 1714-20.

[6] Gomez, S., O'Sullivan, M., Popovici, E.M., Mathieson, S., Boylan, G.B., Temko, A. (2018). On sound-based interpretation of neonatal EEG. 2018 29th Irish Signals and Systems Conference (ISSC), Ireland.

[7] Flanagan, J. L. and Golden, R. M. (1966) Phase vocoder, in The Bell System Technical Journal, v. 45, no. 9, pp. 1493-1509.

[8] O'Shea, A., Lightbody, G., Boylan, G., Temko, A. (2018). Investigating the impact of CNN depth on neonatal seizure detection performance. In International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Honolulu, HI, USA.

[9] ESP32 Thing Hookup Guide, https://learn.sparkfun.com/tutorials/esp32-thing-hookup- guide?_ga=2.168659048.854105827.1575999121-1872461687.1575999121 (Last access December 2019)

[10]Molina E.M., “Protocolo de datagramas de usuario (UDP)”:

http://www.redtauros.com/Clases/Fundamentos_Redes/07_Protocolo_Internet_UDP.pdf (Last access November 2019)

[11]Matt West, (October 2013). An Introduction to WebSockets: https://blog.teamtreehouse.com/an- introduction-to-websockets (Last access November 2019)

(33)

Appendices:

1. Work Packages

Project: Design of a web graphic user interface for real-time and off-line EEG interpretation

WP 1

Major constituent: Background/Introduction to Arduino Start event: 16/09/2019 End event: 30/09/2019 Short description:

Studying the state-of-art of Arduino boards, the different possible implementations.

Study the different code languages required.

Internal task T1: Check for Arduino ESP-32 basics.

Internal task T2: Simple first implementation of Arduino scripts.

Internal task T2: Get familiar with the HTML, CSS languages.

Project: Design of a web graphic user interface for real- time and off-line EEG interpretation

WP2

Major constituent: hardware prototype, software Start event: 01/10/2019 End event: 13/10/2019 Short description:

The Android board implementation in order to send EEG data via web-socket. Analyze the most efficient way of sending packets with different protocols.

Establish the communication between servers and monitoring devices and implement the receiver part using JavaScript protocols.

(34)

Project: Design of a web graphic user interface for real-time and off-line EEG interpretation

WP 3

Major constituent: SW Start event: 14/10/2019

End event: 18/12/2019 Short description:

Web page implementation

Plot the EEG data received.

HTML and CSS course at Code Academy.

Internal task T1: Web page implementation Internal task T2: Plot the EEG data received.

Internal task T3: HTML + CSS course

Internal task T4: Implement the webpage using HTLM, CSS.

Project: Design of a web graphic user interface for real- time and off-line EEG interpretation

WP 4

Major constituent: SW Start event:18/12/2019

End event: 30/12/2019 Short description:

Implement and design a user interface webpage to display the EEG data considering the usability of the final user.

Reach a multiple connection and adapt the system for more than one EEG data sender.

Internal task T1: Design the user interface usability.

Internal task T2: Integrate the graph into the webpage.

Internal task T3: Scalability.

T1: FINISHED T2: FINISHED

T3: Pending on external project

(35)

Project: Design of a web graphic user interface for real-time and off-line EEG interpretation

WP 5

Major constituent: SW Short description:

Implement the established filtering and down-sampling of the EEG signal acquired through the board to finish the system.

Pending on external project

Internal task T1: Filtering

Internal task T2: Down sampling

1.2 Milestones

WP Task Short title Milestone / deliverable Date

2 4 Reception of the EEG data Receive the data sent from ESP32 13/10/2019

3 1 Webpage Running webpage with chart 21/10/2019

3 4 Display EEG Display the chart with proper EEG data 18/12/2020 5 2 Final product Final UI with real-time EEG acquisition 26/01/2020

(36)

Glossary

EEG: Electroencephalogram

aEEG: Amplitude integrated electroencephalography BLE: Bluetooth Low Energy

CNN: Convolutional Neural Network TCP: Transmission Control Protocol UDP: User Datagram Protocol IP: Internet Protocol

WS: WebSocket

HTTP: Hypertext Transfer Protocol SVG: Scalable Vector Graphics

IDE: Integrated Development Environment CDN: Content Delivery Network

GUI: Graphical User Interface RGB: Red, Green, Blue CSS: Cascading Style Sheets JS: JavaScript

DESIGN OF A WEB GRAPHIC USER INTERFACE FOR REAL-TIME EEG INTERPRETATION