Phase Ⅰ

By analyzing the annotations of FaceReader and Semaine database, Phase I aims to found out how Obadiah (Gloomy) and Poppy (Happy) characters influence the emotions of users. Because Obadiah character is gloomy, and Poppy character is happy. If the emotional interaction were effective to influence people in the Semaine annotation, the users, who talk to Obadiah character, should respond more gloomily than the users interacting with other characters. Similarly, the users should interact with Poppy character more happily than the users interacting with Obadiah character. In the results of FaceReader, the user, who speaks with Obadiah and Poppy characters respectively, should express contrary emotions. Other characters would be considered if the results of Phase I had succeed in supporting any hypothesis in the section 1.3.2.. Otherwise, it is unnecessary to continue the research.

Facial expression is the most explicit modality for emotional expression. Therefore, FaceReader was evaluated before LIWC.

4.2.1. Investigate the emotions/characters impacts as annotated in the Semaine annotation

Table 12. Experiments for section 4.2.1.

This table displays the codes for the datasets that we will compare. Each code has one letter and one number. The letter indicates the modality/database type. The number indicate the character Type.

E.g. D1 means the modality/database D (Seamine database) and the character 1 (Obadiah)

The comparison plan of Semaine annotation was designed as table 12. The comparison between D1 and D2 groups can analyze the character impacts on the happy and sad emotions as annotated in the Semaine annoation. If the users of D2 group had achieved higher value of happy emotion, it would be regarded as the evidence for the emotional impacts of Poppy character. Similarly, if the users of D1

Modalities SAL character (Obadiah) SAL character (Poppy) Manually

group had expressed more sad emotion, it would support the assumption about the emotion impacts of Obadiah character.

Table 13. Functionals of basic emotions

During the analysis, the calculation of functionals in each dimension of Semaine annotation was the approach to measure the difference. These functionals included the minimum, maximum, standard deviation, and etc. They are displayed in the table 13. After the functionals analysis, T-test for two independent samples was used to check whether the expressed emotions of D1 and D2 are significantly different from each other.

4.2.2. Investigate the emotions/characters impacts as measured by the FaceReader

Table 14. Experiments plan for Section 4.2.2.

This table displays the codes for the datasets that we will compare. Each code has one letter and one number. The letter indicates the modality/database type. The number indicate the character Type.

E.g. A1 means the modality/database A (FaceReader) and the character 1 (Obadiah)

The investigation would continue if there were any difference in the manual annotation regarding how people respond to Obadiah and Poppy characters. It is

Label Meaning Valence Basic emotions ….. ……

Min Minimum value

Mean Mean value

Max Maximum value

SD Standard Deviation

MinMagnRises Minimum Magnitude of Rises MeanMagnRises Mean Magnitude of Rises MaxMagnRises Maximum Magnitude of Rises

SDMagnRises Standard Deviation of Magnitude of Rises MinMagnFalls Minimum Magnitude of Falls

MeanMagnFalls Mean Magnitude of Falls MaxMagnFalls Max Magnitude of Falls

SDMagnFalls Standard Deviation of Magnitude of Falls FreqChanges Frequency of value Changes

FreqRises Frequency of Rises FreqFalls Frequency of Falls

Modalities SAL character (Obadiah) SAL character (Poppy) Outputs of

Automatic Tools

FaceReader (Facial

necessary to check how characters influence the emotional expression of people as measured by the automatic tools (table 14). The comparison between A1 and A2 groups was used. The procedures of functionals analysis and the T-test (section 4.2.1) were repeated to analyze the measurements of FaceReader. These analysis were used to explore how the characters influence the user emotion as measured by the FaceReader, and whether the expressed emotions of A1 and A2 groups were significantly different from each other.

4.2.3. Investigate the dimensional correlation between FaceReader and Semaine annotation

Table 15. Experiments plan for Section 4.2.3.

This table displays the codes for the datasets that we will compare. Each code has one letter and one number. The letter indicates the modality type. The number indicate the character Type.

E.g. A2 means the modality A (FaceReader or Facial expression) and the character 2 (Poppy).

Table 15 displays the comparison plan between FaceReader and Semaine annotation (A1&D1, A2&D2). Because SAL characters relate to different personalities (Obadiah - gloomy and Poppy - happy). If these characters had the emotion impacts on the users, the expressed emotions of participants should have been aligned with virtual characters in the annotation of Semaine database. If facial expression and FaceReader were reliable to reflect and detect the user emotions, the emotion impacts of characters should be found in the measurement of FaceReader as well.

Table 16. Psychological connections of basic emotions between FaceReader and characters.

Table 7 lists the most used 13 optional traces for each character in the Semaine annotation. The table also displayed the frequency of each selected dimension. From the table 7, the impacts on the users’ emotions of each character are partially reflected via the most frequently used emotional dimension for each character. If facial expression and FaceReader were reliable to reflect and detect the user emotions, the measurement of basic emotions from the FaceReader should be consistent with the emotion distribution in the table 7. According to the most frequently used emotional

Characters

Automatic tools Manually annotations FaceReader (Facial Expression) Semaine (Multiple modalities)

SAL character (Obadiah) A1 D1

SAL character (Poppy) A2 D2

Character Basic emotions of Semaine annotation FaceReader and its basic emotions

Obadiah Sadness Sad

Spike Anger Angry

Poppy Happiness Happy

dimension for each character in the table 7, the corresponding dimension of FaceReader annotation is concluded in the table 16. Table 17 lists all the common dimensions between Semaine annotation and the measurement of FaceReader.

Table 17. The common dimensions between FaceReader and Semaine database

The emotionally corresponding dimensions in the table 16 is helpful to explore whether the characters have the emotion impacts on the facial expression of users. The common dimensions, which are listed in the table 17, are helpful to validate the emotion impacts of characters as measured by the FaceReader, and are beneficial to check whether FaceReader can recognize the user emotion as good as the Semaine annotation.

Figure 9. Pearson Correlation Calculator. i means the index of trace values. x and y represent the dimensions of FaceReader

Pearson correlation calculator (figure 9) was used to measure the agreement between the measurement of FaceReader and the manual annotation of Semaine database. The correlation calculation was based on the dimension match in the table 16. The final result of correlation analysis for each character were a Pearson correlation

Dimensions FaceReader Semaine database

Rating dimensions Arousal Activation Valence Valence Power Expectation Intensity Basic emotions Netural Happy Happiness Sad Sadness Angry Anger Surprised Scared Fear Disgusted Disgust Amusement Contempt

matrix as table 18 (take Obadiah character for example). Each item of the matrix (table 18) represented the correlation between one dimension of Semaine annotation and one dimension of FaceReader measurement.

Table 18. Pearson correlation matrix of Obadiah.