Modifications to JASigning

Deliverables of the projects during which the JASigning system was developed and notes on a website are the main source of documentation for the system. Not all planned features have been fully implemented, some because they are used very infrequently, others because there is insufficient linguistic research on which to base an implementation. Together with my two Deaf collaborators, I identified the avatar functionality we needed for our project. In close collabo- ration with the developers of JASigning, I then found workarounds for those features that were not yet available in the system.

For example, the Gestural SiGML<hamgestural_segment> element (cf. Section 3.6.3), with which non-manual features can be applied to multiple signs, is not yet implemented. To replace its functionality, I modified the timing behavior of the non-manual features I wanted to extend over more than one sign.7

The fitpicturetomanual8 _{attribute that synchronizes the duration of the mouthing and the}

manual activity of a sign is also not yet implemented. Substituting this attribute is not straightfor- ward. In our case, the duration of a mouthing mostly exceeded the duration of the corresponding manual activity. Since the participants of our focus group study (cf. Section 5.3.3) remarked that the speed of the mouthings was generally too low, I sped up the mouthings by 20%.

In addition, I adjusted many of the SiGML-to-morph mappings: For example, I modified the codeSH (head shake) in such a way that it involved fewer movements of the head with higher amplitudes. These changes were again motivated by feedback from Deaf experts.

While I was able to find workarounds for most features that were not yet available in the system, one remaining issue was how to cause non-manual components to slightly precede the manual activity of a sign. For example, the DSGS train announcements contained indexical (pointing) signs. The signs were accompanied by a shift in eye gaze towards the location of the index- ical sign. In order for the signing to appear natural in this case, the onset of this non-manual component should precede the manual activity (pointing) slightly.9

7_{As outlined in Section 5.3.1, the morph(s) underlying a non-manual feature can be anchored to the start and/or} end of a manual activity.

8_{Mouthings are sometimes referred to as “mouth pictures”, in contrast to mouth gestures (cf. Section 2.1), which} are also known as “mouth forms”.

5.3.3 Evaluation

Like Kipp, Nguyen, et al. (2011), I carried out a focus group study to obtain feedback from mem- bers of the DSGS community on how to improve the avatar signing DSGS train announcements. I chose the focus group method over single-case studies in order to provide a more informal setting, where Deaf people were free to exchange their thoughts rather than feel like they were part of a clinical experiment (Huenerfauth et al., 2007).

I followed the recommendation of Kipp, Nguyen, et al. (2011) to provide a sign-language-only setting, i.e., no hearing persons were allowed in the room in which the evaluation took place, myself included. One of the two Deaf members of our project acted as session moderator. We invited seven participants who were active members of the Deaf community and early learners of DSGS, which, different from Kipp, Nguyen, et al. (2011), I believe to be a crucial prerequisite for a successful evaluation. The group consisted of four men and three women of ages 22 to 69 (cf. Table 5.2 for the complete age distribution).

Participant ID Age Gender

1 22 F 2 39 M 3 42 M 4 49 F 5 51 F 6 58 M 7 69 M

Table 5.2: Demographic information about the participants of the study

The chairs were arranged in a semicircle, without table to help provide a more casual and personal atmosphere as well as assure that all participants could see both the screen and each other. One of the participants had Usher syndrome, i.e., he is Deaf as well as gradually becoming blind. Since he has difficulty adjusting to different lighting conditions and backgrounds, we placed one chair in front of a dark background. The moderator asked each participant wanting to make a statement to take a seat in this chair. Figure 5.9 shows the arrangement of seats. The discussion was recorded with four cameras (of which two are visible in Figure 5.9). Nine signed sentences were projected on a screen. The sentences had been chosen so as to reflect important characteristics of the sign language of our corpus, such as use of fingerspelling, time specifications, indexical signs, or lists of signs (cf. Section 3.6.1). For every sentence, the moderator asked for the participants’ subjective opinion. She replayed avatar sequences upon request.

Figure 5.9: Focus group study setting

The participants recommended to slightly raise the avatar’s eye gaze so that it would appear to be directed more towards the viewer. They found the posture of the avatar and the display window appropriate. However, they felt the transitions between some signs to be too abrupt. Moreover, they recommended for the hands to return to a neutral position at the end of every signed announcement rather than to come to rest in the final posture of the announcement. The participants recommended slightly speeding up the mouthings. They also observed that the avatar’s teeth and tongue were hardly visible; they found visibility to be necessary, e.g., when forming the mouthing for the fingerspelling sign -N-. They also found the speed of fingerspelling to be too high.

A long discussion evolved about how to deal with lists of place names. Where several place signs appeared together, we had introduced a short pause after each. The participants found that this was not sufficient. They discussed the following as different possible strategies:

• Preceding every place sign with the sign ORT (‘PLACE’) as a contextualization marker; • Returning the hands to a neutral position after every place sign; or

• Performing a sign like THEMAWECHSEL (‘CHANGE-OF-TOPIC’) or WEGSCHIEBEN (‘PUSH-ASIDE’) after every place sign.

In the end, they opted for a combination of the first two strategies: performing the sign ORT once, then returning the hands to a neutral position after every place sign. The participants also suggested using the contextualization marker ORT together with single occurrences of place signs, even the widely known ones such as ZÜRICH, BASEL, or LUZERN.

Our conventions for time specifications had initially adhered to the format UHR <STUNDEN> PUNKT <MINUTEN> (‘CLOCK <HOUR-NUMBER> DOT <MINUTE-NUMBER>’) to re- flect the fact that they originated in a timetable. However, the participants did not approve of this format. They suggested using instead a phrasing more familiar to them without the sign PUNKT

(‘DOT’): <STUNDEN> UHR <MINUTEN> (‘<HOUR-NUMBER> CLOCK <MINUTE-NUMBER>’) Regarding time specifications, the participants also remarked that a spatial offset between the

signing location of the number of hours and the number of minutes was missing: They pointed out that in a temporal expression like 22 UHR 41 (‘22 CLOCK 41’), the number of hours (22) should be signed in front of the body and the succeeding number of minutes (41) slightly to the right. The same convention was recommended for train names involving numbers, e.g., S6, where S should be signed in front of the body and 6 slightly to the right.

The participants also found that the default transition time between specific combinations of signs was too long. This involved compound-like sign sequences such as BAHN VERKEHR (‘RAILROAD TRAFFIC’), ABFAHRT ORT (‘PLACE OF DEPARTURE’), or FAMILIE WA- GEN (‘FAMILY WAGON’), but also cases in which DSGS uses two signs to refer to a sin- gle concept, like AUGE VORSICHT (‘EYE CAUTION’) for Vorsicht (‘caution’), VERSPÄ- TUNG NACH (‘DELAY AFTER’) for Verspätung (‘delay’), or SCHLIESSEN ZU (‘CLOSE CLOSED’) for schliessen (‘close’).

Following this feedback of the focus group participants, I made several improvements to the DSGS avatar. For example, I caused the hands to return to a neutral position at the end of every signed announcement. I slightly sped up the mouthings and decreased the speed of fin- gerspelling. I introduced the contextualization marker ORT (‘PLACE’) before place signs and for lists additionally caused the hands to return to a neutral position after every place sign. I also changed the format of time specifications such that different sets of glosses and corresponding HamNoSys notations for numbers were created: Instances of <STUNDEN> (‘<HOUR NUM- BER>’) were signed in front of the signer’s body. For instances of <MINUTEN> (‘<MINUTE NUMBER>’), two cases were possible: If the number was between 00 and 09, the first digit was signed in front of the signer’s body and the second to the right; in all other cases (numbers from 10 onward), the digits were signed as one number to the right of the signer’s body. I also eliminated the temporal gap between compound-like sign sequences by introducing additional (compounded) lexicon entries for these occurrences.