Overall Performance

Throughout the years of implementing, we always checked how the virtual director module decides and behaves. At first, it module produced only a log file, showing the active state number and the final probabilities of the selection process. Below there is a short snippet of such a log file:

€

ActivStateNo: 4

Possibility# 0, p=0.81 for NewStateNo: 3 Possibility# 1, p=0.81 for NewStateNo: 3

Possibility# 2, p=0.81 for NewStateNo: 3 Possibility# 3, p=0.855 for NewStateNo: 6 Possibility# 4, p=0.855 for NewStateNo: 6 Possibility# 5, p=0.7695 for NewStateNo: 4 Possibility# 6, p=0.7695 for NewStateNo: 4 Possibility# 7, p=0.81 for NewStateNo: 2 Possibility# 8, p=0.81 for NewStateNo: 2 Possibility# 9, p=0.15 for NewStateNo: 15 Possibility# 10, p=0.135 for NewStateNo: 7 ActivStateNo: 6

Possibility# 0, p=0.81 for NewStateNo: 3 Possibility# 1, p=0.81 for NewStateNo: 3 Possibility# 2, p=0.6885 for NewStateNo: 4 Possibility# 3, p=0.6885 for NewStateNo: 4 Possibility# 4, p=0.81 for NewStateNo: 5 Possibility# 5, p=0.81 for NewStateNo: 5 Possibility# 6, p=0.81 for NewStateNo: 2 Possibility# 7, p=0.81 for NewStateNo: 2 Possibility# 8, p=0.15 for NewStateNo: 15 Possibility# 9, p=0.135 for NewStateNo: 7 ActivStateNo: 5

Possibility# 0, p=0.81 for NewStateNo: 3 Possibility# 1, p=0.81 for NewStateNo: 3 Possibility# 2, p=0.81 for NewStateNo: 6 Possibility# 3, p=0.81 for NewStateNo: 6 Possibility# 4, p=0.81 for NewStateNo: 5 Possibility# 5, p=0.81 for NewStateNo: 5 Possibility# 6, p=0.81 for NewStateNo: 2 Possibility# 7, p=0.81 for NewStateNo: 2 Possibility# 8, p=0.15 for NewStateNo: 15 Possibility# 9, p=0.135 for NewStateNo: 7 ActivStateNo: 2

Possibility# 0, p=0.9 for NewStateNo: 3 Possibility# 1, p=0.9 for NewStateNo: 3 Possibility# 2, p=0.9 for NewStateNo: 2 Possibility# 3, p=0.81 for NewStateNo: 5 Possibility# 4, p=0.81 for NewStateNo: 5 Possibility# 5, p=0.7695 for NewStateNo: 5 Possibility# 6, p=0.855 for NewStateNo: 6 Possibility# 7, p=0.855 for NewStateNo: 6 Possibility# 8, p=0.15 for NewStateNo: 15 Possibility# 9, p=0.135 for NewStateNo: 7 ActivStateNo: 2

Possibility# 0, p=0.9 for NewStateNo: 3 Possibility# 1, p=0.9 for NewStateNo: 3 Possibility# 2, p=0.7695 for NewStateNo: 2 Possibility# 3, p=0.855 for NewStateNo: 5 Possibility# 4, p=0.855 for NewStateNo: 5 Possibility# 5, p=0.81 for NewStateNo: 5 Possibility# 6, p=0.9 for NewStateNo: 6 Possibility# 7, p=0.9 for NewStateNo: 6 Possibility# 8, p=0.15 for NewStateNo: 15 Possibility# 9, p=0.135 for NewStateNo: 7 ActivStateNo: 3

€

State numbers 2, 3, 4, 5, and 6 belong to the standard lecture context while state num- ber 7 belongs to the question context and is only selected if a questioner gets ac- knowledged. Finally, state number 15 is the EndOfLecture state of the FSM, it is only selected when the lecture is over.

The total number of possibilities coming out of the active state is not limited, and is defined by the given FSM and all the possible ways of transition to another shot, like hard cut or cross-fade, for example. This is also the reason why more than one possi- bility can be used to access a new state. They only differ in the way they perform the transition, either as a hard cut or as a cross-fade.

In the next step, we have recorded the videos of all cameras and the audio of the slides camera. In addition, the director module wrote Edit Decision Lists (EDLs) in the style of the common but proprietary CMX3600 format, originally used for the machines of CMX Editing Systems which focused on on- and off-line editing in post-production in the 1970s and 1980s. The scheme of an EDL file is shown in Figure 32.

Figure 32: Explanation of an EDL entry.

The first line contains the title which is used to name the production. The second line shows the first real instruction for the resulting production. Its consecutive number „000… is followed by a token to identify the so-called „reel…. Originally, a reel is a film spool which e.g., contains a continuous shot of a scene. All reels together are called „footage…. Today, a reel is also the electronic representation of a reel called clip. The token gets mapped to the real part of the footage in the next line, containing comments. Here, the token „AX… gets mapped to the clip „03-06-2008_15-31-29- LONGSHOT.mp4.avi…. There are two ways to map tokens to clips: the first one is to define one token per clip by a comment line and only use different tokens per instruc- tion. The second way is to always use the same token and every time map a different clip to that token. The latter way is used by Adobe Premiere Pro when exporting EDLs out of its time line. We adopted this way for our prototype as we wanted to use Premiere to finalize our production at that time.

The letter or letters in the orange-colored ellipse mark which tracks are involved by this instruction. Typical versions are: „V… ‡ only the video track, „A… ‡ only the mono audio track, „AA… ‡ only the stereo audio tracks…, „A/V… ‡ mono audio track and video track, and „AA/V… ‡ stereo audio tracks and video track.

The letter in the cyan-colored ellipse describes the way of transitioning between shots. „C… stands for „hard Cut… and „D… for „Dissolve….

The first two SMPTE timestamps set the „Play In… and „Play Out… points of the source reel, the last two SMPTE timestamps set the „Record In… and „Record Out… points of the resulting production.

This format can be easily imported by the editing software Adobe Premiere Pro. It accesses the recorded materials of the cameras, called „footage reels…. Below, we show an example of what an EDL of our prototype looks like:

TITLE: Sequenz 01

000 AX V C 00:00:00:00 00:00:02:01 00:00:00:00 00:00:02:01 REEL AX IS CLIP 03-06-2008_15-31-29-LONGSHOT.mp4.avi

001 AX V C 00:00:02:01 00:00:12:03 00:00:02:01 00:00:12:03 REEL AX IS CLIP 03-06-2008_15-31-29-LECTURER.mp4.avi

002 AX V C 00:00:12:03 00:01:03:11 00:00:12:03 00:01:03:11 REEL AX IS CLIP 03-06-2008_15-31-29-SLIDES.mp4.avi

003 AX V C 00:01:03:11 00:01:35:07 00:01:03:11 00:01:35:07 REEL AX IS CLIP 03-06-2008_15-31-29-LECTURER.mp4.avi

004 AX V C 00:01:35:07 00:02:01:20 00:01:35:07 00:02:01:20 REEL AX IS CLIP 03-06-2008_15-31-29-AUDIENCE.mp4.avi

005 AX V C 00:02:01:20 00:03:52:19 00:02:01:20 00:03:52:19 REEL AX IS CLIP 03-06-2008_15-31-29-SLIDES.mp4.avi

006 AX V C 00:03:52:19 00:06:03:16 00:03:52:19 00:06:03:16 REEL AX IS CLIP 03-06-2008_15-31-29-LECTURER.mp4.avi

007 AX V C 00:06:03:16 00:06:56:09 00:06:03:16 00:06:56:09 REEL AX IS CLIP 03-06-2008_15-31-29-LONGSHOT.mp4.avi

...

287 AX V C 01:24:58:01 01:25:08:13 01:24:58:01 01:25:08:13 REEL AX IS CLIP 03-06-2008_15-31-29-LONGSHOT.mp4.avi

288 AX A C 00:00:00:00 01:25:08:13 00:00:00:00 01:25:08:13 REEL AX IS CLIP 03-06-2008_15-31-29-SLIDES.wav

At this point in development, we recorded four camera video tracks and one audio track in parallel, and so the SMPTE timestamps of „Play In… and „Record In… as well as the SMPTE timestamps of „Play Out… and „Record Out… are always identical in a line. The last instruction of our EDL handles the audio tracks of the production. As at that time we only recorded one audio track, we use it for the entire production.

When all files are available to Premiere it produces the final video based on the direc- tor†s decisions given through the EDL. This process is called „on-lining… in broad- caster†s slang. Using EDLs gave us the first possibility to produce a real video based on the director†s decisions at a time when the AV Mixer/Recorder was not yet ready. An example of such a video was presented in (Lampi, Kopf & Effelsberg, 2008).

As soon as the AV Mixer/Recorder was functional, we automatically received the final result, as we had planned. During these steps of development, the diversity of the selected shots and transitions behaved as expected because the reaction on sensor in- puts worked properly. We only had to make sure that too many similar sensor inputs did not lead to some kind of a „Denial-of-Service (DoS) attack…. Therefore, we intro-

duced a clearing instance purging all future inputs that were still in the message queue, leading to the same context as the one already active.

In order to keep the change of the context in the way it was planned, we also had to make sure that the FSM would not be stuck in the question or answer context forever in case a user forgot to click on the right button in the heat of the moment. For exam- ple, a lecturer may forget to hit the „Now Answering… button before starting to an- swer, or neither the lecturer nor the questioner closes the answer by clicking on the „Answer OK… button. So, we introduced time-outs for the question context and the answer context which are refreshed by any click on a relevant button of the question manager. Now, it is assured that neither the question context nor the answer context are left too early, and both contexts will finally be left to reset the active context to the standard lecture context.

Our experience with the system shows that for recording lectures our prototype is well suited. For adapting it to other contexts, the FSM and maybe some weights of the dif- ferent shots or the values manipulating the possibilities during the transition selection need to be revised.