you can use today's 3D packages such as Maya to skip this phase if needed (though many productions still use exposure sheets) and use the shape and sound of the voice track within the program itself as a guide.
In this chapter, we'll discuss the basic theory of lip-synching and give you some general guidelines for doing it properly and efficiently. We'll then work through two sample scenes, putting this theory into practice. After completing this chapter, you should have a solid understanding of how to produce your own lip-synched animations and be ready to create your own
"invisible" vocal animations.
drawback is that it takes a good deal of time and some understanding of the technique to get the whole process up and running.
Here is a general outline of how to proceed, using this method, to create effective lip-synch animation.
Preproduction Tasks
As with all animation work, the first step is preproduction. In preproduction, you create the storyline, script (obviously an important step when lip-synching is involved) the look and feel of the animation, and create storyboards that describe the action. If you are working on a series of similar animations (for example, a Saturday-morning cartoon series), you will likely have a "bible" that contains the general art direction, character descriptions, and pos-sibly technical material.
In addition to creating the general look of your models and scenes and creating the dia-log, in the preproduction phase you might also define general issues of how your character(s) will speak. For example, you might decide how realistic the lip-synching needs to be and determine your general methodology. A little forethought at this stage can save valuable time later. For example, if the characters are extremely cartoonish and stylized, you might need only simple "open/closed" positions for their mouths. On the other hand, if the animation requires extreme realism, now is the time to face this challenge head-on and be sure you have enough resources to tackle this task.
Never overanimate your characters, lf they need only simple mouth shapes, don't build a complex and/or hard-to-use system for lip-synching. You are only wasting your—and your company's—money.
Recording Vocal Tracks
Once you make basic decisions about the animation and the dialog track is "locked," or fin-ished, it is time to find voice talent and record your dialog! There are really only two secrets to getting a good voice track for your animation:
• Finding talented, hard-working actors, preferably with interesting, unique voices
• Getting the cleanest recording you can
The number of times otherwise good dialog tracks have been ruined by poor recording is probably too high to count, and even decent recordings often need lots of massaging to fix
problem areas, all of which leads to wasted time and money and a lot of frustration that can be avoided by creating or renting an appropriate facility in the first place.
If you have the budget, by all means rent a recording studio for the time you need it. A good rule of thumb is that you need about an hour of recording time per minute of finished dialog. If you can't afford to rent a studio, see if you can beg facilities from somewhere close by. Often—especially for nonprofit projects—managers of facilities will allow recording ses-sions for little or no charge. The one problem that often arises in these circumstances is that the recording session has to take place at odd hours, which can be stressful for cast and crew.
If you must construct your own recording space, try to keep the following points in mind:
• Find the best microphone you can, and never, ever use the built-in microphone on a camera or a camcorder.
• Find the acoustically deadest space you can. An anechoic chamber is best; otherwise, use heavy drapes, foam, or other sound-deadening materials to reduce echo.
• Remember that floors and ceilings create echo, so lay out blankets, carpeting, or other materials on them.
• Listen carefully to your space or do a test recording. Listen for any kinds of hums or
"leaking" sound from the outside world. Anything from fluorescent light fixtures to air-conditioning can cause a low level of noise in your recording that is difficult to delete.
• If possible, have only your voice actors and the microphone(s) in the room. The fewer machines (recording devices, cameras, computers) in the room, the less noise you will get.
• Never try to create an effect when recording. For example, don't try to capture a natu-ral echo if your characters will be in a cave. With the audio-engineering software avail-able today, it's extremely easy to add this type of effect, but almost impossible to get rid of an incorrect echo or other effect after it's recorded.
The best sound for recording is completely flat and noiseless, save for your actors' voices. Be sure to test your actual actors before you do your final recordings. Often actors' voices—especially stage actors' voices—have an extreme dynamic range, which can cause a poorly adjusted recording setup to clip. It's much better to find this out in a trial recording than after your actors have gone home for the evening!
To find voice actors for your dialog tracks, first decide what your voices will sound like. Next, either hire professional voice talent (if your budget allows) or go scouting for amateur actors who are willing to work for the exposure. Someone who is familiar with local community or college theater programs can find you good talent quickly. If you have the good fortune to be able to audition your actors, listen to what they bring to a particular role. Often a well-trained actor gives you a more interesting reading than what you had in mind. You just have to be able to hear that different is better, not worse.
A point of some debate is whether to rehearse your cast before doing voice-over recordings. We believe a short rehearsal just before recording is beneficial because it helps actors get into the flow of the scene. Others argue, however, that this rehearsal reduces the spontaneity of the recording. Regardless of whether you rehearse, finding a good director can help get the best out of your actors.
During the recording process itself, continue to listen to your actor(s). They will often come up with marvelous spur-of-the-moment line readings that can make a dull line into a memorable or humorous one. One of the best techniques for getting a range of line readings out of an actor is to have them read a line three or four times in a row with just a slight pause between readings. Often this repetition helps them loosen up and try readings that they wouldn't necessarily have tried had they had more time to rehearse. Again, the audio software available today makes creating a dialog track from many individual takes quite easy.
• Creating a Lip-Synched Animation 121
Figure 5.1: A cleanly modeled head, ready for rigging and animation
regularly spaced isoparms or facets that avoid bunching up, especially in areas where a lot of activity will take place (such as the cheeks or mouth area; see Figure 5.1).
In addition, consider how the virtual muscles under your charac-ter's face will pull the skin and lips to create sounds. This task may be easy if your character is extremely simple, but for most models, you'll need to think about how their mouths move. Most often people refer to human anatomy when cre-ating mouth shapes, because most animated talking creatures—
human or not—move their lips like people do. Because most animals don't have the range of motion in their lips necessary for speech, anthropomorphizing their mouths is necessary for convincing mouth animation.
The human mouth is made up of a number of muscles laid out In concentric rings around the lips, allowing us our large range of motion and expression In this area.
The most straightforward way to model this muscle structure is to model the mouth area as a series of rlngs moving away from the lips. When you then pull on faces or control vertices, the mouth area behaves as if there were a more-or-less circular muscle structure beneath it.
Rigging for Lip-Synching
Once your model is built, it's time to set it up for animation. You can rig mouth animation in a couple of ways:
• You can manipulate the skeletal bones inside the mouth area.
• You can directly manipulate the control vertices of the model's surface.
Again, depending on the needs of your animation, you might want to use either or both of these methods. For simple, stylized, or "soft" characters, such as Monsieur Cinnamon that
Creating a Lip-Synched Animation 123
we'll work with later in this chapter, adjusting the surface of the mouth to get the appropriate shapes works well. However, real human mouths depend on a jawbone that rotates below the skull to achieve the major mouth positioning (the lip muscles fine-tuning this motion). As the jaw rotates—an effect difficult to notice if only the surface of the model is moving. In such cases, using bones or, more often, a com-bination of bones and blend shapes is a better solution.
Using bones to rig the mouth can be a simple or complex process, depending on the effect you're trying to achieve. In the sim-plest case, you can use one bone coming from the head joint, which acts as a jaw for the character's mouth, as in Figure 5.2. You must then skin this bone to the jaw area
and weight it properly so that as the bone moves, the lower jaw area reacts accordingly.
Figure 5.2: A single jawbone extends from a head bone, allowing the jaw area of the head to rotate similarly to a real jaw.
Usually a smooth bind is best for skinning the jaw area. You then need to manipulate the weights so that the lower jaw is completely affected by the jawbone while the upper jaw is completely unaffected. Fading the influence of the lower jawbone in the lower cheek area produces a soft transition from the low jaw to the rest of the head.
In more complex cases, you can build an entire "muscular" system in spiderlike fashion out of bones surrounding the lip area. As each "leg" pulls on an area of the lip, the affected skin of the model is distorted along with the bone. Although this complex bone structure can create amazingly subtle effects, it is fairly complex to set up and use and, with the advent of blend shapes, is not used as frequently as it once was.
To rig a character for blend shapes, you make multiple copies of the default mouth area—which is modeled in an expressionless neutral pose—and deform the vertices of the mouth to create various shapes, which become the target shapes that the neutral head will blend to when you animate the mouth. Rigging for blend shapes, then, is creating a library of mouth shapes that you can select, either by themselves or in combination, to create the final mouth shape at each moment of the final animation. For simple characters, the mouth library can be fairly small. Smile, frown, Ah (open), M (closed), and E (half open and stretched)
1. Select each of your target models, and then select your default model.
2. In the Animation menu, choose Deform Create Blend Shape.
3. Open the Blend Shape window (choose Window Animation Editors to see your blend shape controls, as in Figure 5.3.
Blend Shape)
To create mouth shapes, simply move one or more sliders up and down to create the shape you want for the current frame.
Once you create your blend shapes, you will probably want to hide the target mod-els in order to clean up your work area.
One useful aspect of the blend shape method in Maya is that you can use base blend shape targets to create "meta" blend shape targets.
For example, say you want to create the mouth shape for whistling, and you already have target models for closed pursed lips and for the O sound. Rather than create a new target model from scratch for the whistle, you can use the Blend Shape window to combine the O and pursed mouths and then save this new shape as a blend shape target by clicking the Add button in the Blend Shape window.
In Figure 5.4, the third blend shape (whistle) was created from a combination of O and pursed shapes given by the slider positions. Creating blend shapes from other blend shapes is a powerful time-saver dur-ing the riggdur-ing process. You can create five to ten basic mouth shapes and then produce the actual phonemes (Ah, O, E, M, K, and so on) by combining these basic mouth shapes.
Figure 5.4: The Blend Shape window shows the whistle mouth shape cre-ated by combining the O and pursed mouth shapes.
Creating Vocal Clips Using the Trax Editor
With your mouth shapes stored as blend shapes, you can load your dialog track and start keyframing your dialog right away. (To do so, move to a frame, move sliders to get an appro-priate shape, and click the Key or Key All button to set keys on the blend shape for that
Creating a Lip-Synched Animation 1 25
frame.) Although this method may be fine if you only need to lip-synch a few seconds of speech, a much better method for longer stretches of dialog is to use Maya's character and Trax Editor features to create clips for various words and mouth shapes.
To load sound Into your Maya scene, save your dialog track as a . w a v file, choose Flle Import, and browse to the sound file. Once the file is imported, you can see the shape of the sound file in the Timeline by RM clicking the Time-line and choosing Sound <name of sound> from the shortcut menu.
We will look at the Trax Editor more in detail in the first hands-on example later in this chapter, but the general method is to create a character (or subcharacter) for the mouth. The blend shape node is to be added to this character with the envelope and shape names selected in the Channel box and with the From Channel Box option selected in the Create Character options. This last step ensures that only these attributes will be keyed with the character, removing unnecessary keys during animation.
Once the blend shape is a character, take the script and start creating the words the character speaks to form a library of character clips. Because the Trax Editor allows for scal-ing of words, you do not need, at this point, to match the words with any given timscal-ing; so in general you select a standard length (5 or 10
frames) and make all words last that long by default. During the actual synching process, you can adjust this timing, as well as the size of the mouth for the word, to fit the way the word is actually spoken.
To create a clip, simply keyframe the series of mouth shapes for any given word or reaction, choose Animate Create Clip and give the clip the name of the word you just created, as shown in Figure 5.5. Once you animate an oft-used word for your char-acter, you can save it as a clip and then load it for future use.
Figure 5.5: Creating a clip for the word
"goodbye"
It is a good idea to create a neutral mouth shape and "breath" shape in addition to your words. You'll use the neutral mouth shape when the character is resting between sentences, and you'll use the "breath" shape when the character is getting ready to speak again.
If you have lots of dialog or more than one character speaking in your piece, two dis-tinct advantages will accrue from this method in addition to the ease of placing words where they need to be in a more intuitive manner (see the next section):
• Any repeated words ("the" is often repeated multiple times, for example) have to be keyframed only once, because you can reuse the same source clip as many times as needed.
• You can easily share a library of words between characters. As long as they have the same blend shape targets in the same order — even if the blend shapes look different
The sound file is read into the computer, and your library of words is completed, so now it's time to lay out the actual words in synch with your sound. Use the middle mouse button to scrub the Timeline to hear your dialog track, choose where each word comes, and drag that word from the Visor window onto your Trax Editor window.
Words are stored in the Visor under Animation: Character Clips and Poses:
Clips.
When the words are basically in place, select each word (double-click it in the Trax Editor) and use the Channel box to adjust its length and scale so that it better fits the spoken word. If you decide a particular word is not good enough, you can open the word's
keyframes in the Graph Editor (choose View Graph Anim Curves in the Trax Editor menu) and adjust them, as in Figure 5.6.
After laying in, scaling, and weighting your words, add a blend between them if you like (select two clips and RM choose Blend Clips) and test your animation using a quick playblast! All the work on the front end creating character clips pays off here: the animation process itself is much faster than keyframing each word in place as you go. Figure 5.7 shows the Trax Editor and clips for Figure 5.6: Adjusting the animation curves for a word in
the Graph Editor a famous sentence.
Figure 5.7: Creating a sentence using the Trax Editor
Creating a Lip-Synched Animation 1 27
In general, we have found that moving a character's lip-synched animation one or two frames before the actual sound for that mouth shape starts helps sell the reality of the synching. For some reason, matching sounds exactly to the mouth shape makes the visual mouth shapes seem to lag behind the audio track, which is disturb-ing. Because it's easy to move clips and fine-tune timing with the Trax Editor, experi-menting with timing is a straightforward task.
Finishing the Lip-Synch Animation
In the last stage of the animation process, basic technique gives way to art and storytelling.
All the extra work put in here translates into more compelling and believable characters who sell their reality and get the audience to care about the story.
Fine-Tuning and Adding Personality to the Vocal Track
Although the basic lip-synch work is done now, there is always room for fine-tuning and
Although the basic lip-synch work is done now, there is always room for fine-tuning and