Media Parts

As explained in section 3.1.1 (see also example in fig. 3.8) the inner structure of media components is important in interactive applications. It must be possible to specify inner parts of a media component if they should be accessible for other parts of the application, e.g. to associate them with some event handling code or to control their behavior by program code. For this purpose the modeling language

andreas' car : Car tom 's car : Car susan's car : Car Car : Horn : Horn : Horn Horn {unique}

(a) Keywordunique: All domain objects are represented by the same artifact.

indianapolis : Track : Indianapolis m onaco : Track TrackAnim ation m onza : Track : Monaco : Monza Track {one-to-one}

(b) Keywordone-to-one: Each domain objects is represented by an artifact of its own.

andreas' car : Car tom 's car : Car susan's car : Car

CarAnim ation

: Porsche : Porsche : Ferrari

Car

{dynamic}

(c) Keyworddynamic: No a priori information about the assignment of artifacts to domain ob- jects.

Figure 5.11: Examples for Keywords for Media Representations: The left hand side shows the MML example models, the right hand side corresponding example situations at runtime.

should support to define the inner structure in an abstract and simple way. It is not the intention of MML to define complete media objects in the models. Rather, in the viewpoint of MML creative media design should be performed using the established authoring tools for the respective media types. Thus, the definition in the MML model should only define an agreement on abstract level between the media designers and the software developers.

A Media Component can consist of different kinds of parts depending on the media type. Syn- thesized media usually consists of primitives having some kind of semantics, e.g. shapes and lines inside of vector graphics. Based on the primitives, more abstract parts can be identified, e.g. a wheel within a graphic representing a car. In contrast, media captured by sensors can only be decomposed by its natural dimensions, e.g. a specific region in a bitmap image is identified by its spatial x- and y-coordinates. Temporal media always has a temporal dimension which can be used for its decompo- sition, e.g. specific points of time in a video.

3D animation – as synthesized media type with the largest number of dimensions – has the most complex inner structure. Relevant types of inner parts can be identified based on an existing abstract

modeling language for 3D applications calledSSIML[Vitzthum and Pleuß05]. SSIML specifies the

concepts common in 3D standard formats (likeVRML[ISO97a] andX3D[ISO04]) and 3D authoring

Transformation2D Camera3D Light3D Transformation3D

Object3D

Viewpoint3D SubAnimation2D

ImageRegion SubGraphics TextPassage Animation Animation2D Animation2DPart 0..1 0..n +owner 0..1 +part0..n Animation3D Animation3DPart 0..1 0..n +owner 0..1 +part0..n VideoRegion AudioPart Audio 0..n 0..1 +part0..n +owner 0..1 VideoPart Video 0..n 0..1 +part0..n +owner 0..1 DiscreteMedia Image ImagePart 0..1 0..n +owner 0..1 +part0..n Graphics GraphicsPart 0..1 0..n +owner 0..1 +part0..n Text TextPart 0..1 0..n +owner 0..1 +part0..n TemporalMedia CuePoint 1 0..n +owner 1 +cuePoint 0..n MediaPart MediaElement AudioChannel

Figure 5.12: Metamodel for Media Parts.

Transform ation2D Transform ation3D SubAnim ation2D

Anim ation2D

Im ageRegion SubGraphics

View point3D Anim ation3D

3DObject Cam era3D

Graphics

Video Audio Im age Text

Light3D

Figure 5.13: Icons for MML Media Components and Media Parts (some Media Parts do not have an icon representation yet).

Reality applications [Vitzthum05, Vitzthum06, Vitzthum and Hussmann06].

In the 3D domain the structure of a 3D scene (i.e. the virtual world) is often denoted by a scene graph. Implementation formats usually use the same kind of hierarchical structure. A scene graph is a directed acyclic graph (DAG) containing the content of the scenes – geometric objects, lights, view- points, etc. – as its nodes. Positioning and orientation of objects is usually defined by transformation nodes. A transformation is applied to all child nodes. This enables the positioning of groups of objects relative to other objects. For example, the wheels of a car are positioned by a transformation relative to the car’s body and the whole car is positioned within the whole 3D scene by another transformation. The resulting hierarchical structure is very similar to a tree structure. However, it is possible that a node has several parents to enable reuse of objects for increased efficiency of the implementation. For instance, the four wheels of a car can be implemented using four references to the same wheel object and positioning them at the four sides of the car by four transformations. Also, in many 3D formats, only some kinds of nodes – e.g. transformation nodes and group nodes – can have children while the others may only act as leaf nodes in the scene graph.

MML abstracts from these concepts supporting a simplified tree structure for inner parts of media components. Any kind of node may have children, independently of whether it is further decomposed

internally using transformation nodes or group nodes. Also the structure is always a simple tree structure independently of whether it is implemented internally by reusing a node multiple times for efficiency reasons. The nodes are calledMedia Parts, reusing concepts from Parts in UML Composite Structure Diagrams which describe e.g. the inner structure of UML components (see e.g. [Hitz et al.05], chap. 3.5). A Media Part represents a specific structural part of a media component. For example for 3D animations the possible Media Parts derived from SSIML areObject3Dwhich refers to an arbitrary complex geometrical object,Transformation3D,Camera3D,Viewpoint3D, andLight3D3. Only those Media Parts are explicitly modeled in MML which are relevant for other parts of the application, e.g. as they should be manipulated by the application logic. Other Media Parts which are only required for the concrete realization of the Media component are omitted. A parent-child relationship between two nodes specifies that the child node’s position is relative to it’s parent node. An implementation for instance in VRML requires additional transformations between the elements which are though not modeled explicitly in MML.

The metamodel in figure 5.12 shows the different kinds of Media Parts available in MML for the different Media Types. Each temporal media can be structured byCue Pointswhich are abstract points of time within a temporal Media Component’s timeline. A Cue Point can represent for instance the beginning of a scene in a video or a specific point of time in an audio. The Cue Points are abstract as well, i.e. they are described by a meaningful name while their concrete time values needs not to be specified until the final implementation.

2D animations adopt those elements from 3D animations which make sense in 2D space, i.e. objects (SubAnimation2D) and transformations (Transformation2D). Audio can be decomposed in dif- ferent channels which represents on abstract level any decomposition beside CuePoints, for instance a stereo channel or a MIDI channel. Video can be decomposed into regions like an image. Image,

Graphics and Text can be decomposed in the spatial dimension only. AnImageRegion represents

a spatial area within an image. ASubGraphicsrepresents a part within a graphic. A TextPassage

represents a part within a text.

In addition, some types of Media Components can contain other Media Components. For example, a video can contain sound, graphics can contain bitmaps, and animations often can contain bitmaps, sound, or even videos. Such containment can be modeled by references to other Media Components as shown in the next section 5.2.8.

For the Media Parts custom icons have been developed based on user tests as described in sec- tion 5.1.4. They are shown in figure 5.13 A few Media Parts currently have not been considered yet:

Channel,VideoRegion,TextPassage, andCuePoint.