Remarks on Compositionality

The infinity class of the parameter “resolution” is only one aspect of pictorial syntax. It corresponds roughly to the level of linguistics dealing merely with the range of letters; the notorious pixel usually comes into the beholder’s (or creator’s) focus of attention only when the presentation quality of a picture is low. There are other parts of which a

picture is viewed as composed of and which could be rearranged to form another image – thus forming the basis of a morphology of pictures, so to speak. Furthermore, several images can be arranged into pictorial signs of higher order, mimicking the arrangement of words into sentences and texts.

4.2.2.1 Composition of One Picture: Pictorial “Morphology”

The linguistic branch of morphology investigates essentially how words are build from “morphemes” – minimal meaning-contributing particles, like the postfix ‘-ed’ in English, the prefix ‘pré-’ in French, or the stem ‘-wend-’ in German. Mostly, such morphological elements are identified and arranged into classes by means of a rule of interchange: some words beginning with ‘pré-’ can be transformed into other words of French by just changing the prefix to ‘re-’, ‘con-’, ‘de-’ etc. The morphemes may best be viewed as the vehicles of unsaturated partial signs acts without a pragmatic function of their own (unlike predication or nomination) that modify in a more or less specific way the meaning of the whole.

Are analogous “pixemes” relevant for the generic data type »image« or any of its more specific derivatives? It is important to note here that semantic arguments may be used to find such pixemes, but that their description must avoid any semantic “contami- nation”. The characterization of pictures as perceptoid signs of the visual sense modali- ties already suggests that visual Gestalt entities may serve exactly that purpose: closed areas, grouped by neighborhood and similarity (e.g., of coloration); connected lines; some visual pattern inducing directional “energy” (diagonals, arrow shapes).35

On a more formalized level, we may consider geometric entities – lines, curves, dots, areas, etc. as the basic morphological components of pictures. Indeed, such entities are also the standard elements offered by painter programs (like Corel Draw).

Let us concentrate for the moment on lines or strokes. A stroke may be defined prag- matically by the painter’s movement or semantically as the contour line of an object. Beside the potential graphical meaning of a line or the stylistic indications associated with its particular make (not to mention any other expressive or appellative function of dynamism associated to it on the level of pragmatics), there are several dimensions in which a line – just being taken as a line – can vary: most prominently in the course or path it takes. But there are other ranges: is it a continuous line, or dashed, or dotted? Does it consist of strokes of one kind or another? How thick is it? Does its thickness change over its course or not? Is there an internal fine structure to the strokes? Assum- ing a corresponding data type »pictorial line« separate from »image« is, thus, certainly a wise idea.

An extensive treatment of such a data type and its possible implementations has been performed in the context of non-photorealistic rendering (NPR) , a sub field of computer graphics. While Figure 47 exemplifies several types of digital “hairy brush strokes” that have been generated – quite expensively in computational resources – by simulating a brush with several individual bristles applied with changing pressure, Figure 48 shows examples of lines resulting the application of a “style function” to the “skeletal path” of the stroke. Both constituents of the latter case are defined by means of parametric curves: the style describes how a given path (as the core of the line) is to be perturbed in order to result in a corresponding pixeme. Style and path can be viewed as independent ranges determined in each particular picture by semantic and / or pragmatic aspects.

35

The rules of composition of strokes or other pixemes into a picture can be investi- gated by means of the tools of formal languages. Every computer scientist knows by heart the structures called formal grammars – or CHOMSKY grammars – since those are

the major instrument for defining and classifying linear structures like programming languages. Formal grammars based on replacement rules that lead to two-dimensional “pictorial” structures have been investigated essentially under the name of L-systems.36

The expressions generated by an L-system can be interpreted as orders to place sub- structures, and to move or turn in-between. A fairly simple example is defined by the following replacement rule:

P P [ – P ] P [ + P ] P

Interpret “P” as “place a pixeme and move a bit forward”, “+” by “turn right”, “–“ by “turn left”, and the square brackets as stack operations that allow us to return to that point after the bracketed sub expression has been dealt with. The plant-like structures in Figure 49 have been generated by this rule. Obviously the pixemes themselves are not really relevant for L-systems and their relatives, since these grammars basically deal with arrangements and groupings of abstract entities that may or may not be interpreted in a pictorial sense.

For a more extensive approach to pictorial morphology, a data type for pixemes can best be derived from a calculus for geometry. That any pixeme must be a geometric en- tity seems almost too trivial to be mentioned. That inversely any entity in flat geometry – apart from non-extended points – may also be a candidate for a pixeme is at least a good guess. Taking the common Euclidean formalization of geometry leads however to the “unpleasant” consequence that the most basic pixemes must be non-extended points – a concept highly abstracted from experience, that is. Non-standard approaches to ge- ometry like mereogeometries37

here offer an interesting way out. The traditional calculus

36

The “L” stands for “Lindenmayer”, as the botanist ARISTID LINDENMAYER started to use a correspond- ing formal language for describing plants; cf. [PRUSINKIEWICZ & LINDENMAYER 1990].

37

cf. [WHITEHEAD 1929], [LEONARD & GOODMAN 1940] [CLARKE1981], [AURNAGUE & VIEU 1993],

[ASHER & VIEU 1995], [SMITH 1996], and [BORGO & MASOLO 2001]. Figure 47: Enlarged Fine

Structure of Computer-

of geometry develops around the fundamental concept of a zero-dimensional point. In contrast, mereogeometries are based on extended regions as the most elementary enti- ties, which may or may not have (distinguishable) proper parts. The regions are often called “individuals”. Individuals do not have immediate attributes of form or position: only the relations to other individuals, in particular parts, determine form and (relative) location.

An individual may quite well be thought of as a visual Gestalt – thus following the principle of perception psychology of the Gestalt school: one has to consider the per- ceived whole first and introduce the concepts for perceptual atoms as instruments of the explanations of the former, not the other way round. We do not see sets of zero- dimensional points but regional Gestalts. The abstract notion of a spatial entity without extension is secondarily constructed in order to explain some aspects of experienced space, but leads on the other side to severe difficulties as the discussion on infinite reso- lution has shown. The thesis is therefore that the constructs of an individual calculus for the two-dimensional mereogeometry are excellent candidates for a general and exhaus- tive discussion of pixemes.

The syllable “mereo” indicates that part-whole relations form a central aspect of mereogeometries: more precisely, the fundamental data type “individual” in mereo- geometries is primarily characterized by the reflexive and transitive relation of being part of between two of its instances.38

In the words of B. SMITH [1996, 290]: “We adopt

as mereological primitive the relation of parthood or constituency. We say x is a part of y, and write ‘P(x, y)’, when x is any sort of part of y, including an improper part (so P(x, y) will be consistent with x’s being identical to y).” With this relation, more com- plex relations and entities can be formally defined, especially those with topological in- terpretations, like boundaries and interiors. Two individuals are, for example, defined to “overlap”, if there exists a third individual being simultaneously part of both of them. In particular, the concept of a minimal region usually called a “point” (“Pt”) – we may well use “pixel” instead – can be introduced: Pt (x) = def ∀ y (P(y, x) ⇒ y = x). That is: a

38

Some mereotopologies and mereogeometries are based on other relations; for example, AURNAGUE and VIEU [1993, 403] use the symmetric, reflexive binary relation C (for “being connected with”), from

which (among others) the relation used in the text, P (part of/inclusion), is derived: P (x, y) ≡def ∀ z (C(z,

x) ⇒ C(z, y))

Figure 49: Two Example Pictures Generated by (Bracketed) L-Systems, and the Graphical Interpreta- tion for the Rule for the Left Example

point in this sense is a region that has no proper parts (or rather, a region where no proper parts are considered).3940

When the concept »point« is introduced in the data structure as mentioned above, there is no need in any concrete instance for using infinitely many point instances: only the “relevant” points must be instantiated. This also means that there is always a finite resolution. N. ASHER & L. VIEU [1995] propose a formal mechanism called “micro-

scopization” covering a kind of zooming operation by means of a modal extension to their calculus. What is a “point” on one level may be a compound of regions with sev- eral points on a microscopized level. While Euclidean geometry first introduces the con- tinuous range of infinitely many coordinates determining potential points some of which are then chosen to be relevant (still an infinite number in any practical relevant in- stance), mereogeometry starts with a (usually finite) number of relevant individuals (re- gions) we can think of being given in perception. That is, we may indeed assume that the principles governing visual perception determine the regions that are syntactically relevant, hence leading only to the essential “points”.

Mereogeometries are a formal way to deal with geometry in a manner more closely related to visual perception than traditional point geometry. If we accept the view that the central data type of a two-dimensional mereogeometry determines what is a pixeme – namely any connected sub system of individuals, then there is indeed no finite number of possible pixemes – a clear difference to verbal sign systems with their strictly limited number of morphemes. However, any pixeme can be described and dealt with in a unique and generatable manner in the calculus in a finite number of steps: pixemes can be combined to form pixemes of a higher order – until every visually separable Gestalt of a picture is covered.

4.2.2.2 Compositions With Pictures: Pictorial “Text Grammars”

Considering compositions of (or with) pictures to form signs of higher order brings us first back to the compositions of pixemes as by L-systems: did we not in fact arrange pictures of strokes by means of a formula derived by an L-system? Indeed, the arrangement could be, as in that case, one performed in the picture plane as well as one in our usual three-dimensional environment, or even in the separate dimension of time. While such formal systems may be also quite useful for describing part-whole relations in the sense intended here, the two forms of compositions – within one picture, and with several pictures – must not be confounded: pixemes are never used for autonomous signs, while the composition with pictures depends on the status of the component pictures of being quite well useable as independent signs. The linguistic counterpart to the latter is indeed text grammars dealing with the composition of texts from sentences, which can be seen as being already the sign vehicle for a complete sign act, a status not ascribable to single words or phrases. An abstraction of pure syntactic classes analogous

39

There are other ways to introduce a similar notion of a point in other mereotopologies/mereogeometries, some of them leading even to the non-extended Euclidean version. The nub here is that points are logi- cally secondary entities.

40

So far, only the definition for a mereotopology has been sketched: by adding, for example, relations of relative distance between points (point A is closer to point B than to C), and of relative direction be- tween points (point A is between points B and C), the data structure can be extended to a geometry basi- cally of the same expressive power as Euclidean geometry. As an advantage over and above not chosing a highly abstracted starting point but a more perception-like entity, topological and metric aspect can then be dealt with in relative separation.

to »verb«, »noun«, »adjective« is not available, and may, in the light of the considerations of Chapter 3, never be in general.

Only for very restricted domains of use, an association between grammatical catego- ries and pictorial compositions might be possible: TH. STROTHOTTE [1989, Sect. 3.1],

for example, offers a syntactic schema in the context of maintenance instructions. Based on a formalized verbal description, an arrangement or sequence of pictures is to be gen- erated. To that purpose, the noun phrases in question are schematically associated with elementary images of corresponding objects. The verbal groups considered correspond roughly to temporal arrangements of the pictorial compositions linked to the noun phrases that are bound together by those verbal groups. This includes the appearance of a “user’s hand” for imperative moods, or of “think bubbles” for subjunctive moods. Al- though it is quite functional for its definite purpose, the rather small fraction of syntactic categories used indicates the limitation of such an approach. What about adjectives, ad- verbs or conjunctions, for example?

Of course, texts form just one-dimensional compositions: a comparable composite sign with pictures is given by (simple) comic strips, and also by films. While the former is clearly organized in several individual pictures by the “guts” between them, moving pictures do not offer a similar distinction of autonomous pictorial entities as easily. However, taking cuts or dissolves between (continuous) scenes as the temporal equiva- lent of inter-panel space in comics leaves us with exactly those scenes as pictorially autonomous signs, a solution not too implausible indeed (cf. [SCHWAN 2001]).

Comics do not only come in a linear fashion; the more advanced specimens use quite complicated forms of layout, taking into account not only the two-dimensional area of possible placements of one page, but the options of using either two opposing pages with a marked jump of view, or the even further separation of a page to be turned. This is indeed not much different from general layouting, which mostly deals with texts and possibly some pictures or other elements in between – pure text layout and comics lay- out form just the two extremes. A type of pictorial composition in 2D, which is particu- larly interesting here, is given when pictures are shown within another picture, i.e., not just as a morpheme (like the stroke pixeme or even a texture map) but as an autonomous picture on top of the other picture plane. A typical example is the use of an enlargement inset framed by means of a pictorial magnifying glass.

There are compositions of pictures into high-order signs even in 3D space: think of an exhibition. The arrangement of the exhibits intends to establish correspondences and to allow the visitors to see more than just an unconnected set of pictures. For a computa- tional visualist, a comparable task may come into view when dealing with special VR presentation hardware like CAVEs forcing him or her to coordinate the placement of pictures in three dimensions.

We shall not go here into further detail of this particular aspect associated with the data type »image«. Computational approaches to text/discourse grammars become rather helpful in later sections when taking into account more than just syntactic consid- erations of layouting.