General OpenGL Implementation ···········································

The following are important elements that must be implemented in the OpenGL graphics pipeline, called I-GL in this thesis, which represents Implementation of OpenGL. According to their inner relationships, the introduction is from the top of commands to the bottom of the connection between software and hardware.

5.4.2.1 Commands and Orders

The I-GL provides the API of graphics pipeline for application developers with a set of commands. These commands can be used to specify a primitive, set the mode, or perform

Chapter 5 Integrating Mesa-OpenGL into FPGA-based ES 97

an operation. Commands can be called with arguments. These arguments are used to transfer data to the settings of variants, such as primitives and modes. The data are bound to the current context and are transferred to the process that the command executes at the moment when the command is called. The subsequent changes on the data will not have an influence on the current pipeline.

As the OpenGL graphics pipeline is a state machine, commands in the I-GL are always executed in an order that they are received although it can be delayed for a command to really take effect. For example, an enabled texture can be delayed to be put on the surface of an object until the object is drawn, but the enable operation has been performed at the point where the glEnable command is used. This delay relies on the programming inside the I-GL and is a meaningful strategy for programming that allows all the similar operations to be done collectively and enhances the performance of the I-GL.

In general, the relationship between commands used in an application program and the I-GL is client-server. For a simple platform with a small amount of memory and CPU resources and single-threaded programming, such as small-footprint ESs, the relation between an application program and I-GL can be treaded as a caller and a set of called procedures. The application program (the client or caller) calls commands, and these commands are interpreted and processed by the I-GL (the server or procedure service). That is, the application program should issue commands to require the I-GL to create a context for one drawing task and connect this context to the task at the initialisation stage. If the program is not connected to a context with a complete initialisation in advance, calling commands can cause undetermined behaviour. A server can manage several contexts concurrently, each of which is corresponding to the current pipeline state for a drawing task. A client can choose one of the contexts to be connected.

5.4.2.2 Primitives and Modes

In the I-GL, a primitive can be a point, line segment, or triangle. A primitive can be drawn in several selectable modes. Each mode is independent. The setting of one mode does not influence others, but their effects may have a synergy in the frame buffer. Primitives and modes are set with the arguments of commands.

In the I-GL, every primitive is an entirety and must be drawn completely before any subsequent one can influence the frame buffer. Also, the effect of one mode on the frame buffer must be complete before any subsequent mode setting can have such effect.

The I-GL does not provide a complete command for constructing complex geometric objects, however. To describe a complicated geometric object, the surface of the object must be broken down into patches that can be represented by the basic primitives. With a whole sequence of commands of describing primitives and some advanced I-GL mechanisms, the object can be modelled. That is, the I-GL only provides commands and

Chapter 5 Integrating Mesa-OpenGL into FPGA-based ES 98

mechanisms how complex geometric objects are rendered. It is the responsibility of application developers to realise and figure out how the complex objects are described.

Application developments should think about how to divide a complicated-shaped object into primitives and divide a complicated process into simple operations, and how to link them together in order to render an entire object in a feasible and effective way. They also must realise what the effect of a sequence of operations on the frame buffer is. They have to re-test it several times and make sure of it.

5.4.2.3 States

For the I-GL, each state should put its data into the specified location of the memory. The evaluation function should be launched after all the required data are put into their specific locations. The evaluation function does just its tasks including fetching the data from the assigned location, doing operations, and putting its result into the result’s specified location. All the locations in memory have been initialised to allocate to the context of a drawing task when an application program makes a requisition for a window initialisation. This initialisation is the starting of the drawing task.

In the I-GL, there are numerous states that can be divided into two groups, server states and client states. Most states are server ones. A connection from a client to a server needs both the client state and server state, and the transfer between them. An operation in the pipeline, which may be primitive specifying, mode setting, or data evaluating, is a server state. A command starts from a client state and then the processing is transferred to server states when the I-GL processes the command.

5.4.2.4 Controls

From the view of application developers, the commands and mechanisms of the I-GL provide the direct control over the fundamental operations of 2D and 3D graphics drawing. Inside the I-GL, this control can be treated as the states of the OpenGL state machines that drive a drawing task to be executed and completed step by step. This control consists of commands with parameters of primitive setting, view point specifying, transformation matrices, lighting and material equation coefficients, texture mapping, antialiasing methods, pixel update operators, and others. Therefore, the entire efficacy of the I-GL relies on two aspects, how well the I-GL is accomplished and how well the I-GL is used in the application programming.

5.4.2.5 Data and Orders

Geometric objects with different shapes and styles may be drawn in many primitives, such as points (including point sprites), separated line segments, line strips (connected line segments), line segment loops (connected line segments with met endpoints), separated triangles, triangle strips (triangle patches arranged in a long strip and stitched seamlessly

Chapter 5 Integrating Mesa-OpenGL into FPGA-based ES 99

together), and triangle fans (triangle patches arranged in fan and stitched seamlessly together). Primitives are defined as an array of one or more vertices with coordinates. A vertex can be the coordinates of a point. A point can be a single point, an endpoint of an edge, or a corner where two edges of a triangle or other polygon meet.

Besides the positional coordinates, data of a vertex consist of colours, normals, and texture coordinates. Each group of data with the same attribute is processed independently in the same way and order. There is an exception, however. If an object cannot fit into a viewing volume, some of its primitives, such as points, lines or triangles, must be clipped. Vertex data can be modified. In the line or triangle primitives, some new vertices can be inserted into primitives. The new associated data, such as colours, normals, and texture coordinates, must be computed or interpolated.

Typically, data of colours consist of four values, representing red, green, blue and alpha values, respectively. Each of them is initiated as one (with values within [0, 1.0]; (1.0, 1.0,

1.0, 1.0) means white and completely transparent). Data for normals have three values,

representing three coordinates of normal vectors. They are initialled as (0, 0, 1.0), represented a vector directing in z axis. Texture units have four values that represent texture coordinates of s, t, r, and q. For a 2D texture, the s and t coordinates are used, and the two others are not. [s, t, r, q] is initialised as [0, 0, 0, 1.0], defined as homogeneous coordinates.

Vertices of an object are organised in a vertex array. All the associated data are copied in the vertex array and the transformed data of the object are also storied in this vertex array. The transformed data consist of eye, clip, normalised device, and window coordinates, which will be detailed in Section 5.4.2.7. With the window coordinates, vertices of a primitive can be written in the frame buffer with the colours or the texture colours according to the texture coordinates.

5.4.2.6 Vertices

In the I-GL, a data structure is defined specifically for a vertex array. At the stage of the user’s inputting data of an object, the vertex data are placed in the user or client address space. Once a command to transmit the data is used, the vertex data are transferred to the I-GL or server address space.

Blocks of data in these arrays can be specified for vertex coordinates, normals, colours, point sizes, and one or more texture coordinate sets of different geometric primitives in the execution of an I-GL command. For vertex coordinates, the command of

glVertexPointer(size, type, offset, pointer) is used. For normals, it is glNormalPointer(type, offset, pointer). For colours and texture coordinates, they are glColorPointer(size, type, offset, pointer) and glTexCoordPointer(size, type, offset, pointer). All of them have the

Chapter 5 Integrating Mesa-OpenGL into FPGA-based ES 100

these arrays: size represents the number of components of the corresponding data stored in the array, type represents the data type of the values stored in the corresponding array,

offset is the number of bytes for a set of components of the corresponding data, and pointer means the pointer pointing to the location of the array of corresponding data in the

user or client address space. The pointer is the location in memory of the first component of the first element in the corresponding array. Some arguments are not present because they are constant for their data array. Size and type have their allowable values in OpenGL. For example, the acceptable values for type can be BYTE, UNSIGNED BYTE,

SHORT, FIXED, and FLOAT.

All of them can transfer the corresponding array from the user or client address space to the I-GL space. The data in arrays are stored sequentially. The array elements are stored sequentially even though the offset is set to zero. No matter how many values an array element consists of, from one to four values, a single vertex covers an array element. The values in each array element are stored in memory in a consecutive manner as well. The number of values of each array element defines the offset between two adjoining elements. In an array, the pointer to its (i + 1)st element is greater by one offset number of machine memory units (say unsigned bytes) than the ith element.

5.4.2.7 Coordinate Transformations

In the graphics pipeline, there are several coordinate transformations. Data of vertices, such as, vertex coordinates, normals, and texture coordinates have to be transformed before the coordinates are written in the frame buffer to produce an image.

In Figure 5.4, the general sequence of transformations applied to vertices is shown. All the matrixes are 4 X 4. All the coordinates are four-dimensional, including x, y, z, and w coordinates.

Figure 5.4 Coordinate Transformation Sequence

The transformation sequence in Figure 5.4 has an inherent relationship with the geometric pipeline in Figure 5.1 and Section 5.1.1. In the I-GL, the coordinate transformation sequence starts from the transformation from the vertex coordinates of the drawn object to eye coordinates by left multiplying with the model-view matrix. This corresponds to the transformation from the local coordinates to the world coordinates in

Eye Coordinates Clip Coordinates Normalized Coordinates Window Coordinates Vertex Coordinates Model-view Matrix Clipping Matrix Projective Matrix View-point Matrix Left Multiplication Matrix

Chapter 5 Integrating Mesa-OpenGL into FPGA-based ES 101

Section 5.1.1. The transformation from eye coordinates to clip coordinates is carried out by left multiplying eye coordinates with the clipping matrix. It is the function of the clipping module of Section 5.1.1.4. The transformation from clip coordinates to normalised coordinates is done by left multiplying clip coordinates with the projective matrix. The last transformation from normalised coordinates to window coordinates is done by left multiplying normalised coordinates with the view-point matrix. The last two transformations are the function of the projection module in Section 5.1.1.5. In window coordinates, x and y are for the location coordinates of two dimensions on a display screen, and z represents the depth of the object looked into the screen by the viewer.

As mentioned above, the data are put into the specific location of memory when a command is called, and the evaluation should start after all the required data are prepared. The matrix settings with commands of glMatrixMode and glOrtho can be scattered in different parts of the application program, which depends on the application. The order of the matrix settings can influence the effect of the object rendering. The evaluation of all the transformations in a complete sequence is really done at the final stage of the geometric pipeline just before the rasterisation in Figure 5.1, however.

5.4.2.8 Colours

The colour processing is also done before rasterisation. There are two sources of colours: one is the colours of a primitive; the other is the colours of a texture. Typically, colours have four components of red, green, blue and alpha. Following the OpenGL specification, the I-GL accepts each colour component with a value ranged in [0, 1.0]. The colour can be set by glColor command at the beginning of a drawing task.

On the other hand, there are many colour palettes in the LCD and screen space. They include three bytes in whole and eight bits for each red, green, and blue, two bytes in whole and five bits for red and blue but six bits for green, and others. In this project, the LCD device adopts three bytes in whole and eight bits for each red, green, and blue. As the input colours can be in a different format, the I-GL must do the conversions. As a result of limited precision, especially for the hardware platforms of simple ESs, some converted values cannot be represented exactly. Different implementations on different platforms can yield discrepancies in colour tones and shades.

5.4.2.9 Lighting

If lighting is disabled, the current colour is used in the subsequent drawing. If enabled, lighting can influence the colours of the drawn object. A light can be set ambient or position. The former produces a far ambient lighting effect; the latter yields a near spotlight effect. The material reflection can be set as diffuse, specular or shininess. With the material setting, the lighting effect can be evaluated with lighting equations and yield new colour values. The produced values of colour components must be clamped to the range

Chapter 5 Integrating Mesa-OpenGL into FPGA-based ES 102

of [0, 1.0].

The shade mode can also influence the colours of a primitive. If the shade mode is set to flat shade by the glShadeMode(GL_FLAT) command, all vertices of the primitive are to have the same colour. If the shade mode is set to smooth shade by the

glShadeMode(GL_SMOOTH) command, the colours of the primitive are produced by

smoothly interpolation. 5.4.2.10 Texture

In the I-GL, texture functions can insert a texture pattern in a spatial region that may be a section of line or a patch of surface. A texture source can be an image described as an array of colours in one, two, or three dimensional coordinates.

Take a 2D texture image as an example. The texture image consists of a sequence of groups of values. Each group represents the set of values of red, green, blue and alpha components. The first group is the colour value at the lower left corner of the texture image. Subsequent groups are arranged firstly in width and secondly in height. In a width row, the next colour with the coordinates of (i, j+1) is located just after its previous one with the texture coordinates of (i, j). In a height column, the offset between the next colour with the coordinates of (i+1, j) and its previous one with the texture coordinates of (i, j) is the value of width of the texture image. The texture coordinates are ranged in [0, 1.0] in default. Therefore, the coordinates of a texture image have to be normalised in [0, 1.0] before texturing an object.

The I-GL provides a set of functions to perform the operations of texturing. The

glTexImage2D(tTarget, rv, pMode, tWidth, tHeight, bd, tFormat, tType, tArray) can be

used to load a user 2D image to the texture space that the I-GL allocates to the context of a drawing task while the glTexImage1D() is used for 1D texturing. In the command,

tTarget shows the dimensions of a texture, for instance, GL_TEXTURE_2D indicating a

2D texture and GL_TEXUTRE_1D representing a 1D texture; rv is an indicator for a reduction version of a larger texture array; pMode is the style of colour pattern of the texture image and usually adopts the GL_RGBA style with four values for red, green, blue and alpha; tWidth and tHeight are the numbers of columns and rows of the source image array; bd is an indicator for having a one-pixel border around the texture image; tFormat can specify a monotone of single red (GL_RED), green (GL_GREEN), or blue (GL_BLUE), or a colour in the order of blue, green and red (GL_BGR); tType is the data type for the colour values in the texture image array; tArray is the pointer pointed to the location of the source texture image array. The texture image is stored in a 4 X tWidth X tHeight array.

Since the tWidth and tHeight are the numbers of columns and rows of the source image array, they must be set to numbers that are powers of two. With a one-pixel border around the texture image, these numbers have to be the sum of two plus powers of two.

Chapter 5 Integrating Mesa-OpenGL into FPGA-based ES 103

When a texture has to be contiguously mapped several regions on a surface or a line, the boundaries of different texture copies may not align with the positions of pixel boundaries. The glTexParameter(tTarget, tAction, tManner) can be used to set operations that can be done on the texture. In this command, tTarget has the same meaning as that in the glTexImage; tAction has selectable values to set different operation actions that can be done on the texture; tManner defines which way is chosen to do the operation action specified by tAction. For example, given GL_TEXTURE_MAG_FILTER set to tAction and

GL_NEAREST to tManner, the operation action is to enlarge a section of the texture

image to fit a specified coordinate range in the region of a primitive with the nearest colours. Oppositely, the GL_TEXTURE_MIN_FILTER means to reduce the texture image. The tManner can get values of GL_NEAREST (for nearest) or GL_LINEAR (for linear), as mentioned in Section 5.1.2.2.

Sometimes coordinate values in texture space are outside the range from 0 to 1.0. The

glTexParameter can be used to make up for the patterns. In this way, tAction indicates a

coordinate on which the compensatory pattern will be done, such as s coordinate with

GL_TEXTURE_WRAP_S and t with GL_TEXTURE_WRAP_T; tManner has selectable

values, such as GL_REPEAT (replicating with the fractional part of a texture coordinate) and GL_CLAMP (clamping a texture coordinate to the unit interval).

The glTexEnvi command can control how texture elements are applied to an object.