Programming Language Support For. Virtual Environments. A Dissertation. Presented in Partial Fulfillment of the Requirements for the

(1)

Virtual Environments

A Dissertation

Presented in Partial Fulfillment of the Requirements for the

Degree of Doctor of Philosophy

with a

Major in Computer Science

in the

College of Graduate Studies

University of Idaho

By

Jafar M. Al-Gharaibeh

August 2012

Major Professor: Clinton Jeffery, Ph.D.

(2)

Authorization to Submit Dissertation

This dissertation of Jafar Mohammad Al-Gharaibeh, submitted for the degree of Doctor of Philosophy with a major in Computer Science and titled “Programming Language Support for Virtual Environments” has been reviewed in final form. Permission, as indicated by the signatures and dates given below, is now granted to submit final copies to the College of Graduate Studies for approval.

Major Professor: Date:

Dr. Clinton Jeffery

Committee member: Date:

Dr. Robert Heckendorn

Dr. Terence Soule

Dr. Fred Barlow

Department Administrator: Date:

Dr. Gregory W. Donohoe

Discipline’s College Dean: Date

Dr. Larry A. Stauffer

Final Approval and

Acceptance by the College of

Graduate Studies: Date:

(3)

Abstract

Developing 3D virtual environments requires an advanced level of programming expertise in a wide range of programming domains including 3D graphics, networking, user interfaces and audio programming. To compound the problem, virtual environments have strong real time performance requirements. The complexity of developing these kinds of applications comes from two sources: first, the requirements of the virtual environment itself, with its dynamics and size. The second is the programming language used in development, with its strengths and also the limitations it imposes. Unfortunately, most of the tools and libraries necessary for developing virtual worlds are available mainly with low level system programming languages such as C and C++. The complexity and the amount of code required in this family of languages contribute to the overall complexity of virtual world applications making the process of building such applications a challenging process. Because the gap between language and application domain is high, a language for writing virtual environments is needed. Very high level languages such as Python and Unicon, compared to languages such as C, offer very high level programming semantics, syntax, data structures, and rich APIs with built-in support covering a wide range of programming activities such as I/O. With these language characteristics, programs can be made significantly more compact and therefore less complex. However, these very high level languages lack features essential to developing virtual worlds, and more importantly, they fall short of high performance and scalability requirements of virtual environments. This dissertation presents a language/application co-design approach for software development of virtual world. Both the application and the programming language itself evolve over time to meet new requirements. This approach is used in the development of a collaborative virtual environment called CVE, and its implementation language, Unicon. The focus of the co-design is on language extensions for virtual environment development. These extensions include 3D graphics, concurrent programming and 3D interaction. The language is improved to address the complexities and requirements that arose at the application level.

This dissertation answers two main questions: 1) Is it possible to utilize a very high level language with a legacy virtual machine in virtual worlds development where performance is critical? 2) How and where does such a language need to be extended and modified both at the language level and in its implementation to meet the requirements of a multi-user virtual environment? The goal is to reduce the complexity and cost of developing virtual environments, enabling less experienced programmers to participate in developing such applications. The dissertation does not create a virtual world or a new programming language; instead it uses an existing language and complements it with very high level features. The main contribution of this dissertation is the novel design and integration of new features into a very high level goal-directed language. These features not only provide very high level support for virtual environments, but also meet the language design guidelines, maintain backward compatibility, and have very little impact on the syntax. The benefits of the new features are not specific to virtual worlds.

(4)

Curriculum Vita

Jafar Al-Gharaibeh

Department of Computer Science University of Idaho

Moscow, ID 83844 [email protected] Education

University of Idaho, Moscow, ID

Pursuing Ph.D. Degree in Computer Science

Yarmouk University, Irbid, Jordan Master in Computer Science - Feb 2004

Yarmouk University, Irbid, Jordan B.Sc. in Computer Science - June 2001

(5)

Acknowledgments

This dissertation is coming to a conclusion, and the work could not have been done without the assistance and support of many people. I would like to thank my advisor Dr. Clinton Jeffery for his dedication and encouragement throughout the years of this dissertation work. He set a great example for me in his level of involvement and willingness to provide help and guidance even outside work hours. Also I would like to thank the dear respected members of the committee, Dr. Robert Heckendorn, Dr. Terence Soule, and Dr. Fred Barlow, for taking the time to review this work and provide much appreciated feedback.

I would like to thank the Unicon Language community, especially Kostas Oikonomou, Phillip Thomas and Steve Wampler, for providing feedback and test analysis on various language extensions added to the Unicon programming language as part of this dissertation work.

Finally, I would like to thank the corporate and governmental sponsors of this work. This research was supported in part by a grant from the National Science Foundation under agreement number DUE-0402572. This work was also supported in part by AT&T Labs Research and the Specialized Information Services Division of the U.S. National Library of Medicine.

(6)

Dedication

I dedicate this work to my parents who supported my pursuit of education throughout the years. To my wife Hiba who provided comfort and support and daughter Rayanne who brought much joy and happiness.

(7)

List of Figures

Figure 1.1 The focus of this research on three features set in the Unicon language ... 10

Figure 1.2 The three major aspects of this research ... 11

Figure 2.1 Virtual reality: flight simulator [source: www.freeway.org] ... 15

Figure 2.2 Simplified representation of an Reality Virtuality (RV) Continuum [23] ... 15

Figure 2.3 The use of augmented reality in a sport show [source: www.totalfootballmadness.com] ... 16

Figure 2.4 A map of some islands of Second Life viewed from Second Life’s world map tool. ... 17

Figure 2.5 ActiveWorlds Gate, the place where new users first login to. ... 18

Figure 2.6 Modular game engine structure [30]. ... 19

Figure 2.7 Schematic of a game [29] ... 20

Figure 2.8 A scene with avatars generated using VRML [44] ... 25

Figure 2.9 A sequence of frames, the result of a program written in Haskell using Fran [52] ... 29

Figure 2.10 Selection viewing volume centered around the mouse cursor ... 31

Figure 3.1 A 3D window in Unicon keeps track of all the scene content using a display list ... 42

Figure 5.1 Events that are recognized by the Selection3D class and the mapping between these events and their corresponding object tables. ... 58

Figure 5.2 UML diagram for the classes used to manage/control 3D object selection ... 59

Figure 6.1 Threads messaging queues ... 70

Figure 6.2 Self modifying instruction protected by a mutex ... 77

Figure 6.3 Threads, private heaps and public heaps ... 79

Figure 6.4 Tracking the number of running threads in the system ... 80

Figure 6.5 A high level view of the dynamics of suspending/resuming threads for GC ... 82

Figure 7.1 Only one function is required to make an object selectable in Unicon, compared to many functions with C/OpenGL ... 87

(12)

updated with another image ... 89

Figure 7.3 Left: Original texture. Right: the result of drawing on the texture dynamically ... 90

Figure 7.4 Terrain generation and rendering in Second Life using height maps [secondlife.com] ... 93

Figure 7.5 A simple example of 3D terrain rendering in Unicon ... 93

Figure 7.6 Low-polygon model (Left) and high-polygon model (right) rendered in Unicon ... 95

Figure 7.7 3D Model rendering using arrays and lists ... 96

Figure 7.8 Rendering performance of non-animated 3d models using lists vs. arrays ... 98

Figure 7.9 Rendering performance of animated 3D models using lists vs. arrays ... 98

Figure 7.10 The effect of adding more models (static/animated) on frame rate when using arrays and lists ... 99

Figure 7.11 Speed up gains of static models and animated models when switching from lists to arrays ... 100

Figure 7.12 The effect of adding more threads in several programs ... 101

Figure 7.13 Heap size effect on the performance of garbage collection intensive program ... 102

Figure 7.14 Forcing threads with semi-full heaps to garbage collect ... 102

Figure 7.15 Matrix multiplication: each thread works on a slice independently from other threads ... 103

Figure 7.16 Quicksort: at each stage a new thread could be spawned to handle the other half of the list ... 104

Figure 7.17 Different programs and their response to increasing the number of threads ... 108

Figure 7.18 Thread utilization in each program given the number of the concurrent threads ... 109

Figure 7.19 Sequential performance of concurrent and non-concurrent versions of Unicon (concurrency overhead between parenthesis) ... 110

Figure 8.1 NPCs and other users in CVE. An NPC is marked with red ball above their heads ... 112

Figure 8.2 An NPC ID card ... 113

Figure 8.3 An example quest as it appears in a web page ... 115

(13)

Figure 8.6 PENQ NPC quest messages between the NPC, the server and the client ... 118

Figure 8.7 NPCs in CVE virtual world... 121

Figure 8.8 The effect the NPCs’ running modes on the server latency. ... 123

Figure 9.1 A screen shot of the CVE client ... 125

Figure 9.2 Starting a collaborative IDE session in CVE ... 126

Figure 9.3 Unicon 3D model viewer ... 129

Figure 9.4 An example 3D model with an animation of a walk cycle ... 130

Figure 9.5 A number of NPCs walking around in CVE ... 131

Figure 9.6 CVE performance using arrays and lists ... 132

Figure 9.7 The percentage of time spent in the function Refresh() ... 133

Figure 9.8 The time it takes to do one Refresh() ... 134

Figure 9.9 Feature evolution of 3D object selection in CVE: language-application co-design example .... 136

Figure 9.10 A window with a 3D spinning cube used as a source for a dynamic texture in CVE ... 137

Figure 9.11 A virtual computer uses a dynamic texture showing a 3D spinning cube in the CVE virtual world ... 138

Figure 9.12 A user in CVE watching two virtual computer screens sharing the same dynamic texture ... 139

Figure 9.13 The benefits of adding thread support to the CVE client ... 140

Figure 9.14 The percentage of execution time spent in Refresh() fucntion ... 142

(14)

List of Tables

Table 2.1 Applications of Behavior in Java3D [55] ... 32

Table 6.1 Thread's communication queues attributes ... 73

Table 6.2 Summary of the new communication operators ... 74

Table 7.1 Terrain rendering, traditional list vs. real array list ... 94

Table 7.2 Details of the models used in the experiment ... 95

Table 7.3 The configuration of the laptop where the tests were conducted... 95

Table 7.4 Low-polygon vs. high-polygon 3D models rendering using the two list formats In Unicon ... 96

Table 7.5 Rendering Static models (No animation) ... 97

Table 7.6 Rendering animated models ... 97

Table 7.7 Configuration of the test machine ... 108

Table 8.1 The major parts in an NPC profile file ... 113

Table 8.2 Summary of the most important NPC protocol messages ... 118

Table 8.3 Summary of quest commands ... 118

Table 9.1 Garbage collection duration and frequency and its impact on CVE with different heap size while having seven online NPCs. ... 143

(15)

(16)

(17)

(18)

1 Introduction

Virtual worlds have a tremendous impact on the world we live in today. People are spending more and more time in virtual worlds and social media spaces. Moreover, recent years have witnessed the emergence of huge online communities in games and virtual worlds. Millions of people populate these virtual worlds to play, engage in social activities, or do business. Many organizations and institutions use virtual worlds including games for a variety of purposes, such as recruiting and training soldiers in the army [1], or promoting interest in science and engineering [2]. Furthermore, the spectacular success of massively multi-player online role-playing games (MMORPGs or MMOs for short), both in term of number of multi-players and revenue [3], has led to a large amount of interest in educational multi-user virtual environments [4]. World of Warcraft (WoW) [5], and similar games have demonstrated both the mass appeal and the potential of this genre.

One way to build a new virtual world application is to extend and build on top of an existing one. Many virtual environments that are available online for users, such as SecondLife [6], ActiveWorlds [7], or Open Cobalt [8], give users the ability to customize the virtual space and add to it. This approach minimizes the software development cost, but still incurs other costs. In addition to the potential financial cost, users are constrained to the rules, activities, and limitations imposed by these virtual worlds, and content creators are limited to the tools provided by such worlds. For some applications, where domain requirements are substantially different from what is provided by such existing virtual worlds, using an existing virtual environment would require extensive modification that is not feasible for small teams or small projects. Adapting an existing gigantic codebase or using a game engine, in most cases, exceeds the technical skill of inexperienced programmers such as students. That was the case for the Collaborative Virtual Environment project (CVE) covered later in this dissertation.

Another obvious way of building a virtual world is to pick a suitable programming language and start developing from scratch. Usually this is a long and expensive process, defined by the requirements and the goals of the project, and greatly influenced by the programming language used.

In this dissertation, a co-design approach is undertaken to accomplish the task of developing a virtual world. The co-design principle is utilized in many fields such as hardware/software co-design [9]. In hardware/software co-design, the hardware and software are concurrently designed and developed to meet the system objectives [10]. It is popular in domains such as mobile and embedded systems [11] [12], compilers and high performance computing [13]. The popularity of hardware/software co-design suggests an under-utilized analog: language/application co-design. Programming languages are in fact much easier than hardware to co-design alongside the applications that run on them. In many cases this co-design occurs during the creation of a domain specific language, and there are conspicuous examples, such as operating

(19)

system/language co-design [14]. The objective of this dissertation is not to develop a language/application co-design methodology, but apply the co-design approach on the domain of multi-user virtual environments.

The Unicon programming language and the CVE virtual environment framework are on-going research projects at the University of Idaho. The author of this dissertation is involved in both projects. The dissertation facilitates virtual world development through language design, and also improves the Unicon language by means of language/application co-design. Any new feature or extension added to Unicon must conform to the language’s spirit, and also should be made as general as possible so that it can be used in other domains benefiting the entire Unicon language community.

Using this approach of language/application co-design, CVE and its implementation language Unicon, evolved together over the time of the project development. This approach extends the language virtual machine and the runtime system to serve as a game engine, instead of the more typical approach of embedding a scripting language on top of a more conventional game engine. The application makes use of the language’s advanced features, such as powerful data structures, very high level APIs, and concise code. When some new language feature is required by the CVE, or the performance of an existing function in the language does not meet the application requirements, new features are added and/or existing features are enhanced to meet the new requirements.

Extending a language with new features can be done through library bindings or a more integrated built-in approach. Binding provides a direct and usually one to one function mapping to an underlying library. Many languages for example provide bindings for OpenGL. Since OpenGL is implemented in C, most languages cannot call OpenGL functions directly; instead they go through a set of wrapper functions which is referred to as a binding. Built-in support on the other hand, may provide a different vocabulary, constructs, and features that are better integrated with the language and conform to its designs and features. This higher level API does not map directly to the underlying library. One function/feature typically maps to several underlying functions, while other features might not exist in the underlying library. Binding usually provides a more complete access to the underlying library, a more uniform coding style across languages, and better performance than built-in support. But built-in support provides a higher level API which usually translates to shorter and easier to write and maintain source code. In some cases it might also provide extensions that the original library lacks

The Unicon programming language used in this project is an object-oriented descendant of the Icon programming language [15] [16]. Icon integrates Prolog-like goal-direction and implicit backtracking within a conventional syntax and an imperative semantic core. Icon’s traditional domain is string and file processing, and the rapid development of experimental algorithms and data structures. Unicon is a superset that extends Icon along two dimensions: features such as classes and packages for larger-scale projects, and

(20)

extensive access to modern I/O capabilities such as graphics, networking, and databases. Unicon is an open source project available at unicon.org.

The CVE project discussed in this dissertation is hosted on Source Forge at cve.sourceforge.net. More than 15 students participated in its development. CVE is the driving force behind many new Unicon features and performance improvements. In many cases, iterative revision and development on both the language and the application was done to achieve the required functionality and performance sufficient for smooth animation in CVE. CVE was also extended as part of this dissertation to include computer-controlled non-player characters (NPCs), who serve as tutors and record keepers for users’ accomplishments. The design for the NPCs and their quests presented later in this dissertation is called: Portable Extensible Non-player character tutors and Quests (PENQ, pronounced “pink”) [17]. Both CVE and its NPCs are used to demonstrate and evaluate the use of the new language features and improvements.

1.1 Motivation and Goals

Building a collaborative virtual world is a huge task involving all kind of activities such as 3D graphics, user interactions, networking, concurrent programming, modeling and art. This makes developing virtual worlds a challenging process, especially because they tend to be large and complex systems requiring large teams. Virtual worlds are also performance critical which adds to the challenges facing developing these kinds of applications. A large part of the complexity of these applications comes from the programming languages such as C and C++ used to develop these virtual worlds.

Languages such as Unicon employ very high level semantics and features hiding a lot of implementation details and allowing developers to write much more compact code compared to languages such as C. The use of a very high level language to develop a virtual world would greatly reduce the amount of code needed, but very high level languages lack many features required by such application. This dissertation tackles the complexity of virtual world by using the very high level language Unicon, and faces the problem of most missing features by building them into the language. The challenge is to meet all of the virtual world’s (CVE) requirements using an interpreted language with a legacy virtual machine, and add new features with high level semantics to meet the language standard and make these features general language constructs that can be used by and benefit all kind of applications and not necessarily virtual worlds.

The CVE project described in this dissertation was motivated initially by a goal to support distance education in a virtual world. Enabling distant students to attend lectures and office hours, and do homework assignments and labs within the virtual environment, initial efforts focused on reproducing a local CS education environment, including 3D representations of two physical CS departments, avatars, and chat. The project expanded over time to include domain-specific behaviors and tools such as interactive

(21)

collaboration on common CS tasks of editing, compilation, execution and debugging, and later on, game like features.

Within the CVE project are many dissertation-worthy problems, but this dissertation focuses on aspects of the underlying language co-design. The main goal is to build an enabling technology and infrastructures that simplify the process of building a virtual world. This can be achieved by:

 First, using a very high level application programming language that takes a lot less time and code to develop applications than a mainstream systems programming language.

 Second, integrating new features essential to developing virtual worlds into the language.

 Third, addressing any application performance issues mainly in the implementation of the language itself with the least intervention from programmers, or provide the programmers with explicit mechanisms and features that allow them to handle such issues.

Building virtual worlds using an open source very high level programming language opens the door for new capabilities and features that can be built into the language. An open source language is easy to experiment with, and in a very high level language, new features can employ expressive semantics and powerful data structures. The programmer does not have to learn enormous libraries to do tasks such as 3D graphics or networking. This approach hides many of the implementation details in many situations, removing the burden of such low-level details from the programmer and allowing the language to take care of them. Such low level details are usually one of the biggest factors hindering inexperienced programmers from undertaking virtual worlds implementations. With this in mind a new question arises: what features should a programming language have in order to make the task of building a virtual world something feasible even with a limited budget?

CVE is used as a testing platform to validate new language features and also to identify missing crucial features. To further help evaluate new features, a framework was developed that allows easy construction of computer controlled non-player characters (NPCs). NPCs in this framework can be run as part of another process, integrated as threads in the virtual world server for example, or can be run as separate clients giving them a great amount of freedom and independence from the host virtual world, allowing an NPC to be created outside the virtual world and deployed with minimum support from the virtual world itself. These NPCs are tested in an educational setting (the CVE’s main purpose) as an experimental domain to create NPC tutors that make the learning process as fun as playing a game.

(22)

1.2 Research Hypothesis

The research presented in this dissertation is being undertaken to test two hypotheses. The first hypothesis is: a very high level interpreted language can be used to build prototype virtual worlds, which are performance critical. This hypothesis aims to answer the first question of the dissertation. Several standalone tests are used to measure the amount of code needed and also the performance of certain language features essentials for virtual worlds development. In addition to that, experiments are used in CVE to do performance analysis to test if the language meets CVE’s requirement on both client and server side.

The second hypothesis is that features required by games and virtual worlds development can be built into a legacy virtual machine, hiding the implementation details and providing a very high level API. This hypothesis aims to answer the second question in the dissertation. There are three points to cover regarding this hypothesis:

 What are the required features?

o These new features are identified by the requirements of new functionalities needed in CVE

 Can these be built into the language?

o This will be demonstrated by successful integration of the new features into the Unicon language while meeting the language standard of high level; semantics and maintain backward compatibility

 How to measure these new features?

o By presenting several example and CVE experiments demonstrating qualitative and quantitative analysis of the new features. Qualitative analysis includes presenting code examples using these features, demonstrating the ease of use of these new features and their integration with the overall language semantics. Quantitative analysis includes conducting experiments to compare the performance of Unicon with and without the new features, presenting how this affects CVE on both server and client sides.

Enabling the very high level language (Unicon) to deliver the requirements of the virtual world (CVE) answers the first question in the dissertation. The need for new features is determined by studying what functionalities virtual worlds provide, what performance level is required, and what features are missing from Unicon. CVE is a test case for this study. The extent to which requirements are met is evaluated in Part III.

Most of the new features are built into the Unicon’s language’s virtual machine and its runtime system instead of a class library because:

(23)

 These features require access to lower level libraries and data that are not available at the language level, such as 3D object selection.

 Some features are performance critical and have a great potential speed up when built into the language, such as array support.

 The language did not have the infrastructure necessary to do them. For example, Unicon language original designed assumed sequential execution in the virtual machine and the runtime system. These had to be extensively changed to support concurrent threads eliminating race conditions and introducing synchronization and communication mechanism. Changing the language was unavoidable in such case.

The second question in the dissertation is answered through identifying problem areas when new functionalities in CVE cannot be added or do not perform well because: 1) the language lacked certain features, such as 3D object selection, necessary to add these new functionalities 2) the language did not deliver enough performance for very demanding functionalities such as 3D models rendering and animation which triggers the addition of arrays and threads.

1.3 Contributions

The research in this dissertation results in several contributions in programming language design with regard to collaborative virtual world development. These contributions can be summarized as follows:

 Demonstrating the use of a very high level interpreted language to develop a real time virtual environment, including advanced features such as 3D models, animation and dynamic texturing.  Building new language features with a novel design, integrating them into a very high level

goal-directed language with a legacy virtual machine. These features not only provide very high level support for virtual environments, but are also consistent with the language design and spirit, and their benefits are not specific to virtual worlds.

 Blending the semantics and syntax of new features with existing language features in most cases. This has the following benefits:

o Reduce or eliminate the time it takes to learn the new features because they are used in the same way existing features are used. This makes an easy transition for language users to start using the new features.

o Keeping the language syntax and semantics changes minimal which also contributes to the ease of use of the language.

This feature “blending” can be witnessed in several features introduced to the language that are discussed throughout this dissertation. A summary where this technique is used can be seen in:

(24)

o Dynamic textures: are drawables like windows o 3D selection: operate like a GUI interface o Threads: look and feel like co-expressions

o Thread communication: integrated with network communication.

 Flexible design of new features that balances between simplicity and implicit use versus control and powerful use. Examples include:

o 3D selection:

 Direct use of 3D selection API relying on only one function and one keyword. This requires minimal setup, and makes it straightforward for simple scenes with few selectable objects.

 GUI-like interface for 3D selection, requires slightly more setup and writing event handlers, but more suitable for large projects with hundreds of selectable objects.

o Data Protection:

 Thread safe data structures provide a convenient and implicit locking mechanism for fine grained access to data structures.

 Critical regions and mutexes provide an explicit locking mechanism to protect large regions of code and support atomic behavior to several data accesses. o Thread synchronization:

 Implicit synchronization via use of communication operation which provide blocking and non-blocking semantics controlling when to stop and when to resume a thread without the programmer intervention.

 Explicit synchronization using condition variables which give programmers full control over when to block threads and when to move them to work.

o Thread Communication

 Fully shared memory model by passing references to data structures.

 Pure message passing by using only communication operators and channels to share data without passing references to data structures.

 A mixture of shared memory and message passing depending on the situation. For example, threads that work on a huge amount of data can share references to thread safe data structures instead of wasting a significant percentage of time transmitting messages, but when sharing little data with other threads, message passing would suffice.

(25)

1.4 Organization of the Dissertation

This dissertation is structured into four parts. The first part (Chapters 1-3) provides introductory background. The second part (Chapters 4-6) covers all of the new language extensions and features introduced to make Unicon suitable for games and virtual environments development. The third part (Chapters 7-9) presents the evaluation of the work that has been done in part II. The last part (Chapters 10-11) concludes the dissertation and proposes future directions and improvement opportunities.

In part II which is dedicated to the programming language support for virtual environments, Chapter 4 presents the 3D graphics features and enhancements incorporated into the language. Chapter 5 covers 3D selection while Chapter 6 discusses the introduction of concurrent threads to the language. Figure 1.1 demonstrates the many features in a programming language that are heavily involved in the development of a virtual environment. It also shows the focus of this research in case of the Unicon programming language. The size of each slice in the figure gives a crude approximation on the amount of work that goes into each part relative to others but it is not meant to be exact.

Part III, dedicated for evaluation, includes two case studies, first, portable extensible non-player characters architecture, and second, CVE as an example collaborative virtual environment development built using the Unicon language. Figure 1.2 presents the aspects of this research study, and demonstrate the fact that changes on any part might affect other parts.

Figure 1.1 The focus of this research on three features set in the Unicon language

Areas that are covered by this dissertation

(26)

Figure 1.2 The three major aspects of this research

Language features are evaluated by using them to implement substantial components of CVE. The evaluation part (Part III) starts with programming examples and benchmarks in Chapter 7, followed by Chapter 8; NPCs and their role in virtual worlds, including a discussion about quest activities, the design and implementation of the proposed NPC architecture. Chapter 9 introduces CVE, presenting its design and implementation evaluating it and its features as a virtual environment case study for this research.

Unicon

CVE

(27)

(28)

2 Background and Related Work

This dissertation addresses a broad subject relating to many areas. The work presented in this chapter is carefully selected not only to be related to the work in this dissertation, but also to give context for the key innovations and design decisions of the dissertation as well as to give an idea of the breadth of this research.

The first technical subject area necessary to support virtual environments is 3D graphics. Technologies, techniques and hardware for developing 3D graphics applications have been changing rapidly since the 1990s. Doom [18], the popular PC game released in the early 90s, was able to achieve a level of realism that no other PC game was able to match at the time. In addition, Doom featured the ability to play over the network allowing several players to play in the same virtual world. Since then, the expectations of what a game can provide and how it should look have risen, driving great advances in 3D graphics libraries, game design, and giving birth to game engines [19]. Video games that followed Doom drove advances in graphics cards, bringing them to the masses at increasingly lower prices [19]. Despite these advances, developing these kinds of applications remains a challenge, requiring advanced level of programming expertise, big teams and huge budgets.

Many tools, libraries and languages have been developed or tuned to suit these kinds of applications. A survey of collaborative virtual environment technologies by Wright and Madey [20] covers various tools, techniques and frameworks used to develop virtual worlds. This chapter sheds light on some of the previous work relevant to this dissertation, and gives a background on several subjects that constitute the core interest of this dissertation. Section 2.1 presents the definition of virtual environments and provides example virtual worlds and some research in the subject. Section 2.2 discusses game engines, the building blocks of many virtual worlds and popular games. Section 2.3 covers some of the literature related to language support for virtual environments: mainly graphics, user interaction with the virtual world (3D object selection) and concurrency. The last section in this chapter, 2.4 gives a brief overview of the research in NPCs which are an essential part of virtual worlds, and hence, are covered in this dissertation.

2.1 Adaptable Collaborative Virtual Environments

Virtual environments can be defined as computer-generated, three-dimensional settings in which the users of the technology perceive themselves to be located, and within which interaction takes place. As the technological barriers to creating virtual worlds have decreased, researchers have created many collaborative virtual environments to serve various domains. The popularity of virtual worlds has increased, as well as their numbers and users, with hundreds of virtual worlds available online populated with tens of millions of players.

(29)

The term virtual environment is used to refer to a wide range of applications. These applications can be categorized using different criteria as explained in the following paragraphs.

 Based on the number of users:

o Single user: The popular game Super Mario Bros is an example [21].

o Multi-user: a limited number of users (in most cases less than 20) can share the virtual world over a local network. An example from games includes Doom [18].

o Massive multi-user: a large number of users (thousands or more) share a huge virtual world connecting over the internet, such as Second Life [6] and World of Warcraft [5]. This is distinguished from Multi-user above by the large number of users, and also the persistent world which continues to exist and change while the players are not online. The world itself continues to be online.

 Based on appearance:

o Textual: the world is described by words leaving the interpretation to the user’s imagination. MUDs (Muti-user Dungeons) are examples of such worlds. AberMUD is an example MUD game [22].

o 2D: the world is represented by 2D graphics. Examples include the old arcade games and also many Adobe Flash games [21].

o 3D: the world is constructed via 3D graphics. Many popular virtual worlds fall in this category, such as Second Life [6], ActiveWorlds [7] and many other games.

 Based on technology:

o Desktop virtual environment: the virtual world is a conventional application, no special hardware is used. Most computer games belong to this category.

o Virtual reality: includes enhanced elements of immersion, assisted by hardware and devices, such as gloves and head mounted displays. Usually this is used for specific domains like manufacturing and training such as flight simulators (Figure 2.1).

(30)

Figure ‎2.1 Virtual reality: flight simulator [source: www.freeway.org]

o Augmented reality: Augmented Virtuality and Mixed Reality are combinations between real and virtual environments. Mixed Reality refers to any environment that has both real and virtual aspects. If this environment is real with added virtual aspects of information it is called augmented reality. If it is virtual with added real aspects it is called augmented virtuality (see Figure 2.2). For example, augmented reality starts with a real physical environment and the computers alter the view of this environment by adding or removing information (visual, sound, etc...) on top of this view. A hypothetical example would be the ability to walk downtown wearing glasses that block all ads replacing them with a photo of a tree. Augmented reality is widely used in military, industry and scientific research. It is also used in sports, while watching a soccer match for example; TV viewers can see lines, numbers and so on projected onto the soccer field explaining an “offside” situation (Figure 2.3) or how far a free kick is from the target .

Figure 2.2 Simplified representation of an Reality Virtuality (RV) Continuum [23]

Real Environment Virtual Environment Augmented Virtuality (AV) Augmented Reality (AR) Mixed Reality (MR)

(31)

Figure ‎2.3 The use of augmented reality in a sport show [source: www.totalfootballmadness.com] Multi-user 3D desktop virtual environments (from now on referred to as virtual environments) are the focus of this dissertation. Many of the ideas and the features discussed here are applicable to other kinds of virtual worlds, but discussing the full range of virtual, augmented and mixed reality research is beyond the interest and capacity of this dissertation. Some closely related research on virtual worlds is presented here. The following subsections cover a few popular virtual world applications. While there are hundreds of such applications to choose from, those presented here represent unique examples because they can be adapted to different uses. Two virtual world examples are discussed, Second Life and ActiveWorlds. Menneche et al [24] present an introduction to Second Life and other virtual environments. They also offer a road map for research in virtual worlds in general, with the future and potentials of such worlds from different perspectives.

2.1.1 Second Life

Second Life [6] is a 3D online social virtual world. Launched in 2003 by Linden Lab, Second Life acts as a platform, giving its users a virtual world that they can build and hang out in, doing many of the things they do in real life. Unlike conventional games, there is no specific task or mission a user should accomplish, a big monster to kill, or an enemy to fight, making it confusing to users at first who try to identify it as a game.

The virtual world in Second Life is referred to as a grid, and composed of “islands” as can be seen in Figure 2.4. Some of these islands are huge and accessible by the public where others are private and owned by individuals, institutions, or companies. New islands can be created on demand and come with a price in addition to monthly fees. Second Life, charges an initial $1000 setup-fee (April 2012) for having a virtual private region (what is called an island), and $295 monthly lease (April 2012) [25] . The costs add up quickly depending on the size of the land requested and the services provided.

The line is projected onto the screen to show the offside line

(32)

Figure 2.4 A map of some islands of Second Life viewed‎from‎Second‎Life’s‎world‎map‎tool.

Second Life features the idea of user-created content. It is a dynamic world where its content is continuously updated and new content is added. The drawback is the high bandwidth requirement, which slows down the virtual world rendering in general. Second Life also includes its own scripting language, allowing users to attach behaviors to certain objects and make them respond to certain events. The scripting language is a useful feature in Second Life, but there are many constraints on what can be done and how much can be done using it. It cannot be used to implement arbitrary new features, and worse, users who wish to add objects to the virtual world have to upgrade their accounts in Second Life from standard to premium membership and buy a piece of property in the virtual world. Even then, the number of objects (or graphic primitives) is limited and constrained by the size of the land. For example, a piece of property approximately the size of the University of Idaho Janssen Engineering Building (JEB) allowed about 450 graphical primitives, enough to represent only a very simple structure representing the outline shape and floors, but not individual rooms, of JEB.

2.1.2 ActiveWorlds

Similar to Second Life in many aspects, ActiveWorlds is another big online virtual world. The project started in the 90s to provide a 3D equivalent of a web browser, and grew up over the years to become a popular online virtual world. Users in ActiveWorlds are called citizens, and they have to pay a monthly fee ($6.95/month (April 2012)) to be able to use all of the features and access all of the contents in the world. There is also a free membership called tourist mode, which provide very limited access to ActiveWorlds [26]. Figure 2.5 is a screenshot of ActiveWorlds showing the place a new user first sees after their first login to the virtual world.

(33)

Figure 2.5 ActiveWorlds Gate, the place where new users first login to.

ActiveWorlds is composed of “worlds”, similar to Second Life’s islands. Many of these worlds are owned by ActiveWorlds and are public for all citizens, while some worlds are owned by citizens. It is up to the world owner to make it public or private. A collection of interconnected worlds composes a universe, which makes an independent, standalone, and large virtual world. ActiveWorlds’ main universe is composed of hundreds of worlds and it is the one that ActiveWorlds users have access to. Users such as institutions can buy their own private universes, to create a completely independent, to some degree customizable, virtual world. These universes come with their own servers which have to be leased from ActiveWorlds yearly, adding to the overall cost of owning a universe. There is also a limit on the number of users and the size and content of these universes. Active Worlds Educational Universe is an example service package that can be bought from ActiveWorlds. The package price is $650 with $395 annual renewal fee (April 2012) allowing only 20 simultaneous users.

Second Life and ActiveWorlds can be customized to some degree by changing the existing world content or adding new content. They are popular virtual worlds with a lot of documentation covering them. While these worlds allow customization through content creation and scripting, there is still a hard limit on what and how much users can add. They are not flexible enough to allow the creation of a new arbitrary world or application to allow new kinds of activities or use cases, such as supporting a collaborative editor inside the virtual world. They are not extensible enough to allow plugging in a new technology or the support of a 3D model format for example. They also come with a price and continuous cost as mentioned earlier.

(34)

2.2 Game Engines

One of the popular methods to build new virtual worlds is to use a game engine. A game engine is “a system designed for the creation and development of video games” [27]. A short and useful summary of game engines can be found at [28]. The purpose of the game engine is to provide reusable software components and features required when developing a game or virtual world. Usually it provides an abstract and in some cases platform independent layer on top of the underlying implementation and libraries. The philosophy behind the game engine is the idea of separating the game content and game specific behaviors from the game engine itself [29]. Figure 2.6 shows a modular game engine structure and pointing where the game engine fits in the big picture.

Figure 2.6 Modular game engine structure [30].

Some engines are very specialized and may be used as subsystems of other tools and engines such as the popular Havok physics engine, while other game engines provide several functionalities that include some or all of the following:

 3D graphics rendering  2D drawing

 Graphical user interface (GUI)  Physics simulation

 Multimedia support (audio, movie playback)  Scene graph management

 Input/output and networking  Artificial intelligence (AI)

(35)

Bishop et al, summarize the features found in many game engines. They define a game engine so that it includes only those elements that have no effect on the actual game content in addition to the game’s main event loop. Those elements are indicated by dashed-lines ovals in Figure 2.7.

Figure 2.7 Schematic of a game [29]

Game engines are a form of middleware because they are positioned between the game itself and the underlying system. There are hundreds of game engines available for developers to choose from, covering a wide range of game designs and features. Many of these game engines, especially those used in professional games, come with a very high price tag, while other engines are open source, free or have a low price but are less robust and have fewer features. According to [31], cryEngine is the most expensive game engine, with a price of more than $200,000.

Among the hundreds of game engines, in addition to cryEngine, popular commercial game engines include Unreal, CrystalTools, Gamebryo, and Jade, just to mention a few. Open source and free engine examples include Ogre, Panda3D, Quake3, and OpenSceneGraph. Delta3D is a popular open source game engine used in many projects [32] [33]. Devmaster website [34], provides a comprehensive list of game engines available. A report created by ELIAS project [35], represents a good survey of many open source and low cost game engines. It also discusses the features of game engines with advice on how to pick one to develop a game.

One drawback of most game engines, in addition to the price, is limited portability [32]. Content is usually specific to the selected engine and to suit a specific game genre. The portability issue includes the ability to use the engine in very different game designs, limited portability to move the game engine between different platforms and operating systems, and limited ability to migrate game content to a new game engine if needed.

(36)

Another problem with using game engines is that they are usually large and complex black box systems. They do not always give as much of an advantage as is hoped for. According to Fristrom, a technical director and a designer at a game company, when he was talking about using a game engine [36]:

“You'd think it would turn a three year project into an eighteen month one, but in my experience it really only saves about six months. You'll find yourself saying things like, ‘If only the engine was designed from the start for network play!’ and ‘If only the engine was designed from the start for a full-state save game!’ And so on.”

While this discussion is informal and might not be true for all projects, it still sheds some light on the difficulties game companies face. Six months is a big time saving, but clearly, game engines do not magically solve all of the issues and complexities related to developing games and virtual worlds. They only solve a small fraction of MMO needs, and in many cases development teams have to cope with the inflexibility of the engine.

One more issue in game engines is that, due to performance requirements, the majority of them are written in C++ or C. Few of them provide binding to other languages and/or incorporate the capabilities of scripting languages such as JavaScript and Python. While C and C++ are a lot easier than assembly language, the language of the early games, programmers still require extensive experience and mastery of these languages to be able to write a meaningful game, or develop a virtual world. They lack many features and data structures found in higher level languages that save the programmers a lot of time.

The highly specialized and often expensive game engines are usually geared toward professional programmers and high performance games. A simpler and more manageable approach is needed for less trained programmers and students. Darken et al, point out that the highly specialized game engines model works perfectly for the game industry, but simply does not work for the training community [32]. A new simplified approach can be built around the programming language itself. Building many features found in a game engine into the programming language, and hiding most of the implementation details, and melting those features with the natural flow of the programming language itself, does not require the programmer to learn different tools or incur too much extra complexity in the code. The following section covers some features required by virtual world development, and the support for such features in some programming languages and libraries.

To summarize, game engines are powerful tools, built to be used for one thing as their name implies; i.e. developing games. They encapsulate many of the features and libraries necessary to build virtual worlds, making it easy to integrate many of the virtual world’s functionalities. Moreover, most of them are performance tuned to meet the requirements of demanding games. However, most game engines are big, expensive, platform dependent, pieces of software. They are complex and hard to learn and master. They

(37)

are not flexible, when a new feature is needed that is not provided by the engine it has to be implemented externally.

While using a language in place of a game engine might not provide the level of specialization or performance provided by a standalone game engine, there are advantages to such an approach. First, with a very high level language, powerful features and data structures can be used to simplify the overall software design. Second, the language approach provides one consistent programming style across all APIs, syntax and features provided by the language. A game engine on the other hand, is separate from the language used in the software under development, making it hard to match what the engine provides with how the language works. This might cause the programmer to keep jumping between the concepts of the engine and the language, unlike the consistent and smooth flow of programming style using the language approach. Third, using the language allows the programmer to build all of the needed features and tools with complete freedom and maximum flexibility, without being restricted to a set of features or mechanisms found in a game engine.

2.3 Language Support for Virtual Environments

The following subsections introduce a set of features in programming languages that are essential to develop virtual environments. Some languages and libraries that provide such features are covered and discussed along the way. [37] presents a previous work aiming to help reduce the cost of building virtual environment.

2.3.1 3D Graphics

The huge advances in graphics hardware in the past two decades allowed for a dramatic leap in the degree of realism that computer graphics can achieve, with richer scenes and larger virtual worlds. However, this also increases the amount of data to manage and code to organize, exploding the complexity of such applications. Computer graphics sits at the center stage when it comes to games and virtual environments. Graphics programming is probably the hardest task in developing such applications both in terms of programming complexity and performance requirements. When creating a game or virtual world, graphics programming usually is done using one of the following five approaches:

1. A game engine.

2. A specialized programming/scripting language. 3. A graphics library.

4. A general purpose programming language that has a built-in graphics API. 5. A mixture of the above.

(38)

Depending on the application under development, one approach might be more suitable than others. Each has its own advantages and disadvantages. A game engine for example might give access to advanced features like animation, constructing and managing the scene graph, and provide the game physics, but it is not flexible or customizable; the programmer is constrained as to the languages that can be used with such an engine and the systems that can run it.

2.3.1.1 OpenGL and Direct3D

The two best known graphics libraries are OpenGL and Direct3D. OpenGL (Open Graphics Library) is a standard cross-platform graphics API. It is very well documented and supported on almost all platforms. OpenGL sets the standards for 3D software and is widely used in all kinds of graphics applications, especially in industry, design and education [38]. Direct3D is owned by Microsoft and only supported on Microsoft platforms, but it dominates the PC game industry.

The OpenGL API is available for several languages including C, C++, Java and many others, while the Direct3D API is only available in C, C++, C# and Visual Basic [39]. Both libraries include hundreds of functions that deal with core graphics programming with little or no support for window systems, 3D models, and animation. Programmers have to rely on other libraries and tools to add such support. The huge APIs, and supporting libraries create very complex programming environments. Only experienced programmers become proficient in dealing with the complexities associated with such systems. The following sample OpenGL program written in C draws a rectangle:

glBegin(GL_QUADS); // Start drawing a quad primitive glVertex3f(-1.0f, -1.0f, 0.0f); // The bottom left corner

glVertex3f(-1.0f, 1.0f, 0.0f); // The top left corner glVertex3f(1.0f, 1.0f, 0.0f); // The top right corner glVertex3f(1.0f, -1.0f, 0.0f); // The bottom right corner glEnd();

The code above represents a deprecated (as of OpenGL 3.0) style of rendering a set of vertices in OpenGL. Newer and more efficient functions were introduced in OpenGL version 2.0, but they are neither shorter nor simpler to use. The code leaves a lot of the rendering details out, such as color and relative location. It also does not cover many of the programming details required for this code to work such as creating a window or preparing a frame buffer.

2.3.1.2 Domain Specific Languages for Computer Graphics

Domain specific languages are designed to serve a particular domain, utilizing the theory and vocabulary for that specific domain. Usually this allows the language to be more expressive, more readable, and more productive for such domains, but it is not useful or even not usable outside its domain [40].

(39)

Shading languages are a good example of domain specific graphics languages. These languages feature C-like syntax and are used with programmable graphics processing units (GPUs). Shading languages are usually designed to achieve maximum performance utilizing the GPUs available on many computers. Cg, GLSL and HLSL are three popular shading languages. Cg (C for graphics), a language developed by Nvidia, was one of the first languages widely adopted because it is platform independent [41]. GLSL (OpenGL Shading Language), as its name implies, is designed to be used with OpenGL [42]. OpenGL and GLSL work together and share states to maximize the performance of graphics hardware. Microsoft introduced HLSL (High-Level Shader Language) to be used with their DirectX framework [43].

While shading languages can deliver very high performance graphics, they work very close to the hardware. They are very low level, compared to what very high level languages feature. However they represent a great potential with the performance they can deliver for certain graphics operations. Shading languages can be used in the implementation of such operations in the runtime systems of a very high level language. By doing so, certain graphics features gain the boost they need where performance is critical. Doing such integration falls within the interests of this dissertation, but it is left as a future work, largely because of the amount of work involved in such integration, and because it is orthogonal to the primary contributions of this dissertation.

Apart from graphics focused languages, some domain-specific languages were developed to help build virtual environments, in particular, to help develop such worlds to be used on the web, usually using a plugin inside a browser. Two languages are presented here: VRML, and X3D. Both VRML and X3D are closer to model file formats than to programming languages. Files written in these languages are fed to a browser plugin that serves as an engine that renders the data in these files to build virtual worlds.

Virtual Reality Modeling Language (VRML) was the first language to establish itself as a standard for 3D graphics and virtual reality development on the internet [44]. Graphics information is usually saved to text files with .wrl extension. If a VRML plugin is installed in the browser, the browser launches the plugin to translate textual model information into 3D graphics scenes. VRML provides a high level abstraction for 3D scenes. Objects in the scene are organized in a hierarchy and referred to as nodes. The appearance of objects can be controlled via property nodes. The whole scene is rendered by traversing all of the nodes in the graph hierarchy. Figure 2.8 presents an example scene generated using VRML. The following code provides an example of VRML file content:

(40)

#VRML V1.0 ascii Separator { Material { diffuseColor 0 0 1 } Cube { width 3 height 3 depth 3 } WWWInline { name “http://vrml.test.org/horse.wrl” } }

Figure 2.8 A scene with avatars generated using VRML [44]

Extensible 3D Graphics (X3D), a web graphics language based on and backward compatible with VRML, is an international standard for Web-based graphics [45]. X3D uses XML to encode graphics data and scene graph information. This facilitates moving such data between different platforms, and incorporating it into Web services [46].

VRML and X3D are examples of languages that were created specifically to facilitate the process of building virtual worlds. They are specialized tools that describe the content of the world rather than how to build it. What they are missing is the power of a general purpose programming language that is needed to cover the wide range of programming requirements necessary to build a virtual world.

(41)

2.3.1.3 Languages with Built-In 3D Graphics Support

Many languages incorporate built-in support for 2D graphics, but few provide built-in support for 3D graphics. That is mainly because most programming languages including mainstream language were designed before the 3D graphics era. Another reason is that there are standard graphics libraries that represent a portable mature alternative for what a language can provide such as OpenGL. Support for 3D graphics in a language can be done through library bindings or a using a built-in API.

Java3D represents another effort from Sun Microsystems and its partners to bring 3D graphics to the Java programming language with the first version released in 1998. Java3D is built on top of OpenGL, Direct3D or recently JOGL rather than a direct binding to a particular underlying library. It has a higher level semantics, as well as more abstractions and features that facilitate 3D graphics programming, separating it from the lower level library such as OpenGL. It supports stand-alone programs as well as Java applets that run in a browser [46]. It includes high level features such as scene graph creation and management, and is capable of loading several model file formats. The default rendering in Java3D is OpenGL. A JOGL rendering exists as an intermediate interface to OpenGL in order to support systems that do not have direct Java3D OpenGL interface such as Mac OS X.

Java3D is a rich 3D graphics API with a wide range of capabilities and features not found in lower level APIs such as scene graph support and 3D model file loading and rendering. The API defines over 100 classes in the core package alone [47]. In addition to that, three extra packages are also required in most Java3D applications bringing the total number of classes to several hundreds. Here are four packages that are usually needed to write an application using Java3D [47]:

 javax.media.j3d: Java3D core classes. Contains the Java3D low level classes such as graphics primitives.

 com.sun.j3d.util: Java3D utility classes. Provides the higher level functionalities such as scene graph creation and management.

 javax.vecmath: Vector math classes provide the math for vector, matrices, etc., used in 3D graphics applications.

 java.awt: Abstract windowing toolkit used to create and manage windows.

Due to the huge number of classes in Java3D, and the constrained and specific programming style that has to be followed when writing code in Java3D, the learning curve is very steep. Java3D provides a powerful set of tools, but the number of classes and the steep learning curve hinders programmers from using it, especially inexperienced programmers who are not accustomed to using such complex tools to write a virtual world. The following program from www.java3d.org demonstrates the use of Java3D to draw a