Using Ontology to create 3D Animations for Training Purposes

Using Ontology to create 3D Animations for Training Purposes

Salvatore Parisi, Jochen Bauch, Jan Berssenbrügge and Rafael Radkowski Department of Computer Integrated Manufacturing,

Heinz Nixdorf Institute – University of Paderborn {Salvatore.Parisi, Jochen.Bauch }@hni.uni-paderborn.de {Jan.Berssenbruegge, Rafael.Radkowski}@hni.uni-paderborn.de

Abstract

3D Computer-based animations are nowadays used for training purposes in a wide range of industrial applications like assembly, maintenance and operations. Their creation, also known as authoring, is usually a time consuming task, demanded to professional 3D designers, who need at first a good understanding of the involved entities and actions in order to realize the customized animations. The proposed methodology deals with the use of an ontology in order to filter and understand generic natural language training requests; once identified the proper actors and actions, they are associated to the corresponding models and movements to be performed in the virtual environment and translated in a 3D graphic format template. The result is a customized animation which can be created by a non-expert designer and then visualized by the worker through many modalities, on a desktop computer or on a PDA for example. Role of the ontology is to reduce the overall complexity of the animation authoring process by assuring the necessary comprehension of customized training requests as well as reusability and extensibility of the structure of the modeled object and of animations’ components in different domains..

1. Introduction

Virtual reality represents nowadays a widespread approach both for entertainment as well as in industrial scenarios. From product development (fig.1), where digital mock-ups can be modeled and tested in a virtual environment, saving costs of real mock-mock-ups and reducing the time to market, to the training of workers for operations, maintenance and assembly tasks, to the simulation of manufacturing processes.

3D Computer animation is a subfield of computer graphics and of the more complex Virtual Reality approach, since it still maintains the characteristic of interaction but no immersion is provided to the user. In the training scenario, which is described in this paper, the user can learn the task that has to be executed just by watching a 3D animation (fig.2) of the involved objects through their counterparts in the virtual environment. The trainee can navigate in a 3D environment and manipulate objects by selecting and moving the involved parts, thus “learning by doing”. In this way, it becomes very easy to understand which parts are involved, where they can be found and in which sequence they must be treated.

The authoring process represents the generation of multimedia content, like a 3D computer animation. It is usually demanded to a professional 3D designer that, once understood the requirements, models the scene through 3D modeling software, as well as through existing 3D models, and eventually animates it. The difficulty here is correctly identifying the involved parts, how they move and which “cause-effect” relationships characterize them.

Not all users require the same quantity and quality of information; specific information can be, according to the different role of the worker, essential or useless. For example, focusing on the manufacturing sector, a machine operator needs to have an overall understanding of the machine and of the parameters for the manufacturing process, while a maintenance worker needs the detail of the internal parts, e.g. electric cables, which are not needed by the former user-type and instead could cause confusion to non experts. Therefore, animation authoring must be able to manage different level of details and apply the correct one to the correct user-type, which can be depicted as a “role.”

Figure. 1. Product development process [21]

Figure. 2. Sequence of pictures in a 3D animation

Vocabulary plays also a crucial role since technical parts have specific names, which are usually not known by people that are not expert in the domain. Also actions can be vague when expressed in natural language commands: which subtasks the verb “remove” includes,

in the sentence “remove the drill bit”? A 3D artist is in fact normally not an expert in the technical domain, like Manufacturing or Automotive ones; therefore he has usually not enough confidence with mechanical or electrical parts and their vocabulary.

In addition, the overall complexity explodes in case a high number of animations for different purposes and user roles need to be authored; usually products may differ just in a very few parts but nevertheless the author has just the 3D models as starting point while already some animation atoms could be reused for some concepts and parts, that are often used for more than one animation.

The problems analyzed above show the necessity to supply industrial customers, as well as 3D artists, with a new methodology that is able to:

• Automatically understand customized training requests;

• Identify the involved parts and the actions to be performed through a shared terminology;

• Assure a reusable data structure;

• Manage different kinds of end-user roles and the level of detail necessary in the animation;

• Generate the animation automatically, starting from the training request and from the corresponding 3D models.

Chapter 2 deals with state of the art in literature and commercial software solutions regarding “intelligent” animation authoring; also an overview of ontology and its use for training purposes is described.

Chapter 3 describes the ontology-driven methodology and its single steps, ending with a test case scenario.

The paper ends then with some conclusions and the plan for the related future work.

2. State of the art

The challenge to automatically generate computer animations is of course very fascinating but it remains bound to a double input constraint: on the one hand, a description of the animation, also known as “script” and usually described through “natural language”, and on the other hand the necessary 3D models that build up the animation itself.

In the industrial training scenario, the former could be an instruction manual or a generic training request for a specific user-type; the latter can be derived from a PDM system or, in a very simple case, directly through the corresponding 3D model. But how to translate the commands from natural language to a formal description that can be understood by computers and then generate the animations?

In literature such approach is referred to as “text-to-scene”: one of the first approaches can be found in “Wordseye” [1], where the text is translated in 3D pictures but not into animation. This is achieved through the use of a large database of 3D models and poses to depict entities and actions.

Another approach is the generation of car accidents animations in CarSim [2], where according to written road accidents reports, a 3D animation of the accident itself is automatically generated.

Figure. 3. Screenshot of a Carsim animation

In Confucius [3], a semantic approach, by means of the WordNet lexicon [4], is used in order to animate virtual humanoid characters.

The most advanced software solution on the market is represented by the ParallelGraphics, now Cortona, Virtual Manuals. It is a comprehensive set of software applications, which enable industrial companies to produce 3D simulations using existing CAD and PDM data. A wizard-like authoring process, which comprehends predefined animation types by common terms, is the closest approach to the automation of 3D animations authoring.

Other solutions are able to deliver content via e-Learning and Virtual Reality but they are not able to automate the authoring process, which is left to the creativity of the 3D artist.

The approach proposed in this paper goes further, using ontology as a core of the 3D animations generating process, in a sort of “intelligent authoring.”

The use of Ontology for training purposes can be already found in [14], but just for a computer-based Training, which does not include the display of 3D environments or animations.

The term Ontology has been borrowed from Philosophy where it means theory of existence; it tries to define what exists in the world by introducing a system of categories and their characteristics and relationships. The term has been used in the last decades in the Artificial Intelligence field to define a new branch which deals with concepts of the real world and their relationships.

In Computer Science there is not yet a shared definition but there is enough convergence about the one provided by Gruber and later modified by Borst [5], that ontology is a “formal specification of a shared conceptualization”. Another definition comes from Mizoguchi [6] where ontology is defined as “a theory (system) of concepts/vocabulary used as building blocks of information processing systems”. Currently ontology plays a crucial role in what is called the “semantic web,” the evolution of the World Wide Web from syntax-based to semantic-based documents.

3. Ontology-driven Authoring

The role played by ontology in the methodology proposed in this paper, can be summed up in the following points:

• Common knowledge-base of different concepts, covering also everyday life; • Shared terminology;

• Extensibility and reusability of concepts;

• Possibility to manage different roles and their information access; • Possibility to model the training object through its parts and relationships; • Context-based definition of the verbs

In order to realize an “intelligent authoring” of 3D animations, a methodology (Figure. 4) has been developed.

The starting point of the methodology is the training animation request, which in a first phase can be thought in a text form, even though the approach remains valid also for other kinds of input, like voice recognition. The request is characterized by a title, which corresponds to the topic of the training session, and the content, made up by a sequence of subtasks. Two possibilities for the input can be foreseen, according to the existence or not of previous training material in digital form:

• Loading an existing sequence of steps, e.g. from an instruction manual

• Enter the title of the animation, which delimitates the context of the target area, and eventually the single steps that build the animation itself

The input text is analyzed at first from a lexical point of view by means of a natural language parser. This is necessary to identify the different parts contained in the animation request and have a first categorization into nouns, verbs, adjectives, prepositions, etc. State-of-the-art natural language parsing uses nowadays a probabilistic approach that analyzes the structure of a sentence according to its maximum likelihood; this can be found for example in the Stanford lexical parser [5].

The resulting lexical entities, filtered from less important parts like prepositions, adverbs, etc., are then passed to the ontology for semantic identification. The same word, in fact, according to the context, can assume different meanings: for example, a “resource” can be referred to a raw material, which has to be machined, or to the machine itself.

Before dealing with ontology matching, some preliminary steps about the definition and development of the ontology are necessary. Building an ontology from scratch is a complex and time consuming task; in addition, as in its nature itself, it must be shared by the users’ community. Therefore it is more suitable in the proposed approach using an existing ontology as “upper ontology” and then using multiple domain ontologies for specific concepts.

Two starting points can be the Suggested Upper Merged Ontology (SUMO) [10] and the OpenCyc [11] ontologies, both very rich upper ontologies and candidates for the IEEE “standard upper ontology”. They provide a collection of general concepts, called “classes”, together with their relationships, about common sense knowledge from a higher-level point of view. Those classes can be eventually populated by “instances” that represent the individuals with their own differences but nevertheless included in a class; a “manufacturing machine” can be considered as a class while the different models are instances of the upper class. Since conceptualization depends on man’s vision of the world, another possibility is that the “manufacturing machine” class has two more subclasses “lathe” and “drill”, and then as instances the respective lathe and drill models. The representation of the target world varies from rough to refined concepts, according to the required level of detail: the more parts are involved in the target domain, the higher the level of detail must be.

The upper ontology provides then the necessary general knowledge of concepts, which can be enriched with a “domain” ontology that models and deals with a specific domain, e.g. “car”, “manufacturing machine,” etc.

Therefore, starting from an upper ontology, it is possible to model a second ontology which will be the target scenario for the training. The domain ontology includes refined

concepts of the target scenario as well as their instances and verbs which, once combined, can have different meanings and therefore different translations in the virtual environment.

Due to its completeness, not only of classes but also instances, the ResearchCyc Ontology has been chosen as upper ontology. This version contains more than 300.000 concepts and more than 3 millions assertions; furthermore the Stanford Parser is already included even if more tests are necessary in order to verify the integration of the applications.

The entities coming from the natural language parser are searched through the ontology and matched with the concepts’ definitions.

Nouns describe the physical actors, both subjects and objects, and must be correctly identified together with their relationships: e.g. the concept “manufacturing machine” is characterized from at least one structural, mechanical, electrical part, etc. therefore once recognized the “manufacturing machine” noun, it cannot be considered alone but must be represented with all the necessary sub concepts that are strictly connected to it.

The same can be considered for the 3D models: the “drill” tool (Figure. 5) is not only a concept that must be defined in the ontology, together with its relationships, but it represents also an object with a precise geometry and orientation in the virtual environment.

Figure 5. Ontology and model matching

The matching of the objects contained in a 3D model can be done through state-of-the-art tools, like PDM systems, which can perform search of parts through metadata.

Core of the training request is the verb. It usually defines the action performed by the subject on the object. The action can be basically of two types: abstract, when there is no practical action and the action cannot be visualized in the virtual environment, such as “maintain” or “avoid”, or physical, where the action can be represented through movements and rotations of the considered object, e.g. “remove” or “mount”.

In the first case, appropriate metaphors must be used in order to translate abstract actions in something visual: a system of signs can be used together with the physical verbs to show what is allowed or what must be avoided, as well as some text description can be used to avoid misunderstandings.

Physical verbs are easier to describe because they involve some practical action; the problem here could be differentiating the same verb according to the related object. The

Drill Drilling Tool Position (x,y,z) Dimensions (h,w,d) Tool Store IsPartOf

ontology is able to link entities in order to get different meanings: the verb “remove” combined with the word “screw” gives the event “unscrew” while “remove” and “material”, in the context of a piece refers to a machining operation.

There is also another lexical part that plays a very important role in general training requests: adjectives, as well as past participles, are able to describe important details that need to be represented in the training animation scene. An “open” part of a machine and a “machined” piece are just a couple of examples of objects that are enriched by the adjective or past participle with additional information.

The description of objects and actions coming from the ontology are sent to a module that translates the commands in a formal way, according to the chosen format for the graphic file. In table 1, the “remove the drill bit” sentence, which can be thought for a maintenance worker, is analyzed for better understanding.

Table 1. Ontology description and translation remove is action

hasContrary: add

hasRange Object (Drill, Part, Screw, …) remove object

cause to: Move Out

Translation Object Length Drill is Object isPartOf ToolStore has 1 bit

hasParameter: Position, RotationSpeed hasMovement: Move, x, y, z; Rotate Z ID:Tool2 Viewpoint ToolStore Highlight Drill ID:Tool2 Bit is Object isPartOf Drill hasParameter: length

hasMovement: Rotate z, Move z ID:Tool2-1 Viewpoint Drill Highlight Bit ID:Tool2-1

The verb “remove” is an action, has a contrary “add”, which can be used for the “assembly” role, and has as range the concept “object”: this could be one mechanical part, e.g. “drill”, “screw”, and causes to move out the object itself through the “translation” command, which translates the object for its length. According to the object, e.g. “screw”, also a rotation, together with the translation could be necessary, even if this is not the case.

The next entity is the “drill” object which is contained in the more general “tool store” concept; it has at least 1 bit, has some parameters, like position, rotation speed and so on, it can move through x, y, z axes and rotate around the z axis and it is characterized by an identification code. The result will be that the viewpoint command will bring the camera on the higher level object “ToolStore” and the object “Drill”, which is identified by a specific ID number, is highlighted for an easy identification. The characterization of the part and its retrieval from a general model database, like a PDM system, is outside the scope of this paper. The approach must be also able to manage the different level of details required by the “role” that every end-user plays. This can be achieved by a management of layers in the CAD model, which can be showed or hidden according to the context: the layer of “ electric cables” will not be visible to roles, like “machine operator” or “assembly”, which do not need such details.

The resulting information will be then collected and translated in a graphic format, in order to be visualized through the corresponding viewer.

Up to now the choice of the appropriate graphic format is to be investigated; the possible scenarios are the building of a scene graph with the involved entities as nodes or the use of a predefined standard template of a graphic file format, like X3D.

4. Test case: KoBaS project

The idea developed in this paper has been inspired by the KoBas, “Knowledge Based Customized Services for Traditional Manufacturing Sectors Provided by a Network of High Tech SMEs”, Project [11] and, even if it as not yet been implemented, has as natural target the manufacturing machine field.

Modern manufacturing machines have been transformed by the need of flexibility from specific machines to machining groups, complex and sophisticated environments, which are able to execute different task.

Machine producers are also able to offer a huge number of product configurations which can be adapted to customer needs. But manufacturing groups have their own assembly instructions, specifications, controls and maintenance that remain independent from the complete manufacturing machine. This means that the overall complexity can be reduced recovering data from a limited database, in this case from an ontology. The advantage is a data structure that can be shared and reused, maintaining a semantic coherence.

Training for manufacturing machines regards basically at least four roles: • Machine operation

• Assembly

• Maintenance • Sales

Each role has its own need of information and redundant or not needed content is to avoid. The methodology proposed in this paper, even if not part of the project itself, aims to give the KoBaS Project, an instrument that is able to optimize the authoring of customized training

animation, reducing the costs of the authoring and the necessary time amount. It opens also new scenarios for other industrial application that can benefit from an “intelligent authoring.”

5. Conclusions and future work

The methodology proposed in this paper aims to automatically generate 3D computer training animations starting from a sequence of natural language requests. By means of an ontology matching, the involved parts as well as their actions, are identified and then into a graphic file template. The result is a customized animation to be displayed through the corresponding viewer.

The approach is flexible since it can be adapted to every engineering domain just by modelling the related domain ontology.

Future work is focused on the implementation of a small prototype for the whole process, including:

• Development of a domain ontology for the manufacturing field and ontology query process

• Choice of the graphic data format and scenegraph and development of a standard animation template

6. References

[1] B. Coyne, R. Sproat, WordsEye: an automatic text-to-scene conversion system. Proc. 28th annual conference on Computer graphics and interactive techniques, Los Angeles, CA, 2001, 487 – 496. [2] O. Åkerberg, H. Svensson, B. Schulz, P. Nugues, CarSim: an automatic 3D text-to-scene

conversion system applied to road accident reports, Proc. of the 10th conference on European chapter of the Association for Computational Linguistics - Volume 2, Budapest, 2003, 191 - 194. [3] M. Ma, P. McKevitt, Visual Semantics and Ontology of Eventive Verbs, Natural Language

Processing - IJCNLP 2004, First International Joint Conference, Hainan Island, China, 2004, 187-196.

[4] G. A. Miller, WordNet: An on-line lexical database, International Journal of Lexicography 3, 4, 1990, 235-312.

[5] P. Borst, H. Akkermans, J. Top, Engineering Ontologies, International Journal of Human-Computer Studies, Vol. 46, No. 2/3, 365--406.

[6] R. Mizoguchi, Ontological Engineering: Foundation of the next generation knowledge processing, Proc. Of WI 2001, First Asia-Pacific Conference, Maebashi City, Japan, 2001, 44-57

[7] D. Klein, C.D. Manning, Fast Exact Inference with a Factored Model for Natural Language Parsing, Advances in Neural Information Processing Systems 15 (NIPS 2002), December 2002. [8] Pease, A., Niles, I., and Li, J., The Suggested Upper Merged Ontology: A Large Ontology for the

Semantic Web and its Applications, Working Notes of the AAAI-2002 Workshop on Ontologies and the Semantic Web, Edmonton, Canada, 2002.

[9] http://www.cyc.com /

[10]http://www.pdm-if.org/pdm_schema/ [11]http://www.kobasproject.com

[12]Hall, G., The Internet, PDM and shared engineering - the future of product realization, PDM Implementation - Pains and Gains (Ref. No. 2000/054), IEE Seminar 20 April 2000 Page(s):5/1 - 520

[13]R. Mizoguchi, K. Kozaki, T. Sano and Y. Kitamura, Construction and Deployment of a Plant Ontology, Knowledge Engineering and Knowledge Management - Methods, Models and Tools -, The 12th International Conference, EKAW2000.

[14]L. Jin, W. Chen, Y. Hayashi, M. Ikeda, R. Mizoguchi, Y. Takaoka, M. Ohta, An Ontology-Aware Authoring Tool - Functional structure and guidance generation, Proc. of AIED'99, pp.85-92, Le Mans France, 1999

[15]L. Jin, M. Ikeda, R. Mizoguchi, Ontological Issues on Computer-based Training, PRICAI-96 Workshop on Knowledge-based Instructional Systems in an Industrial Setting, 1996, pp. 55-66. [16]Tsampoulatidis, I. Nikilakis, G. Tzovaras, D. Strintzis, M.G., Ontology based interactive

graphic environment for product presentation, Proceedings of Computer Graphics International, 2004, 644- 647.

[17]M. Ma, P. McKevitt, Semantic representation of events in 3D animation, Proc. of the Fifth International Workshop on Computational Semantics (IWCS-5), Tilburg, Netherlands, 2003 [18]C. M. Oresky, D. B. Lenat, A. Clarkson, and S. Kaisler, Strategic Automatic Discovery System,

Knowledge-based simulation: methodology and application, New York: SpringerVerlag, 1991, pp. 223-260.

[19]T. O'Hara, S. Bertolo, M. Witbrock, B. Aldag, K. Panton, D. Schneider, N. Salay, J. Curtis, Inferring Parts of Speech for Lexical Mappings via the Cyc KB, Proceedings of the 20th International Conference on Computational Linguistics (COLING-04), Geneva, Switzerland, August 2004.

[20] C. Matuszek, J. Cabral, M. Witbrock, J. DeOliveira. An Introduction to the Syntax and Content of Cyc. Proceedings of the 2006 AAAI Spring Symposium on Formalizing and Compiling Background Knowledge and Its Applications to Knowledge Representation and Question Answering, Stanford, CA, March 2006.

[21] J. Gausemeier, P. Ebbesmeyer, F. Kallmeyer, Produktinnovation – Stretegische Planung un Entwicklung der Produkte von morgen. Hanser. 2001

Authors

Born in 1976, he studied Management and Production Engineering at the University of Palermo, Italy. In 2002 he won an Erasmus scholarship at the University of Paderborn, where he discovered the field of Virtual and Augmented Reality applications in industrial environments. He also wrote his diploma thesis on this topic and graduated in 2002. In 2004 he won a scholarship for a Ph.D. in Dynamic Intelligent Systems at the International Graduate School – University of Paderborn. His research topics are Virtual and Augmented Reality, Artificial Intelligence and Mobile computing.

Jan Berssenbrügge

Born in 1970, he studied computer science at University of Paderborn, Germany. In 1998 he received a M.Sc. degree in computer science from University of Paderborn. From 1998 until 2004 he joined the Virtual Reality group at the Heinz Nixdorf Institute and worked as a Ph.D. candidate in the Department of Computer Integrated Manufacturing at the University of Paderborn. In 2005 he received a Ph.D. degree in mechanical engineering from University of Paderborn. Since 2005 he works as post-doc in the Virtual Reality group at the Heinz Nixdorf Institute. His research interest is in 3D computer graphics and in the development of Virtual and Augmented Reality applications in the field of Virtual Prototyping.

Rafael Radkowski

Born in 1976, he studied from 1998 to 2003 mechanical engineering with focus on product development at the University of Paderborn. He finished his studies as best in year. From 2003 to 2006 he was a scholarship holder at the graduate school „Automatic Reconfiguration in Open Systems“ at the Heinz Nixdorf Institute. In this context he joins the working group “Computer integrated Manufacturing“ under direction of Prof. Dr.-Ing. J. Gausemeier. His research focus is virtual prototyping, particularly the use of Virtual and Augmented Reality and simulation within the product development process. He finished his Ph.D. in 2006. Since 2006 he works as a post-doc in the workgroup of Prof. Gausemeier.

Jochen Bauch

He studied computer science at the University of Paderborn. Since 2001 he works at the Heinz Nixdorf Institute: his research topics are visualization, Virtual Reality and Virtual Prototyping.