Scheme overview

To allow exploration of the differences between games created with and without Narrative Threads, I created a new scheme for expert evaluation of narrative-based games. The scheme was created with reference to work on peer-based evaluation carried out as part of the

Adventure Author project and developed based on observations drawn from the design studies outlined in Chapter 4. The scheme is designed to be used by educators and researchers with knowledge of the game creation process, as it assumes some familiarity with basic game creation terms and concepts. It is targeted specifically at narrative-based games created by young people aged 11-15.

The requirements for the scheme were that it should allow an educator or researcher with experience of game creation projects to assess games efficiently and produce numerical scores across a range of categories which reflect the quality of the game. The first step was deciding on the key categories for inclusion in the scheme. This was carried out with reference to categories used within the Adventure Author software. A default worksheet for use with their Comments Cards software was created by the AA literacy specialist and included twenty-four questions

designed to support peer evaluation of games¹. These questions were chosen as helpful aspects for young designers to focus on when giving formative feedback on their peers’ narrative-based 3D games. These provided a good starting point for the scheme as they covered a number of important aspects of narrative-based games created by young people using the NWN2 toolset.

However, they were not intended for use as an assessment schema, and were aimed at young people rather than teachers. To form the basis of the new scheme the existing questions were examined and condensed into nine overall categories of approximately equal importance to game quality: Storyline, Visual Interest of Areas, Player Guidance, Player Purpose/ Goals, Player Choice, Characters, Dialogues, Imagination and Challenge. The nine categories each deal with different aspects of the game, but in some cases there is a small amount of dependency and crossover between categories. For example, Storyline and Dialogue are separate categories, but in order to receive the highest mark for Dialogue, some game conversations would need to be well-integrated into the storyline of the game.

The next step was providing support for the assigning of numerical scores for each category. A scale of 0-10 was chosen for scoring, and descriptors were written to reflect what a game should be like to receive full, half or no marks on each of the categories. These descriptors were generated based on my experiences of witnessing the creation of one-hundred and twelve games during the design studies. The key characteristics observed in this set of games, and the range of quality levels observed during creation were used to inform the descriptors. The descriptors were designed to help the rater judge what a game should be like in order to warrant certain scores on each category with a view to promoting a reliable and repeatable approach to scoring.

A single mark representing the overall quality of the game can be calculated by summing the individual numerical ratings for each category, giving a mark out of ninety.

Additionally, common game features observed during the design studies were used to create a series of tick boxes to record the presence or absence of noteworthy elements. These allow further analysis of the features which are embodied in games of varying levels of quality, and encourage the rater to consider all relevant factors when assigning the numerical score.

However, these elements are not necessarily positive or negative, so no meaning can be interpreted from the number of ticks given (for example, ‘utilitarian conversations’, one of the tick boxes in the Dialogues category, only seem to exist to make a scripted event happen). In two categories the rater is asked to give short descriptions of examples of game elements of the corresponding category, instead of using tick boxes.

1available at http://judyrobertson.typepad.com/adventure_author/comment-cards.html, last accessed 21/06/2011

The scheme was sent to the AA literacy expert for feedback, and some wording changes were made as a result, as well as the addition of a further tick box in two of the categories. The categories and descriptors are summarised in the following sub-sections, and the final scheme marking form is shown in Appendix C.1.

6.2.1.1 Storyline

The rater is asked to give a score out of ten which reflects the extent to which the game has a clear storyline. The level descriptors specify that for a game to receive full marks on this category there should be a clear storyline running throughout the game which is supported by the visual design of the areas, characters, conversations, buildings and objects. To receive half marks there should be some story ideas communicated to the player through some elements of the game. For a game to warrant receiving zero marks on this category there would be no evidence of any story ideas in area design, character or object choices.

The tick boxes in this category record whether or not storyline is conveyed through the following elements: terrain/ texturing or tileset of area(s); objects placed in area(s) (e.g. trees, buildings, furniture); character appearances; character behaviours; character names; area or object names; character conversations; text on objects (i.e. signposts); descriptions of objects.

The rater can also tick ‘other’ and record the presence of additional elements used to convey storyline.

6.2.1.2 Visual Interest of Areas

The rater is asked to give a score out of ten which reflects the extent to which the game areas are visually interesting. The level descriptors specify that for a game to receive full marks on this category the game areas should be very visually interesting and should successfully convey a convincing sense of the environment in which the game is set. To receive half marks, areas should have some interesting elements. There should be some sign of a coherent idea about the locations in the game, but there may be elements which are incongruous or seem to have been placed without thought. For a game to receive zero marks on this category the game areas would be very sparse and very little attention would have been paid to landscaping or decorating.

The tick boxes in this category record whether or not the designer has made use of the following in their area design: raising or lowering ground to create a landscaped terrain; different textures to create pathways and other features on the ground; trees and vegetation; buildings and other manmade structures; creatures which appear to have been added as scenery or ‘extras’; water;

special effects (e.g. fire or light bolts); identifiable locations which help to emphasise an episode in the game (e.g. a dragon’s lair). The rater can also tick ‘other’ and record the presence of additional elements related to the visual interest of areas.

6.2.1.3 Player Guidance

The rater is asked to give a score out of ten which reflects how well the designer guides the player about where to go. The level descriptors specify that for a game to receive full marks on this category the player should be guided in subtle but clear ways which are seamlessly

integrated into conversations or area design elements (such as paths or gaps in mountain ranges etc.) To receive half marks the player must be given some clues as to which way they should go, but these might be hard to understand or the clues themselves might be hard to find. For a game to receive no marks on this category the player would have been given no idea about where the interesting elements of the game are or how to find them.

The tick boxes in this category record whether or not the designer has made use of the following to guide the player around the area: subtle hints in conversations; explicit directions in

conversations; pathways; signposts; text in objects (e.g. a book); landscaping elements such as a clearing in trees or a valley. The rater also has the opportunity to tick ‘other’ and record the presence of additional elements used to guide the player.

6.2.1.4 Player Purpose/ Goals

The rater is asked to give a score out of ten which reflects the extent to which the player has a clear purpose or goals in the game. The level descriptors specify that for a game to receive full marks on this category the player should be given clear information about and motivation for their goals. To receive half marks the player should be given some clues as to what their objective might be, either through conversations, behaviours or area design, but other elements might be left unexplained (for example, the reasons why certain characters are their enemies and must be defeated.) For a game to receive no marks on this category the player would have no idea about where they are, why they are there, who is on their side or what they should do.

The tick boxes in this category record whether or not the designer has made use of the following to give the player information about their goals or objectives: subtle hints in conversations;

explicit explanations in conversations; character behaviours; text in objects (e.g. a book). The rater can also tick ‘other’ and record the presence of additional elements used to give the player information about their purpose in the game.

6.2.1.5 Player Choice

The rater is asked to give a score out of ten which reflects the extent to which the player is able to make meaningful choices in the game. The level descriptors specify that for a game to receive full marks on this category the player should be able to make choices which make a difference to how the game narrative proceeds. The player should also have a real sense of agency about elements such as which quests to tackle and which characters to befriend and which to make enemies. To receive half marks the player should be given some choices in elements such as which way to proceed, which areas to explore or how to respond to characters, but these choices

might not have a noticeable effect on how the plot proceeds. For a game to receive zero marks on this category the player would have no choice at all in how to proceed through the game.

The tick boxes in this category record whether or not the player has the opportunity to make choices of the following kinds: taking different routes through an area; choices of what to say in a conversation; choices of what to say in a conversation with noticeable consequences. The rater also has the opportunity to tick ‘other’ and record the presence of other ways in which the player can exercise choice within the game.

6.2.1.6 Characters

The rater is asked to give a score out of ten which reflects the extent to which there are interesting characters for the player to meet. The level descriptors specify that for a game to receive full marks on this category a number of the characters in the game should have

interesting personalities or intriguing motivations, and be well integrated into the storyline. To receive half marks at least one of the characters in the game should have an interesting

personality or goal which the player finds out about through conversations, appearances or behaviour. For a game to receive no marks on this category none of the game characters would have anything interesting about them in the way they look, act or what they say.

For the second section of this category the rater is asked to pick two game characters and explain why they are interesting or uninteresting.

6.2.1.7 Dialogues

The rater is asked to give a score out of ten which reflects the extent to which there are

interesting dialogues for the player to engage in. The level descriptors specify that for a game to receive full marks on this category a number of the characters in the game should have

conversations which the player can engage in and which help advance the main storyline or sub plots in some way. To receive half marks at least one of the characters in the game should have a conversation with some element of interest. For a game to receive zero marks on this category there would be no characters the player can speak to in the game.

The tick boxes in this category record whether or not some conversations in the game include the following elements: humour; convincing dialect/language chosen to illustrate character’s background; good descriptive language; plot-advancing information; ‘utilitarian’ conversation (only existing to serve the purpose of making something happen through an attached script).

The rater also has the opportunity to record the presence of other notable conversation characteristics.

6.2.1.8 Imagination

The rater is asked to give a score out of ten which reflects the extent to which the game shows imagination. The level descriptors specify that for a game to receive full marks on this category the game should have an imaginative storyline and inventive elements which are well-integrated and support the narrative. To receive half marks the rater should judge that the designer has had some interesting ideas and tried to incorporate elements which are novel, interesting or

humorous in the characters, objects, areas or storyline. For a game to receive no marks in this category no aspects of the game would be judged to show imagination or creativity and elements would appear to have been included at random.

For the second section of this category the rater is asked to give an example of an imaginative element in the game, if there are any.

6.2.1.9 Challenge

The rater is asked to give a score out of ten which reflects the extent to which the challenge level of the game is appropriate. The level descriptors specify that for a game to receive full marks on this category the whole game should be well-balanced with quests or missions which are pitched at the right level of challenge so that they are interesting but not impossible. To receive half marks the rater should judge that the designer has got the challenge level right in some places within the game. For a game to receive no marks on this category the game would be much too hard or much too easy (either the player keeps dying, or else there are no

challenges or threats).

The tick boxes in this category record whether or not the game includes the following challenge related elements: a challenge or quest to be completed; a battle which is too hard for the player (involves dying multiple times); a battle which is of appropriate difficulty for the player (can be won after a short to medium length fight). The rater also has the opportunity to record the presence of other challenge related elements which are present in the game.