Preliminary Testing

To better prepare for our final testing of the program on the INF1050 students, we decided to have a preliminary test on a small group of people. The main purpose of this test round was to see how users interact with the interface of the game, what parts of the game are

understandable/unclear and what they think about the potential for the game. Our hope was that testing on a small group of people before the final testing would make our game better, and hopefully we would find and correct small bugs we have overlooked.

8.1.1 Goal

The purpose of this test was not to gauge the learning potential of the game, but rather we focused on the interface and ease of use. As the test subjects explored the game we focused on their immediate reactions to the interface and game mechanics, trying to figure out what confused them and what elements of the interface that lead them to take certain actions within the game. The main purpose of this was to find out what parts of the interface that needs improvement before our final tests. Secondarily we focused on the game-play aspects of the game, and how they affected the general view of the game, giving us valuable input on how to balance the difficulty and progress. Whenever the test subjects encountered any of the

learning aspects of the game, we had either made sure the problems where trivial, or we gave them the answer, so that they could focus on the interface related to the learning aspects rather

than the learning aspect itself. The feedback from these tests will hopefully give us enough information to make the game-play and interface enjoyable, so that we can focus on the learning aspects for the final series of tests.

8.1.2 Evaluation technique

When we looked at the goals we had set for this first preliminary test, we found out that we needed to be in the same room as the testers to be able to pick up everything the testers actually meant and felt about our program.

8.1.3 Think aloud evaluation

Think aloud evaluation is a testing method where the tester and the person giving the test (test head) are sitting in the same room. After a brief introduction to the program by the test head, the tester is allowed to test the program at their own leisure. The tester is encouraged to tell their thoughts about the program aloud while examining the program. The test head can then sit and evaluate the progress of the tester, while at the same time picking up the testers

comments and emotional body language. A good thing with think aloud testing is that the test head can easily guide the tester with comments and tips if the tester is stuck or pondering about something.

8.1.4 How the test was conducted

The test subjects were given a document that briefly described our master thesis and the purpose of this test. During the whole test procedure, we had one interviewer focused on interacting with the test subject, either to make sure they tested the right elements of the game, or to answer any questions. The other interviewer would take notes. After having read through the entire document explaining our project and their tasks, the test subjects were presented with the program, which they where free to explore on their own terms at first. Our hope was that the in-game help and explanations would be enough to guide them through the game, which largely seemed to hold true. As they tested the program we asked them to verbally describe what they where doing and why, and give general feedback on the game. After a short while of free form testing, we guided the subjects towards the tasks described on their info sheet. These tasks where designed to ensure that the test subjects explored all the different parts of the interface.

Test subjects

We performed the test on two persons with different backgrounds. We chose one master student from IFI and the other one is just finished with his bachelor degree from MatNat. Tester1:

Gender: Male Age: 25

Degree: Just finished bachelor program at MatNat (UiO)

Prior knowledge related to our field of work: Taken the first basic computer programming course (INF1000) lectured at UiO. Know the basics about how a computer is used.

Tester2: Gender: Male Age: 24

Degree: Currently writing his master thesis at IFI, UiO

Prior knowledge related to our field of work: Played a lot of computer games, good at computer programming, taken several computer courses at UiO.

The master student told us he had good prior knowledge about programming and design of user interfaces. He has taken INF5270 which is a course much concerned with user interfaces and usability of computer programs and web pages. The bachelor student on the other hand had very little education about the things we are testing, but he told us that he had used a lot of different computer programs in his education. By using two test persons with such different backgrounds we hoped to get two different views on our program, and pick up different thoughts and aspects from the testers on our game.

The text that was given to the testers can be viewed in appendix B.

8.1.5 Results

When first presented with the game, both testers were curious on what the different elements in the user interface did. They both created an avatar with a name without problems, and entered the real game. The first problem both testers encountered was to find out what the

different location names meant. They quickly understood that it was a central issue to manage their time, but they had a hard time understanding where to move to perform certain tasks. For instance tester2 said: "Where do I start?" (Translated to English by authors) When given a tip that each place-button had a mouse drag-over tool-tip with a description on what action one can perform at the different places they seemed to understand a bit more. Still they both thought that the tool-tip was a bit slow and tiresome to use to find a specific location. After some minutes both testers understood the basics on how to move the avatar around the game, and where the action buttons for the places popped up etc. After a while, when they had gotten used to the user interface, they both liked the simplicity of it. Tester2 said: "It is not fancy or anything, but the issue here is the content, not the graphical."

Both testers also pointed out that there was no information at start-up on when the rent for the hired apartment was due, what it cost and where to pay it.

The mini games were well implemented according to both the testers. An issue tester1 mentioned was on the consistency in the layout of the questions. He meant that the questions always should come at the upper part of the question window, and again they lacked some prior information on how to answer on some of the questions. The multiple-choice question should be clearer on what it is that marked the right answer, and the drawing-question could use some more information on how to actually draw. Even with these comments, they easily managed to understand what to do and complete the questions in a good way. They also felt they lacked some feedback on how they did in the questions.

When it came to applying for work, both testers were very confused about where to start the application process. Tester2 got rejected from 4 jobs, and said "There should be a way to better understand what jobs one can get. They should be arranged after difficulty or

something". At one point Tester1 also got the message that he should become better at System Development in order to obtain the job, and asked if he needed just one more level or several and asked where he could obtain that information. Another issue they both complained about the during job application process was that if one failed to qualify to a job one jumped two sub-menus back instead of one.

During the gameplay the testers felt that they lacked a nice overview on their total progression and other small information windows on things like where they worked, what their current salary was etc. Tester2 also meant that every time an avatar stat changed during gameplay it

should be listed in the bottom log window. In general both the testers felt that there was little information on what they where supposed to do, and information about the different elements in the user interface.

When it came to playability it wasn't surprising to hear that they both felt the game to be boring and it lacked variation. For this testing there where just a few educational mini games implemented, and they felt they where just wandering around without a goal. Both testers pointed out several bugs that we were aware of, but these were mostly related to gameplay issues that we had not finished implementing yet at the time of the testing. Another major problem that we saw clearly throughout the testing was the issue on the difficulty balancing time, cost and income.

8.1.6 Evaluation

We noticed that we need to give a lot more information to the players. When we as developers play the game, we know a lot of the background mechanics and automatically play the game a certain way. The testers had no idea about the background mechanics and played a very different way, and this showed us many potential problems that can occur during gameplay. We need to add a more thorough introduction screen with the different elements in the user interface and about the gameplay in general. This could be added in form of a long explaining text or as a tutorial with images. Furthermore we need to change the names on several of the locations in the game in order for the name to represent what kind of place it is in a better way.

The mini games were understood well and liked by both the testers. We also showed them how easy it was to add new questions to the program, and they really liked this feature. Still we need to do some small adjustments to give all the questions the same layout in order to preserve the consistency though out the gameplay.

The work application process really needs to be looked more into. We need to make it more intuitive for the players to understand which jobs they gave a chance at getting. The job application process should not be a guessing game as it is now.

All in all we need to add much more information about several small things in the game. The testers mentioned a lot of issues we already were aware of, but also pointed out several others

that needs to be fixed before we test the program on a larger user group.