Factorial Analysis and Experiment Design

Initially, our evaluation of the experiment intended to do a complete factorial analysis of both Pac-Man and ghost factors simultaneously. Ideally, having both sets of factors in the analysis would provide insight into how to improve the tactics of both the player and opponents. However, this led to high k values (maximum of 20) which could not

be analyzed utilizing commercial statistical programs such as SPSS or Minitab due to memory limitations. The commercial software was capable of doing a full factorial analysis for a maximum of 8 factors, which was well below our desired interval. To compensate for this large amount of factors we began investigating other methods of analysis based on fractional factorial designs.

The rst issue of fractional designs for our experiment was confounding, in which the value of some of the eects cannot be determined, only the combination of their inuence can be. This was potentially a large issue for our experiment as we required the knowledge of how each factor eected the game. The intent for an adaptive sys- tem is to keep changes minimal and subtle, the factors can not be separted we risk drawing attention to the adaptive system because we must change a higher number of factors, it also increases the risk of altering an unintended aspect of the game. With confounding of factors we potentially diminish the ability of the adaptive system to perform more detailed adjustments. The second issue of using fractional designs for our experiment was that fractional designs are based on the assumption that higher order interactions have small eects. However, our algorithms were specically se- lected with the intention of producing higher order interactions and emergent behav-

ior. Intuitively we suspected and then later observed in early runs of our experiment that the algorithms we selected based on producing emergent behavior would pro- duce higher order interactions with large eect values. A nal reason why fractional designs were not well suited for our study was that the advantage of that method is removing a non-signicant factors during the early stage, although this method could be used to identify insignicant factors to be removed our intention was to retain all factors throughout the process in case they were signicant to other factors settings or players.

Ultimately, the inability to use either large values with a full factorial or frac- tional factorial design led to the restructuring of the problem space to create a more manageable size. Thus we restructured the problem by dividing the players factors and the rest of the game factors into subsets, thus creating groups with manageable factor sizes. We separated each algorithm set into groups of 10 factors, which was still above the commercial term limit of 8. However, by exploiting the properties of full factorial design, we were capable of developing our own program that would initially allow us to investigate up to 15 factors. The separation of factorial design is one of the use properties we utilized to perform our analysis. For instance, if the design had 12 factors the separation of the design to 10 factors would create 22 or 4 cases. In

these 4 cases, the values of the 2 separated factors are implicitly dened in the model. Table 5.2 illustrates the implied values of the 11th and 12th factor in each case. In this example, each case produces its own model, which results if 4 models and every term having 4 values. The limitation of separating the design is that we do not have access to the intersection values of the separated factors. Thus the main eects and any interactions between factors and factor 11 or 12 are unavailable, unless the design is reconstructed.

We selected to use groups of 10 factors for several reasons, the rst being that our initial factorial analysis with higher factors values showed very low R-Sq results. This was partial due to a limitation of the commercial software which limits the model to a maximum of 127 terms. Secondly, grouping to 10 factors drastically decreased the computational time required to perform the analysis. The decreased

Factor 11 Factor 12 Case 1 -1 -1 Case 2 1 -1 Case 3 -1 1 Case 4 1 1

Table 5.2: Example of separating the factorial design, the values of factor 11 and 12 are implicit applied to the model's coecient and terms.

results of the R-Sq values was expected because as the number of terms increases the commercial limitation of 127 factors becomes a greater constraint, as additional signicant eects may occur outside the top 127 terms. Our program utilized the fact that terms eects can be independently evaluated using the eects table and the response variable. Using this property we avoided the expensive computation memory problems at the expense of computational speed. The eects and sum of squares (SS) where independently calculated then recombined and sorted to dene an ordered list of the terms with the greatest eect on a specied response variable. We loaded the ordered terms into our commercial software (Minitab) for this project to calculate the signicance for the terms and R-Sq values for the model. At this stage, we encountered another limitation of the commercial software, Minitab could only included a maximum of 127 terms per model. Although our experience with SPSS allowed a model to load above 200 terms, even performing analysis on models with 127 terms in SPSS took signicantly longer. This limitation occurs only for the commercial software as we are capable of including all terms into the adaptive game models. However, despite the limitation of 127 terms per model we still produced adequate R-Sq values for the algorithms. Depending on the results of the lack-of-t tests the terms excluded from the model may not to contribute signicantly to the experiment results, and thus 127 term could produce adequate models.

The restructuring of the player factors created a couple of advantages on top of allowing us to complete the analysis. This organization allows each session to be treated as an individual player, and eases the creation of models for the adaptive models of each algorithm. The factor separation allows for an eective method to investigate alterations to game objects or game design that are too large or noticeable

to be included in the adaptation process, such as level design changes or attributes that are viewable to the player. The separation also allows us to organize and observe player information in a similar fashion to how it would be received in an online setting, which is a potential logical progression of this research. In addition this organizational system is better suited for extracting player information, which will be demonstrated in the proof of concept adaptive system.