Pilot study

Two members of administrative staff in the Design School were selected for the pilot study. They were also selected based on the requirements of being computer literate and not a designer. A date was confirmed with both of them for the session.

In order to do the final design of the study, a revision was done to the tests based on the pilot study. The issues, found in the pilot study, were that:

1- A slide should have been added to the PowerPoint to introduce a generic MC toolkit to the participants, and another slide to identify most of the features in one place. Therefore, they would understand the area and the features better.

86

2- Two of the features included in the group 1 (see Table 6-1), were not appropriate for test 1 for ranking. In other words, two features in group 1 had to be revised to be eliminated (Table 6-3), these were: ‘DoF’, and ‘need-based or parameter-based’. This is because they belong to group 2 (see Table 6-2) based on the role of grouping, explained earlier.

3- There are many features needed to be ranked according to each other, and this is a long list of fourteen features (group 1 (see Table 6-1)), which may require much time and effort in comparison to test 2.

4- To do the statistical analysis later, a weight should have been added to each answer choice. This was not considered in the pilot study. However, it could be done after the pilot had been performed.

5- As mentioned earlier, the question for test 1 should have been prepared earlier before the sessions to make it consistent without conveying different meanings to participants. It is not easy to individually explain one simple task to number of people and express the same meaning. However, in the session held, the author’s meaning was expressed to the participants as clearly as possible, and they understood the meaning well. For instance, for test 1 (for the ranking), it was explained to participant 1 to put the features in order, based on how important she thought they were. Participant 2, was asked to sort the features in order by considering how much he preferred them. Considering that, therefore, they should have been asked one single question. For addressing those problems, the following amendments were done in each category sequentially:

1. For the method of presentation, two slides were added to the PowerPoint file. One of them included what a MC Toolkit is, and the other one gave an overview of all the features in one place.

87

Table 6-3 Revised Group 1 (Revised group of features of group 1 (Table 6-1)) Revised group 1 of features

Visual feedback Virtual use of object

Physics providence Selecting manufacturing and assembly method

Users’ design access Reference object

Technical information Guidance (help in using the system) Validation with feedback Click and show option (interactivity)

Offline access Price update

Material selection Libraries of modules and producible Side by side comparison Flexible navigation

3. A revision was done to test 1; It was decided to merge the two parts of test 1, instead of them being done separately. The process, which was used for test 1, was demonstrated in Figure 6-2.

The interpretation of scoring for test 1 is as follows: each feature was scored between 1 and 13. Next, the eight highest scored features were separated out. Then, these eight were ranked 1 to 8. Next, the lowest ranked feature of these eight was assigned a weight of 6, and the highest ranked feature was assigned a weight of 13. The eight lowest scored features that remained were assigned a weight between 1 to 5, based on their ranking within themselves. Therefore, at the end of test 1, each feature was assigned a weight from 1 to 13.

88 Figure 6-2 The procedure of ranking and scoring for test 1

4. As explained before, a weight was assigned to each answer choice.

5. For the statistical analysis, a weight was assigned to each feature. The weight was then used for the mean or other basic statistical calculation. After collecting data in the survey, the mean value for each feature was calculated as follows:

𝑴𝑴𝑴𝑴𝑴𝑴𝑴𝑴 =

𝑴𝑴

=

∑ 𝒘𝒘𝑴𝑴𝒘𝒘𝒘𝒘𝒘𝒘𝒕𝒕𝑴𝑴𝟏𝟏

𝑴𝑴

Where n is number of participants. Calculating the mean from the samples, the features were finally sorted based on their mean value.

For evaluation sessions:

6. In order to ask a similar question from all the participants, the question was written down precisely in order for them to be told exactly what the enquiry was about.

Some of the comments of the participants were as follows. During test 1, participant 1 suggested that he would re-arrange the order if he was supposed to do the ranking again, after the first try. Also, he came up with this idea of putting other designs in other users’ design library- for which the permission of the co-designer (the customer)

89

may need to be asked- a discount as an incentive may encourage the users to give the permission. Therefore, other users’ design can be made available to others as well. In addition, he pointed out that the ranking depends on the product.

Participant 2 is also mentioned that the ranking depends on the product. She explained that she ranked in second place since MC toolkits were new to her, and she needed help to know how to go through the system. The average duration of each session was around 40 minutes.

In according with what has been done for the pilot study, the first study was conducted similarly with the corrections done.