Data collection and analysis

I collected data for this qualitative study over a period of three consecutive years during which time I conducted evaluations at two national but independent annual science fairs, two kindergartens, a technology club, a conference, and my laboratory. The collaboration with the SciFest Africa (Scifest n.d.) and ScienceUnlimited (ScienceUnlimited n.d.) event organisers was mutually beneficial: For the organisers, our evaluation workshops added to the list of attractions while my research benefited by having access to a large number of children. For my research, the fairs provided access to children of mixed ethnicity and gender and from varying and undetermined social and financial standing. During these years, I was concurrently participating in the TekkiKids (Marais, Smith & Duveskog 2007) technology club research project that also had children as participants.

Evaluation sessions conducted at the fairs were documented using both video and photographs.

Sessions at the club, kindergartens, conference, and my laboratory were documented using photographs only. I archived all photographs and video recordings and later used these to compile conference papers.

For the second and third iteration, I interacted with participants and directed their activities in person. This put me in immediate physical proximity with each child. I adjusted my interaction approach based on my instantaneous observations. For example, I would offer assistance when the user was uncertain on how to proceed with an activity. At other times when the participant was clearly confident in her actions, I would retreat so as not to influence her thinking process. I could thus immediately identify design problems and confirm which design aspects worked well.

5.3.5.1 Science fairs

Thousands of children attended the two fairs each year. Both hosted a series of lectures and workshops with the number of participants at my Grahamstown workshops being approximately double that of Pretoria. Teachers, parents, and children made group and individual bookings. Two workshops per day was the norm. Participants at the science fairs were mostly school children on a school outing. Some participants were home-schooled and only a few participants were adults.

The event organisers required that workshop attendees book their places but I did not enforce this rule. Instead, I encouraged everybody to participate irrespective if they had adhered to the policy or not. Latecomers were not turned away either. A limit was set on the number of participants that could attend a lecture or a workshop. The number was based on either the size of the venue or set by the presenter. For my workshops, I set an official limit of 20 participants but I did not enforce this rule either.

Some sessions were oversubscribed yet at other times, as few as two children attended. I adapted the activities in all cases to accommodate the number of participants at a session. When only a few children attended the workshop, I adapted the activities to allow each more time to interact with the artefact. However, when a workshop was oversubscribed not all children had an opportunity to engage with the artefact or complete the questionnaires.

On average, four children per workshop session directly engaged with the artefact. With eight sessions per event per year, the average number of direct interactions was thus 64 children for each of the first, second, and third designs respectively. As many as 20 adults per year interacted with the programming environments outside of the formal sessions.

I set a time limit of one hour per workshop session. I considered this time to be a reasonable balance between mental and physical exhaustion for both the participants and researchers, and the time needed to 1) obtain written participant consent, 2) introduce the artefact, 3) for up to seven children to interact with the programming environment, and 4) for them to complete written questionnaires.

Children attended the workshops either as individuals or in groups. Some were home schooled, others attended private schools, and yet others attended public schools. Often all the participants would be from the same school while at other times the attendee composition would be a mix. I had no control over the number of participants that attended the workshops nor their age, school category, gender, or ethnicity. The ages of child participants ranged from three to 18 years.

I initially evaluated the first artefact at the Pretoria science fair. Based on the learning that emerged in Pretoria I adjusted the workshop format in anticipation of the workshops to be held in Grahamstown later that year. An assistant introduced the first programming environment to the participants and helped the children construct programs.

I evaluated the second artefact in both Pretoria and Grahamstown. For this evaluation, participants were not only users and testers but they were now also informants. As informants and before introducing them to the programming environment, I requested the participants to sketch signs of their own design that represent car motions (the first iteration had identified a need for this

activity). Approximately 50 designs were legible and I classified and grouped those by hand according to similarities. Some designs were not processed as they included written text. The balance of the collected informant data are not usable for reasons that include unfinished drawings and incomprehensible markings.

For both the second and third artefacts, I interacted directly with the participants by introducing the programming environment, directing the evaluation activities, and assisting the participants as they interacted with the system. This put me in immediate physical proximity with a participant. I also interacted with him and adjusted my approach based on my instantaneous observations. Not only was I able to immediately identify design problems using this approach but also confirm which ones were good. I thus had little need to access the videos and photos taken by research assistants.

The evaluation conducted in the first iteration highlighted certain problems and my initial goal with the second analysis was to determine how well I had addressed those problems. The second artefact was therefore analysed according to three initial themes: First, I was interested in the way the user interpreted the signs. I also wanted to observe how the user manipulated the objects. The third theme was to gauge how successful the users were in constructing programs. I based my analysis of the third artefact on these themes. At the same time that I collected data on all the artefacts, I compared it with those already allocated to a theme and I created new themes when data did not fit an existing one.

5.3.5.2 TekkiKids

TekkiKids was a joint research project funded by the CSIR Meraka Institute, the University of Pretoria, and the University of Joensuu in Finland. It was modelled after the University of Joensuu’s Kids’ Club and focussed on primary school children aged nine to 12 years. The aim of this three-year project was to encourage children to follow a career in science, engineering, and technology. To this end, three groups of 12 children each were established on a voluntary basis. Six children from two schools respectively comprised one group. Participants were pre-selected by their respective schools and we requested that both well performing children and those who were not top performers be selected. Thirty-six children thus participated in the project activities. The fortnightly technology club sessions lasted two hours. In contrast to the science fairs, the children that participated in the club were selected based on their school’s recommendations.

In addition to the science fairs, the second artefact was also evaluated at the club. The second artefact evaluations and design solicitations at the club followed the same format as for the science fairs. One noteworthy difference is that the same children attended the workshops over many

weeks and were thus at ease with the environment, each other, and me as session coordinator.

Interactions were therefore informal and relaxed.

5.3.5.3 Kindergartens

In addition to the science fair evaluations, the third artefact was also evaluated with the help of approximately 40 children at two kindergartens. Two researchers accompanied me as we conducted evaluations in Pretoria and the Johannesburg area. The evaluation design differed from those at the science fairs in that only one evaluation was conducted per kindergarten and much more time was spent explaining the artefact to the children.

5.3.5.4 Interactive exhibition

The fourth artefact was evaluated using data collected during an interactive exhibition (Smith 2009c) at an interdisciplinary conference in Italy. I observed approximately 20 children and adults as they used the programming system.

I did my analysis along four themes. The first considered how well the magnet-based positioning mechanism functioned. I considered the design to be a success if the user placed the disk on the surface and it remained centred. A second requirement was for the disk to return to its original orientation when slightly turned. I also wanted to determine if the repurposed materials remained functional: Had any artefact part come dislodged during use the event would have been recorded as a failure. Finally, I needed to confirm that the tangible program could produce a result on the computer screen.

5.3.5.5 Laboratory with children

Another opportunity to observe how children interacted with the third artefact presented itself when a group of about 100 children of ages 16 to 19 visited our institution. Of this group, approximately 10 directly participated in an informal evaluation session while the other children looked on. Since I was the only researcher at this event, I requested the children to take photographs using my still camera. This freed me from taking pictures and allowed me to direct the proceedings and interact with a participant as he constructed a program. My close interaction gave me an opportunity to detect design problems and identify which design aspects worked well.

5.3.5.6 Laboratory without children

In the laboratory, I compared the fifth and final artefact with a prominent tangible programming environment. I did this by identifying between the two environments language structures that coincided and those that differed. I then constructed program segments using both languages.

I analysed the data according to three criteria. First, I wanted to determine the user effort required when converting a personally meaningful object into a program element. To answer this question, my initial objects were constructed using cardboard pieces cut from household product packaging. I then printed markers and attached these. The software was then put in Mapping mode and the object mapped to either an action or a numerical value. I next experimented with small wooden cubes and dowels procured from an arts-and-crafts store. The cubes and dowels are from a child’s toy and fit together to form interesting structures. I again applied optical markers to the cubes. I also attached personally meaningful paper “flags” to the dowels. This and other tangible assemblies were then mapped to either functions or parameters. Finally, I constructed programs using the objects.

The second criterion was to strike a balance between the object size and the number of objects that could fit the construction surface. A balance is important for two reasons: The marker must be large enough to be identifiable using the camera and yet the objects must be small so that many would fit.

I found the balance by first fixing the size of the construction surface assembly and I then determined through inspection the smallest marker that could work with the image recognition software.

Finally, I set out to determine if my artefact could be used to implement program structures of the prominent tangible programming environment. To this end, I chose instruction set elements from that environment and discovered that some constructs mapped directly to my own language and yet others had to be manipulated into a compatible structure.