Post Launch Changes

After launching, we addressed another issue that had arisen in Cycle Two, the “no results found”

page which hindered users’ ability to improve their searches (see “'No Results Found' Pages Hindered Search Improvement”, page 86).

Instead of displaying an empty page (see Figure 37), we now displayed a message indicating that nothing could be found, on a page that was consistent with the rest of the site (shown in Figure 41). We believed that the presence of other navigation elements on this page would make for easier recovery from a search which yielded no results.

5.3 Evaluate

The goal of evaluation was to observe unprompted and unassisted use. We would not be present when users logged in, and with the Link team on leave for most of December and January we had no feedback from students or church leaders. We therefore had to rely exclusively on automated logging.

5. Cycle Three – Website Deployment

Figure 41 New page presented to users when no results were found for a search. Previously the site hampered recovery by presenting a page with only an error message. In this version all of the navigation elements that

are present on other pages are visible.

Users were not required to log in, and we knew that some who were not intended beneficiaries had heard of it (other warehouse staff and their contacts). Taking data from non-beneficiaries into consideration could impair our research. They might have greater exposure to technology than our users, or have no personal interest in the content. We therefore ignored requests from a number of sources:

 Any request generated from our university or The Warehouse (identifiable by IP range allocated to those institutions)

 Requests generated from outside of South Africa (identifiable by IP geolocation)

 Automated requests, for instance from search engine crawlers (identifiable by user agent) We implemented the above rules as a Ruby language script which mined daily log files generated by our web server for information specific to this project, and produced CSV files which we reviewed in Microsoft Excel.

5.3.1 Results

We list here analysis of site usage before the end of 2011. Churches were only informed of the site’s launch after login names were created on 29 November, so we ignored requests made before then. We record 78 page views in 18 sessions. In Table 10 we list the ten sessions which consisted of more than one page view. These account for 70 of 78 page views.

Two sessions can be tied to specific students from Lavender Hill because they used allocated login credentials. Both had seen the site before, LW in Cycle One and JL in Cycle Two. Both sessions involved at least one search and viewing at least one entry. Both users accessed the site through a Blackberry device. User agent strings identify that both used the same model, but different

5. Cycle Three – Website Deployment

95 cellular service providers. Either separate devices were used or a single device shared but separate SIM cards used (common practice for sharing costly devices while maintaining separate identity [75]).

Table 10 Site visits made between November 29 and December 31 2011. Only five sessions involved even one search, and only two users followed links to entry content. These two could be identified as participants in

previous cycles from Lavender Hill, both using Blackberry devices.

Start

Date End Date #

Requests #

Searches # Entries Browser Logged In?

29/11/2011

Only five sessions listed in Table 11 involved any searches, and the only sessions which involved an entry being viewed are the two we have discussed already.

Including single requests, a maximum of 18 of the 55 Link students to whom we thought the site had been advertised could possibly have visited. Some of these might not be Link students, as filtering failed when Warehouse staff accessed the site from outside their offices. Repeat visits would also have lowered the number of unique visitors. For instance, of the eight single request visits, four were visits to the same page from the same ISP and using the same version of the Safari browser, but more than 24 hours apart. It was possible that these were all a single user.

5.3.2 Discussion of Results

The 18 visits recorded could represent as much as 30% of the 55 users that the Link team had registered on the site. However, only two instances definitely involved intended beneficiaries. It was a positive sign that they remembered how to use the site without further intervention, especially LW who had only seen it in Cycle One, before Cycle Two changes.

5. Cycle Three – Website Deployment

The remaining visits demonstrated minimal engagement (9 searches from 3 visits, and no entries viewed). These users were either uninterested in the content or encountered a problem that prevented them from proceeding further.

5.4 Specify Learning

5.4.1 Internet-Supported Intermediation

With only two students having viewed web content in over a month, the site was – disappointingly – not providing support for intermediation.

Negligible use indicated that no suitable secondary intermediaries had acted. Access constraints and financial considerations – which did not affect previous, controlled evaluations – could have been the main reason for the lack of activity. This would have been consistent with our understanding of access to the conventional web from other work (see Section 2.5). Successful operation in the previous cycle (and by two students in this) indicates that the cause was unlikely to be fear or a lack of skill. It was more likely that the one page visits were from people who were not the intended beneficiaries, who would be more likely to visit without interaction.

Unfortunately, two other possibilities existed which prevented us from drawing strong conclusions. First, news of the site’s launch may not have been communicated to all students. In Section 2.3, we discuss how both the Link team and partner church groups were involved in this communication. At least two visits were from students, but it was possible that others had not heard.

Second, the Link career guidance content may not have been of interest to the majority of students, or such content might be available from other sources. It would be important to investigate the other possibilities before assuming this to be the case, as the need for content had been a fundamental assumption of the programme since before our engagement began.

5.4.2 Adoption

The two Lavender Hill students’ use of mobile phones instead of computers indicated that the latter were harder to access. This would be a legitimate outcome of our investigation into adoption, although if this was easy, the question of why so few had done so remained. Some explanation might lie in the fact that we had not optimised the website mobile Internet browsers.

.