Practical Framework for Analysing Impacts of Online Engagement
In developing a framework for researchers to analyse their research, the Public Engagement with Research Online (PERO) team sought out feedback and suggestions from both academic
researchers and BCE practitioners by holding workshops, primarily within the University of Warwick. The case study (focusing on the CAGE research group) employed quantitative and qualitative analysis using web-based public discussion responding to research, framed within a theoretically and methodologically robust impact evaluation framework.
A few points were reiterated by many of those individuals that PERO worked with in the development of this tool:
• The tool had to be simple to understand.
• The framework needed to be explicit in how it was to be used.
• If necessary, the framework needed to identify sources of training in particular methods.
Therefore, we have proposed a framework, which is straightforward to implement, yet capable of producing robust and valid impact evaluation results. This framework illustrates the impacts of seeking to engage publics with an original piece of research online.
The framework consists of four steps:
1. (If appropriate), generate and insert Google Analytics code on relevant (e.g. institutional or personal) webpages communicating research ideas that are the subject of the impact evaluation.
2. Gather and validate keywords from Google Analytics, the web, and/or in-person events. 3. Use the acquired, validated keywords to gather online public discussions, in public spheres,
that reference key themes and/or original research, quality checking the resulting data. 4. Analyse randomly selected webpages / discussions qualitatively and/or through quantitative
content analysis.
Google Analy+cal
Code
• Register for Google Analy+cs and develop code.
• Insert Google Analy+cal code into all of the relevant web pages. It is best to do this before research is desiminated online. This allows the researcher to monitor the visitor traffic and keywords before and aAer the dissimina+on.
Capture Keywords
• Develop a list of keywords using the results from the Google Analy+cs output
• The keywords should be of keywords provided by Google Analy+cs and keyords taken from the visitor source pages.
• To obtain the keywords from the visitor source webpages, run the ar+cle text through simple text analysis tool.
Iden+fy Webpages
• Using the keywords conduct a web search to iden+fy web discussions. • Confirm that each selected web discussion is discussing the relevant case • Record the iden+fidy webpage's web address in a database.
Qualita+ve Analysis
• Randomly select websites from the database for qualita+ve analysis.
• For qualita+ve methods and analy+cs skills training, we suggest taking the ‘Evalua+ng Impacts of Public Engagement and Non-‐Formal Learning: Qualita+ve Methods’ workshop.
• To effec+vely measure the impact of dissemina+on online, the procedure of either qualita+ve or quanita+ve content analysis online discussions at these web addresses would need to occur prior to and following dissemina+on.
Case Study
This framework of online public engagement impact evaluation has been applied to a specific instance of online public engagement conducted by Professor Andrew Oswald. Oswald is an applied economics and quantitative social scientist at the University of Warwick. His research has focused on the economic and social determinants of happiness and well-being. In 2004, Oswald released a piece of research "Money, Sex, and Happiness: An Empirical Study" with David
Blanchflower in the Journal of Economics, which was then followed up with a number of offline and
online public engagement events, including:
• Organiser, International Conference on Happiness, Adaptation, and Prediction at Harvard
University, 2007
• "Happiness PowerPoint Talk: Esmee Fairbairn Lecture, Lancaster", November 2006
• "How did we get into the crisis, and how will human happiness be affected?" TEDxWarwick,
February 2009
• 'Happiness, Lottery Winners and Your Heart', University of Warwick, July 2009
• ‘Modern Society and the Economics of Happiness’ University of Warwick Podcast,
September 2011
Step 1: Develop and Insert Google Analytics Code
The first step in the framework requires the researcher setup the initial tools needed for analysis. The fundamental tool for online analysis is Google Analytics. Google Analytics is an analytical-‐software package initially designed for marketers and individuals without significant information technology skills. Therefore, the software is relatively simple to begin working and provides numerous free learning tutorials. The package has been constructed to assist users in conceptualising a number of metrics (e.g. number of visitors, unique visitors, sources, and landing pages), which illustrate how the website is operating. Google Analytics will generate detailed statistics on each of its metrics. These detailed statistics are particularly useful in building a foundation for online public engagement research, as researchers are able to track how their publics are engaging with their websites through landing pages, keyword usage, and sources.
There are a number of steps needed to begin this process. The PERO research team suggests that to return the most robust results, the Google Analytics code be established prior to the public engagement event. If the code is established and Google Analytics is collecting data on public engagement prior to the actual event it will allow for the researcher to conduct a higher level impact evaluation of their public engagement online.
In this case study, a Google Analytics account had not yet been established for Andrew Oswald’s research websites. The first action required was the establishment of a Google Analytics account. To setup an account on Google Analytics, navigate to www.google.co.uk/analytics and click on the button ‘Create an Account’ (Figure X). Once opened, the researcher will need to sign-‐up for an account (Figure X).
After setting up the account, the next step is to edit the profile, providing the website url and homepage name (e.g. index.html, home.html)(Figure X). In the case of establishing a profile for the Andrew Oswald personal webpage, no default page is named as the home webpage is www.andrewoswald.com, not www.andrewoswald.com/index.html. The profile names referee to either the University of Warwick CAGE website or Andrew Oswald’s personal webpage.
After establishing the profile, Google Analytics returns a unique tracking ID and tracking code (Figure 5). The code is the basis for tracking the visitors to the website. The statistics returned will be sub-‐standard without
Figure 2: Google Analytics – creating an account.
Figure 3: Google Analytics edit profile screen.
properly entering the code into the desired webpage. The code returned from Google Analytics was inserted into every webpage relevant to this study:
• The CAGE website homepage, including each of these webpages:
Andrew Oswald’s individual page
Happiness, Adaptation, and Prediction Conference
• Andrew Oswald’s personal website, including each of the webpages:
CV & Research History
Presentations
Academic Publications
Media & Non-‐Technical Articles
The computer programmers for these websites inserted the Google Analytics code into the HTML programme code for each of the webpages. However, inserting the code can simply be accomplished by copying the code from Google Analytics and pasting at the end of the desired webpage HTML code.
Once the Google Analytics code is inserted into each of the appropriate webpages, the researcher can move on to step two. If, however, the researcher is struggling to setup the Google Analytics code, a number of tutorials and videos can be found by navigating to http://www.google.com/analytics/learn/index.html.
Step 2: Gathering Keywords:
The second step in the framework involves the establishment of keywords to be used in the detection of public engagement with the research online. Keywords are those words that are most frequently used by the audience to engage with the research. In the case of Andrew Oswald’s research, the audience used words such as ‘religion’, ‘sex’, ‘ happiness’, ‘money’, ‘poor’, ‘wealthy’ and ‘Andrew Oswald’. These words are used frequently by the research audience to discuss the research and can be used to reveal further public engagement occurrences, which may not be otherwise apparent to the researcher.
There are three distinct techniques available for gathering keywords for online public engagement analysis. The first two methods involve using Google Analytics output. The first method is a direct capture of the keywords that are collated by Google Analytics. Google Analytics collates a list of keywords that the
researcher’s audience uses to find the specific piece of work. In this instance, Google Analytics collects all of the keywords that the audience for Andrew Oswald’s research use in search engines (such as Google, Yahoo and MSN) to arrive at his homepage on the CAGE website. To access the keyword data on Google Analytics the researcher must follow the following steps:
1. Log-‐in to Google Analytics
2. Click on the top toolbar tab ‘Standard Reporting’
3. Then, click on the left-‐hand toolbar tab ‘Traffic Sources’ and ‘Overview’
4. Summary source data will appear, including keywords; to get a more comprehensive list click on the link ‘View Full Report’.
This is the initial set of keywords the researcher’s audience is using when engaging with the researcher. However, this list of keywords is derived from a similar audience, one which is generally in search of specific information on the researcher (i.e. Andrew Oswald). The number of keywords selected is dependent on the depth of evaluation the researcher would like to pursue. In this case a list of 20 keywords was recorded in a spread sheet (Figures 6 and 7).
Figure 6: List of source keywords, provided by Google Analytics.
To build a stronger database of keywords, a secondary tool for gathering keywords is necessary. Google Analytics offers another option for source data: website source referrals. These sources are sites where the website visitor originated. Keywords can be gathered from these original webpages by analysing the text within these sources. To gather these keywords, the follow these steps:
1. Then, click on the left-‐hand toolbar tab ‘Referrals’ under the category ‘Sources’
2. A list of the top 10 referral website will appear. After removing each of the host websites, a list of 8 websites remained. Each referral webpage needs to then be analysed for keywords. Using simple text analysis software, several free options exist online; further, keywords become apparent and can be added to the previous list. The keyword list was cleansed of common words such as articles, pronouns, and conjunctions.
The rational for gathering keywords from your multiple sources is to guarantee all potential target audiences are included within the evaluation. Although a researcher may assume that they are aware of the keywords the audience would use when engaging with their research they cannot be sure of the paths of public engagement that may occur. For instance, with Dr Oswald’s research particular keywords can be identified by the researcher: Andrew, Oswald, happiness, money, economy, and Warwick. These are all terms that
Figure 7: The list of Google Analytics keywords cleansed and entered into a spreadsheet.
appear abundantly in the paper presented by Dr Oswald and in the number of events in which he participated. However, after using the keyword analysis, as described, the tool also pinpointed that the public also used terms such as: religion and infidelity. Without using an appropriate keyword selection process, the analysis of reach and significance would produce results that do not generate a comprehensive picture of the researcher’s online public engagement.
Step 3: Gather webpages referencing keywords and/or original research
The third step of the evaluation framework requires the researcher use the established keywords to gather discussion of webpages online to evaluate the public engagement with the research online. The objective of gathering webpage discussions is to collect the discourse that occurs following the publics’ engagement with the research. Using this method, the discussion can be analysed for common themes, cognitive changes, and the way discussion builds.
To begin collecting webpages a number of web searches need to be run on the keywords. Using Oswald’s research as the beginning point, top keywords were run through Google Search: Andrew Oswald and happiness. A Google Search on these keywords returned a list of 36,900 results, as this number was large a random selection of these results was implemented. Prior to performing the web search for the websites, a sample size was established of 100. As well, the web search was limited to a depth of 5 result pages, per web search (Figure 8). To randomly select for webpages, every second result page was analysed.
Figure 8: Google Search results from a web search of “Andrew Oswald” and Happiness.
Add these webpages, including the discussion to sample webpage database.
Results Analysis: depth of 5 result pages, analysing every second page
The subsequent steps were followed to build a database of public engagement online: 1. Open each webpage
2. Confirm the webpage relates to the research topic and/or that there is public discussion on the topic.
3. Add the web address to a sample database.
To build a comprehensive database, it is essential to repeat the web search using a variety of combinations of the keywords (e.g. ‘Oswald and sex’, ‘happiness and economics’, ‘CAGE and happiness’). Each of these keyword searches will return a variety of online public engagement with Andrew Oswald’s research. Using this search method will incorporate unpredictable aspects of public engagement that were not previously apparent. Case in point, “Money, Sex, and Happiness: An Empirical Study" was not predicted to be picked up by religious publics. Yet, religious audiences have used the research as a platform for discouraging greed. In addition to using each of the relevant keywords, using a variety of search engines and search options will return greater diversity of public engagement results. Google Search has a specified ‘Discussion’ option in which a search can be conducted on forums, discussion boards, and community pages. These results may not have otherwise been apparent in a web engine search. Using this method a diverse sample of online public engagement populated the database. The websites include discussions such as:
• http://www.physicsforums.com/showthread.php?t=31238 • http://britpolanalysis.blogspot.co.uk/2011/09/economists-‐disagree-‐shock.html • http://forums.windrivers.com/archive/index.php/t-‐62952.html • http://learning.blogs.nytimes.com/2011/03/15/what-‐are-‐some-‐recent-‐moments-‐of-‐happiness-‐in-‐ your-‐life/
Step 4: Analyse and Extended Evaluation
The final component of the framework involves the statistical analysis of randomly selected webpages through qualitative and/or quantitative content analysis. Quantitative content analysis methods contribute to the investigation and understanding of public conceptualization, whilst using broader qualitative analysis identifies the public discussions of unanticipated publics, who engaged with the research. It is at this stage that evaluation can be conducted to determine what, if any, influence has occurred following the online public engagement event.
The process of analysis begins by random selecting of individual cases from the webpage database for analysis. For this case study, Random.org was utilised to generate a random selection of 10 cases (of the original 100) (Figure 9). These figures were input into the Random.org software, which in turn produced a list of 10 integers between 1 and 100 (Figure 10). These integers were used to select the corresponding case number in the webpage database (Figure 11). After selecting the cases to be analysed, the discourse from each webpage (Figure 12) needs to be imported into a text document to be cleansed and prepared for analysis (Figure 13).
Figure 9: To generate the random numbers used to select case studies for analyses, Random.org was utilised to produce a list of 10 random integers between 1 and 100.
Figure 10: Random sample numbers, generated by Random.org.
Figure 11: Random case selection of Andrew Oswald’s "Money, Sex, and Happiness: An Empirical Study" online public engagement results.
Figure 12: Online Discussion referencing the dissemination of Dr Oswald’s research
Either quantitative or quality analysis can be conducted on the prepared data. To develop the tool set necessary for evaluating the online discourse, it is essential that the researcher (or evaluation consultant) have good command of the relevant analytical methodology. Therefore, under the recommendation of the research team, persons who wish to use either methodology should enrol in an evaluation training
programme, such as the ‘Evaluating Impacts of Public Engagement and Non-‐formal Learning’ hosted by the University of Warwick. These courses are concise and targeted towards a broad audience, interested in public engagement evaluation.
Case Study Analysis. In an exploration of Dr Andrew Oswald’s impacts online, following the online dissemination of his paper, “Money, Sex, and Happiness: An Empirical Study”, a series of distinct online discussions emerged. A preliminary analysis of one of these cases has been conducted using a qualitative analysis approach. This case focused on the discussion of the paper between 9 individuals in May of 2007. These individuals belong to an online Sudanese forum, which discusses a number of economic concepts, including: living in excess, currency and regulations.
This discussion consisted of a debate between the 9 individuals on the role of money on an individual’s happiness. A number of the participants discussed how Dr Oswald’s theory applied to a number of situations in which individuals with extreme amounts of money were unhappy or lonely, often due to the wealthy individual’s change in roles within society. One person offered the following example:
“I think you are lonely when you are isolated by some societal status/role. You become lonely at the top, not to be mistaken with lonely at the bottom (for the pariah and the renegades). The minute you are rich. Money is funny: here we are talking about extreme wealth. In Sudan if you are wealthy you have to manage many people's expectations and desires, and they never look at you the same ever.”
Whereas, three of the participants indicated their disbelief in the basic premise that the extremely wealthy were less happy than those in the middle class or even those in poverty. These arguments developed from a sense in which these individuals argue not all those with wealth are lonely. Additionally, it is along this theme that the individuals argue for happiness as a mind-‐set and not a class issue. The first statement speaks out against Oswald’s research principle of reducing the idea that money is necessary for lifetime happiness:
“Great, you tried the life of the rich and its goodies and you found it boring, empty and lonely. I just want a similar chance to prove for myself that money won't make me happy. I promise if while being rich I felt unhappy and lonely, I will donate my wealth to the people of Sudan.”
Whilst, other arguments support the concept of money does not equate to happiness. Instead support the idea that happiness is a state of mind.
“I really see this very clearly. It is not money that makes people miserable or happy. It is the people
themselves. Guns in the hand of thugs is crime, in the hands of law enforcement is deterrent to crime. Rain in the Sahara is blessing, in New York is misery. So, not all of those people in suburbia live miserable lives. Let's forget about money for a second and talk about happiness. Many people live their life thinking happiness is a destination. When I buy that house, marry that beautiful woman, have those smart children, then I will be
happy. When I get that promotion, get the corner office, and a marked parking stall, then I will be happy. Sadly they never arrive.”
These are examples of the text from which Oswald is cited. The participants in this case state their
understanding and attitudes regarding the happiness and economy theory; however, the participants did not offer methods of improving happiness, whilst also creating an economically secure society.
