5.4 Data Collection Plan
5.4.1 Instruments Used for Intervention Preparation
In order to guide class instructions, didactic units are used, regarding skills, competencies, time managing (how much time to devote to each activity) and the contents. Writing it, different methodologies and exercises proposed in different studies for developing required innovation competencies are taken into account as well as what students have to learn during the course and how classes are planned. Didactic units are used to specify (and differentiate) the goals, content, methodology and evaluation scheme of each topic in class planning (Gómez & González, 2006).
A template used for development of a didactic unit consists of eleven elements which include:
1. Introduction
2. Didactic Objectives 3. Evaluation Criteria 4. Contents
5. Activities and Exercises 6. Methodology
7. Attention to Diversity
8. Organization of Space and Resources 9. Evaluation Procedure
10. Evaluations Instruments
94 11. Session Organization
5.4.1.2 Semi-Structured Interview with Experts
The interview is prepared taking into account the elements from the didactic units to which the experts have access before the interview. On the basis of previous studies each element is assessed based on the sub-dimensions (indicators) described in the Table 5. Each indicator is covered with a set of specific questions.
Interview Indicators per Study Objectives
MEI Competencies to Develop - General viewpoint on whether the course helps to
Final Project - Opinion on the proposed
final project Final Evaluation - Opinion on proposed
evaluation and assessment
95 5.4.2 Instruments Used for Intervention Evaluation
5.4.2.1 Youth Innovation Skills Measurement Tool (YISMT)
Currently, several measures exist to measure the innovation capacity. One of the best known measures of the capacity to innovate is Kirton’s innovation adaptation (KIA) measure (Kirton, 1976). However, it has been designed to be applied in a workplace context and is limited to identifying those employees with strongest entrepreneurial attitudes. Other instruments developed later (e.g. Robinson, Stimpson, Huefner, & Hunt, 1991) also concentrate on measuring entrepreneurial attitudes and not innovation capacity and skills as whole which are not limited to only entrepreneurial ability.
A study commissioned by NESTA (the National Endowment for Science, Technology and the Arts), an organization in the UK whose aim is to transform the UK’s capacity for innovation has proposed a Youth Innovation Skills Measurement Tool (YISMT) (Appendix I) which measures general skills needed for innovation within young people (Chell & Athayde, 2009). This tool perfectly fits the objectives of this research and is utilized during pre-test and post-test stages.
Through previous research the creators of the YISMT have identified the generic skills which underpin innovation and which the tool aims to measure (Table 6). Each variable is composed of a number of items identified by the authors of the instrument (Appendix II).
YISMT Variables
Variable Description
Creativity Imagination, connecting ideas, tackling and solving problems, curiosity Self-efficacy Self-belief, self-assurance, self-awareness, feelings of empowerment, social
confidence
Energy Drive, enthusiasm, motivation, hard work, persistence and commitment Risk-propensity A combination of risk tolerance and the ability to take calculated risks Leadership Vision and the ability to mobilize commitment
Table 6
Additionally, the authors of the instrument distinguish between different types of innovators. Respondents are asked to indicate how much they agree or disagree (on a scale of 1-7) with a series of statements about pursuing five different innovative
96
pathways: a social innovator, a cultural innovator, an economic entrepreneur, an inventor, and an innovative entrepreneur. If respondents score 7 (i.e. strongly agree) on any of the dependent variable statements (which measure future intentions towards an innovative career pathway), they are categorized as having a stronger intention to pursue an innovation pathway. If respondents do not score 7 on any of these statements, then they are categorized as having weaker intention of pursuing an innovation pathway (Chell & Athayde, 2009, p. 33).
Types of Future Innovator and Associated Dependent Variable Type of Innovator Statement of Future Intention to Be Innovative
Social Innovator I intend to do something no one else has ever thought of before to bring about positive changes to society or the environment.
Cultural Innovator I intend to design or create something new, such as in music, software, dance, TV or fashion.
Economic Entrepreneur
When I leave college I intend to spot opportunities to make a lot of money.
Inventor I would like to invent something that is new to the world.
Innovative Entrepreneur
My ambition is to set up a successful company that offers something completely new.
Table 7
The YISMT aims to address a gap in educational assessment by offering a robust measure of young people’s innovation skills. The research to develop the Tool aimed to (1) identify robustly the important components of innovative capability in young people; (2) demonstrate ways of revealing this capacity; (3) understand innovative behavior within secondary schools and sixth form colleges; and (4), where appropriate, identify any individual school or college initiatives that would promote the development of innovative attitudes and behavior.
Even though the YISMT has been piloted and tested in schools and six-form colleges with participants of the study being aged between 14-19 years old, the authors particularly emphasized that the tool is addressed to young people without major working experience. Moreover, the items of the tool focus only on personal generic abilities without making reference to any specific learning environment since the tool has been used at schools as well as at colleges. The authors stated:
97
Innovation skills are generic and can be developed through science, arts and vocationally-oriented subjects, as well as through a wide range of extra-curricular activities. The Tool can be used as a benchmark to measure the impact of enterprise and innovation programs and initiatives (p.4).
5.4.2.2 Rubric for Data Assessment and the Use of Creativity Exercises
In this study rubric is used to assess pre-test and post-test exercises which evaluate innovation capacity of MBA students before and after attending the course designed for developing MEI competencies.
The exercise involving studies in this research consists in reading a case-study which presents a management problem related to business innovation (Appendix III). The case-studies are chosen in such a way in which they can be perfectly comparable to each other and thus, despite describing two different companies, can be equally used in pre-test and post-test.
The first case-study is about the Danish company Lego which produces children toys and, following some problems related to the costs of production, is transitioning itself into a strategy combining toys and digitalization. The second case-study is about a famous company Nintendo which is also related with toys and games, particularly, video games. Both case-studies are comparable in the following ways:
- Both case-studies describe the companies from the entertainment/game sector;
- Both case-studies provide an outline of the history of the company, explaining the challenges and problems it faced;
- Both case-studies describe the companies which are currently finding itself in transition towards considerable product innovation;
- Both case-studies provide an example of a company which is innovating by revolutionizing a user-experience. Lego – by offering an increasing variety of user configurable products in which users can modify or even design from scratch their own toys. Nintendo – by offering an innovative video game controller which enables the creation of games that mimic real-world movements like throwing, hitting, or aiming. In both cases, users and their new experiences are in the center of product innovation;
98
- Both case-studies describe the companies which came up with revolutionizing innovations within their corresponding market segments;
Further, the students are asked to answer the following questions:
1. Which kind of innovation do you find in this case study? Please identify process, products or services.
2. Which kind of process, services or products you consider sustainable for innovation based on this case study?
3. Which innovative ideas do you have for development of the company?
4. If you were innovation manager of this company what would you be doing? Please describe your actions and plans and ways of making your plans possible.
5. As innovation manager of this company, what skills you think you would need to have to be successful? Which of these skills you currently have? Which of these skills you don’t have and hope to improve?
The first two questions are related directly to the situations described in each case study. However, they help to figure out whether a student understands what business innovation means and how he or she understands the innovation process. The third question aims at understanding the level of creativity of each student. The fourth question allows seeing what kind of innovative management techniques each student can suggest. Finally, the fifth question aims at evaluating whether a student has awareness of the skills and competencies needed for innovation and whether he or she has a personal awareness of having or not having these skills.
The students are not given any further explanations or tips on completing the assignment apart from these questions. The first question aims to define how the students understand business innovation before and after participating in the course (content); the second question measures the extent to which the students understand the management practice of innovation (innovativeness of management techniques taught);
the third question contributes to the study of the creativity competence development as a results of the course (creativity); the forth question calls for solutions and innovative approaches (understanding of competencies needed for innovation); finally, the fifth
99
question assesses students’ self-perception and awareness of personal capabilities needed for innovation and expectations/results from the course.
Each student’s answer is rated by applying the Likert scale from 1 to 5, where 1 is a “Fail” – no answer is provided or the answer is not correct/ relevant to the question;
2 is “Poor” – an answer is provided but parts of it are not relevant; 3 is “Satisfactory” – an answer is provided, it is relevant but it is not developed in detail, the approach proposed is not innovative/creative; 4 is “Good” – an answer is provided in detail, it is relevant, but there is a room for improvement; 5 is “Excellent” – an answer is provided in detail, it is fully relevant to the question, the approach proposed is very creative.
Rubric for Data Assessment (Practical Exercises)
Indicator Measurement
Fail
1 Poor
2 Satisfactory
3 Good
4 Excellent Understanding of business innovation 5
Understanding of the innovation process
Creativeness of a solution proposed Innovativeness of management techniques proposed
Awareness of personal soft-skills
Overall level of assignment elaboration Overall level of adequacy of answers
Table 8
The creativity exercise used in this study is adopted from Cheung et al. (2006) who proposed a series of creativity-measuring exercises which contain verbal divergent and drawing production tasks as well as alternate uses tests. These tasks require students, for example, to list all the uses of a newspaper that they could think of.
Another example of the task is when the students have to list as many ideas as they can about effective ways of learning their major subjects, and ways of creating jobs in Hong Kong.
In this study the students are given a sheet of paper with several sets of parallel lines (Appendix IV). They are asked to supplement the lines by drawing something which would convert these lines into an image. The students should think of as many ideas as they can to get as many possible alterations of the lines and images.
The Rubric is used to assess the results of the creativity exercise. Each indicator in the rubric is rated by applying the Likert scale from 1 to 5, where 1 is “Very Poor”; 2
100
is “Poor”; 3 is “Satisfactory”; 4 is “Good”, and 5 is “Very good”. The following indicators are rated in the rubric when assessing the results of the creativity exercise:
- Number of drawings – a number of drawings a student came up with;
- Relevance – the extent to which the exercise was completed correctly;
- Creativity – the level of creativity approach applied to the drawings;
- Elaboration – the level of attention to detail in the drawings provided;
- Number of the same drawings in both exercises – measures the same/very similar drawings in pre-test and post-test creativity exercise;
5.4.2.3 Student Diaries
Similarly to Debreli (2011) in this study the diaries are used for the whole duration of the course to collect data related to students’ progress, self-perception of changes as a result of participation in the MBA course methodologically adjusted to innovation, feelings and thoughts as well as confidential feedback. It is communicated to students that the diaries should be as honest as possible with the purposes to improve the course in the future and under no circumstances affect their final grade. The students are asked to submit the diary after each class session. Because the main problems with using diaries as a data collection instrument are related to recording of the irrelevant information by the participants and lack of meaning in participants’ statements (Debreli, 2011), the students are asked to submit the diaries in a semi-structured format with the following categories:
1. What new have I learnt during this class session?
2. What this class session has made me think and feel?
3. Have I improved myself (my innovation capacity) as a result of this class session? In which way? If not, why?
4. Can I use in real life what I have learnt in this class session and where?
5. My suggestions for improvement of this class session would be…
The final entry of the diary is prepared by students in the following format:
1. What have I learnt during this course? What has especially impacted me?
2. How different have I become as a result of this course? What has changed in my mindset as a result of this course?
101
3. Please describe how you think the course has helped you to improve the following skills. If it didn’t help, please describe why?
- Creativity skills - Entrepreneurship skills
- Understanding of Modern Technologies - Management of diversity
- Soft skills such as Team-Building, Leadership and Communication - Change and risk management
4. My suggestions for improvement of this course would be….
5.5 Data Collection Procedure
5.5.1 Intervention Preparation
The preparation of an MBA course prototype design consisted in elaborating didactic units which included the methodologies believed to develop MEI competencies and as described above in the section 4.3.2. The preliminary MBA course design was then sent to the innovation experts by e-mail in order for them to get familiar with the course design in advance and before the interview. After that, face-to-face personal interviews were conducted with innovation experts upon prior appointment. The interviews were recorded and then transcribed.
5.5.2 Intervention Evaluation
The intervention was prepared for a group of 25 students enrolled during the academic year 2014-2015 in the MBA program at Global Business School Barcelona.
Different data were collected based on the nature of the instrument applied and as described below:
- Youth Innovation Skills Measurement Tool: The questionnaires were distributed among participating students (in paper) in the beginning of the first class session and at the end of the last class session.
- Pre-test – post-test creativity measuring exercise: The exercise was distributed among participating students (in paper) in the middle of the first class session and together with the quantitative questionnaire during the last class session.
102
- Pre-test – post-test exercise involving case-studies: The first case study was sent to students by e-mail straight after the first class. The students were asked to complete the exercise before the second class as part of the homework and send their answers back in word document. The second case study was sent to students after the last class session; the students had a deadline of one week to send their answers back required as part of their homework.
- Students’ diaries: The diaries were collected within one week after each class sessions and always before the next class session. The students were asked to write the diary as part of their homework after each class and send their diary each week to a specially designated for these purposes e-mail address.
5.6 Information Processing
5.6.1 Processing of the Interviews
The interviews conducted within the Phase 1 of the study were recorded and then transcribed. The qualitative data in English were then imported into the Qualitative Data Analysis Software – MAXQDA. Within the software the text codes were created according to the interview topics outlined in the Table 5. With the help of the software, the suggestions for course improvement and recommendations emerged and were then analyzed. After this comprehensive analysis and in accordance with the recommendations made by the experts, the initial didactic units which composed the preliminary course design were modified.
5.6.2 Processing of Data from YISMT Questionnaire
The quantitative data collected via YISMT questionnaires were entered into a database designed specifically for this study. Then the data were analyzed using the SPSS statistical software. The processing of these quantitative data was focused on descriptive analysis (mean, standard deviation) with the purpose to understand the sample characteristics; Psychometric analysis – which included scale internal consistency and reliability of scale dimensions by applying a Cronbach’s alpha and factor structure and exploratory factor analysis which was performed for informative purposes only considering a small sample size that was not enough to obtain replicable
103
results (Costello and Osborne, 2005); Correlations – in order to identify the relationships between variables; Paired t-tests were performed in order to observe differences within each before-and-after pair of measurements following the recommendations of Winter (2013) who demonstrated that a paired t-test is feasible with extremely small Ns if the within-pair correlation is high; Mann-Whitney test was applied for the analysis which required the study of answers that included all possible options within one question.
5.6.3 Processing of Data from a Rubric (Case Studies)
The pre-test case study of each student was evaluated first without looking at the post-test case study of a particular student. Each student’s answer was rated manually according to the rubric described in the section 5.4.2.2. by applying the Likert scale from 1 to 5, where 1 is a “Fail” – no answer is provided or the answer is not correct/relevant to the question. The data obtained were entered in a Microsoft Excel sheet. After the pre-test exercises of all students were evaluated, the evaluation of post-test cases studies began in a similar way. The data were entered in a different Microsoft Excel sheet. The means and change percentages were calculated by using a corresponding function within Microsoft Excel.
5.6.4 Processing of Data from a Rubric (Creativity Exercise)
The creativity exercises were evaluated according to the rubric described in the section 5.4.2.2. in three stages: 1) The number of drawings provided in pre-test exercise is calculated and the rest of the indicators of the rubric are rated by applying the Likert scale from 1 to 5; 2) The number of drawings provided in post-test exercise is calculated and the rest of the indicators of the rubric are rated by applying the Likert scale from 1 to 5; 3) The exercises completed by the same student in pre-test and post-test are compared and the number of the same/very similar drawings in both exercises is determined. All the data obtained were entered in a Microsoft Excel sheets. The means and change percentages were calculated by using a corresponding function within Microsoft Excel.
104 5.6.5 Processing of Data from the Student Diaries
The diaries were submitted by the students in Microsoft Word format as
The diaries were submitted by the students in Microsoft Word format as