Formative evaluation

The goals of the grazier study were to: (i) benchmark grazier understanding and use of climate information; (ii) understand how they make stocking rate and management decisions; (iii) examine the extent to which property records, computers and software are used; (iv) identify tools and information sources used to aid decisions; and (iv) determine preferred methods for accessing information. A formative evaluation approach was adopted and a structured questionnaire used to collect data. Questions were designed with decision-making specifically in mind (Luukkonen-Gronow 1987), to explore evaluation objectives and to contribute to achievement of project goals. Two methods were used to assess grazier understanding of climate phenomena and terms commonly used in climate science: (i) 5 climate test questions (Table 1, Section C; Table 3) and (ii) self-rating their currrent knowledge or skill in relation to 15 climate

This study reports on an 11-month research project, capstoned into a formative evaluation. A formative evaluation is a type of action research, where the process involves an examination of a program, usually in its early stages, to determine what is or is not working, and to serve as a blueprint used to inform strategies that move the program forward (Taras and Davies 2013). This type of evaluation falls in line with the third wave notion that strategy formation should be evidence-based and focused on prioritizing and assessing outcomes accordingly (Feser 2014). As the BN Advantage Economic Development strategic plan is in the formative stages of implementation, a formative evaluation can function as a suitable method to use for this project. The focus of this study is on the vibrancy of the local entrepreneurial ecosystem in the Bloomington- Normal-McLean County, Illinois area. Entrepreneurial literature advances the premise that just inventorying the inputs into a system (universities, education levels, etc.) is not enough to measure the vibrancy of and provide insight into the sustainability of the ecosystem. These are rather determined through analysis of the outcomes produced by the inputs when they work together (Katz and Wagner 2013, Roja et al. 2014, Stangler 2015-Bell-Masterson). In order to capture this vibrancy, an approach that incorporates detailed qualitative and quantitative data about the current ecosystem compliments goals of this formative evaluation.

A two stage formative evaluation of the e-HIT was undertaken. For the first stage, the prototype e-HIT was circulated to a group of e-Health experts. These experts included senior clinicians, managers and academics each of whom had extensive experience of e-health imple- mentations within the NHS (n = 13). These experts were asked to complete the e-HIT for an e-Health initiative they had personal experience of, and on the basis of this experience, to comment critically on the e- HIT. Specifically, respondents were asked whether they thought the e-HIT would be useful to senior managers considering, planning or undertaking an e-Health imple- mentation, what would make the e-HIT more useful, what were the positive features of the e-HIT, and what features needed modifying. Respondents were asked to suggest modifications which would improve the e-HIT, and for other general or specific comments not covered by the questions listed (Additional File 2).

and improve the quality of education. Communication in the evaluation process is very crucial. The formative evaluation does not only judge the student’s performance, but also assess the quality of the learning process. Formative evaluation should be carried out during the learning process, and an instant evaluation should be carried out to help teachers understand the actual teaching quality and deficiencies in the teaching process. This is to allow the teachers to make adjustments in real time, so that students can figure out the methods and objectives of learning in the process. With this, both students and teachers will have a full understanding of their own situation, thereby promoting student learning. Therefore, formative evaluation is a useful evaluation method that aims to improve the teacher’s teaching process and improve students’ learning ability.

The formative evaluation supports the utility of the web-based platform but it is clear that the current ver- sion of the website does not provide considerable value. This is because parents still had to print copies of the Trackers as well as other SWITCH® materials and sub- mit them manually. It was important to test this basic system first to show that it at least produced similar or equivalent outcomes. Nevertheless, web-based programs are supposed to provide a higher level of interactivity than the print materials to actively engage the users [29]. The online version of SWITCH tested in this current study, however, had limited interactivity. Thus, a more comprehensive and interactive web-based platform needs to be established to enable features such as direct logging of Trackers online by parents and automated re- cording and monitoring of completions by the program leaders. The planned enhancements to the content man- agement for SWITCH® will enable school programmers to manage communication and to take advantage of other web attributes (e.g. ability to link to web resources and social media). Parents would also have an easier time interacting with the program and their child since they would be able to directly track progress using email/web tools that are common in contemporary soci- ety. Thus, we expect that interactions and engagement can be dramatically increased.

The need for such an approach became especially ap- parent in the formative evaluation as the interviewees, based on their professions, stated a range of goals they felt most pertinent to future KT Platform programme refine- ment. Consistent with other KT needs assessments [7, 23], the divisions of interest range in (1) scope, from project-specific to policy level; (2) focus, including collab- oration, planning, dissemination, application, training and science; and (3) degree of technicality requiring beginner to advanced KT skills and knowledge. The variety of KT activities and ongoing needs stated across the literature, and in our own results, highlights the need for the KT Platform to continue development and delivery of varied training opportunities as well as consultation services with the ability to embed experts into project teams. Further- more, the establishment and upkeep of KT Alberta, the community of practice, will be vital to connect those with KT interests and needs to KT training opportunities, col- laboration, knowledge and supports.

This paper studies the formative evaluation and applies two algorithms: Naive Bayes and C4.5 decision tree, to mining data in students' online learning, and uses Matlab programming to predict students' final exam of four classifications (great/good/medium/bad). According to the evaluation results, it is found that the performance of the C4.5 decision tree is superior to the Naive Bayes algorithm in this problem, but the result is not very satisfactory. Next, the data label classification is changed from four categories to two categories, in order to predict students' final exam of two classifications (good/bad), and then we introduce Logistic regression. We use the above three algorithms to perform data mining similarly. From the results, we can see that Naive Bayes performs better than other algorithms under this problem, and has achieved a high degree of accuracy. Further, we use the database to obtain two attributes of students' online learning: duration and number of submissions, and add them to the data table. These three algorithms are also used for mining. According to the evaluation results, the prediction performance is not significantly improved than before. Therefore, it is concluded that there is no need to consider the duration of learning and the number of online submissions when studying the data of this group of formative evaluations. Above all, this paper uses data mining technology to form learner’s formative evaluation of online learning, and then, provides learners with timely and effective evaluation feedback, so as to help learners identify problems in the learning process, continuously improve their own learning, and give full play to the advantages of formative evaluation, so that learners can continuously improve.

Abstract. This paper reports upon the design and use of a lightweight evalua- tion method, especially designed to examine complex interactive systems. The approach is illustrated through a case study involving an interactive tool de- signed to help enable users examine large scale data arising from authentication activity in higher education institutes. The evaluation approach illustrated is to enable the lightweight assessment of usability issues within complex interactive systems and identifying opportunities for significant design improvements. Specifically we argue that this method benefits from capturing key generic fac- tors that underpin the effectiveness of tools for working with complex data. The paper concludes by reflecting upon the effectiveness of the lightweight struc- tured assessment approach and how it supports to formative evaluation.

engagement of equalities-related community members in the mechanisms supporting LAA implementation at service level. It is too early to be able to provide a comprehensive insight into the way in which LAAs will affect equalities work – as the National Evaluation of LSPs: Formative Evaluation and Action Research programme 2002-5 Final report 35 indicates, “One effect of LAAs might be to increase the focus on the “most important” issues at the expense of others seen locally as less important, but alternatively the process could provide an opportunity for government to bring such neglected issues up the agenda.”

Understanding a health service from the user’s point of view has important and widely applicable implications for the future implementation of CHW for MCH pro- grammes in Uganda and for the broader public health policy arena. The purpose of our research is to better understand mothers’ perceptions and acceptability to a newly initiated CHW for MCH strategy at the initial stages of programme implementation through a partici- patory formative evaluation. By conducting this type of investigation, it is anticipated that the beneficiaries be- come an active part of the monitoring and evaluation process. Moreover, the implication is that findings can potentially contribute to inform programme iteration for health-care policy and decision makers from an early stage so that appropriate, timely changes can be made. Finally, a greater understanding of the perceptions of target beneficiaries may lead to specific changes that make the programme more acceptable, therefore in- creasing utilisation and ultimately improving MCH outcomes. This research is timely given an increasing demand for a shift in methods and standards of evalu- ation. Programmes are increasingly moving from output evaluations to the evaluation of results and impact, while developing more collaborative approaches to programme evaluations [3,40].

The first phase of evaluation will provide real-time feedback on how the pioneer schools model is working during the curriculum design and development phase. The subsidiarity approach being taken is both innovative and different to recent models to reviewing the Welsh curriculum, so it’s important to understand how the model is working and whether anything needs to change before it is realised fully.

An approach is taken that attempts to avoid any reduction in intrinsic motivation and minimises subject specific content, while increasing the range of video game design principles beyon[r]

Smoking is one of the leading causes of preventable morbidity and death in England, and is an addiction largely taken up in youth. Given the need for early prevention strategies, this formative study evaluated sports coaches’ perceptions of a preliminary SmokeFree Sports (SFS) campaign. The research was conducted in five youth clubs with eight coaches who attended a Level 1 Brief Intervention Training workshop. Semi-structured interviews were conducted, and a questionnaire assessed the impact of the Brief Intervention Training on coaches’ self-efficacy to deliver smoke free messages at pre, post and follow-up. Interviews were transcribed verbatim and pen profiles were constructed to provide an overview of coaches’ perspectives of SFS with regards to: a) Brief Intervention Training, b) SFS implementation and c) suggestions from improvements. One-way repeated measures ANOVA revealed a significant increase in coaches’ self-efficacy towards knowledge and delivery of smoke free messages from pre to post Brief Intervention Training that was maintained at follow-up. This formative work suggests that there is potential in educating coaches and using sports to deliver health education around smoking. Recommendations to improve the Brief Intervention Training included making it more interactive and specific with respect to sport and particular age groups. SFS could also be trialled in structured setting and with other sports. These findings will be used to inform and tailor the design and development of a larger, definitive SFS intervention.

Smoking is one of the leading causes of preventable morbidity and death in England, and is an addiction largely taken up in youth. Given the need for early prevention strategies, this formative study evaluated sports coaches’ perceptions of a preliminary SmokeFree Sports (SFS) campaign. The research was conducted in five youth clubs with eight coaches who attended a Level 1 Brief Intervention Training workshop. Semi-structured interviews were conducted, and a questionnaire assessed the impact of the Brief Intervention Training on coaches’ self-efficacy to deliver smoke free messages at pre, post and follow-up. Interviews were transcribed verbatim and pen profiles were constructed to provide an overview of coaches’ perspectives of SFS with regards to: a) Brief Intervention Training, b) SFS implementation and c) suggestions from improvements. One-way repeated measures ANOVA revealed a significant increase in coaches’ self-efficacy towards knowledge and delivery of smoke free messages from pre to post Brief Intervention Training that was maintained at follow-up. This formative work suggests that there is potential in educating coaches and using sports to deliver health education around smoking. Recommendations to improve the Brief Intervention Training included making it more interactive and specific with respect to sport and particular age groups. SFS could also be trialled in structured setting and with other sports. These findings will be used to inform and tailor the design and development of a larger, definitive SFS intervention.

The impact of the campaign on pupil attainment is reported in the South and West Yorkshire Impact Evaluation report (Sibieta et al., 2018). Most case study interviewees perceived that there had been positive outcomes for pupils. These claims were based on school data, work scrutiny and/or informal observation. As noted elsewhere, attribution of positive outcomes is problematic given other changes taking place in the case study schools, and interviewees acknowledged this. Nonetheless, it is important to note that interviewees were very positive about the outcomes for pupils: ‘It’s made a massive difference to our children, so it was absolutely the right thing to do’ (CSI, head teacher). Examples of improved attainment and/or accelerated pupil progress were reported in most case study schools (CSA, CSB, CSC, CSD, CSE, CSF, CSG, CSI, CSJ, CSK, CSM, CSO). Schools CSA and CSG reported improved attainment in the progress of lower-achieving pupils. In school CSG, this was attributed to teachers working across all pupil groups:

looking and predictive approaches, which she suggests could facilitate further innovations in the field. She describes an approach which is grounded in a detailed ten-year strategic plan which sets out the proposed future direction of the SALC. In order to make the evaluation of multiple aspects of the SALC as efficient as possible, Mynard suggests a carefully scheduled timeline of ongoing research projects, designed to investigate different services at regular intervals over the ten years of the strategic plan. Finally, Mynard suggests that future evaluations could also be predictive as well as retrospective, taking advantage of the

If the literature clearly delineates one or more procedures that would attain acceptable levels of change in the selected mediating variables in the targeted population and channel, and this level meets investigator needs, then the investigator may proceed to Step D. In most cases, however, the investigator will need to design one or more procedures, based on the knowledge of the theory and its mediating variables and of other change procedures (e.g., persuasive messages[74], skill development[75]), and test their effects on the targeted mediating variable(s) in the targeted sample and channel. Elsewhere, this has been called evidentiary research[26]. The investigators should pre-specify a level of change they are willing to accept to progress to the next step for each targeted mediating vari- able, and below which they are not (because it would compromise the efficacy study). If in step C moderating variables are identified, the investigators need to devise and test separate procedures for each of the subgroups. The evaluation should include both quantitative meas- ures of the targeted mediating variables and qualitative

The NCLB Waiver Provision has promoted a flurry of activity in the development of state principal evaluation frameworks. Established based on guidance from the U.S. Department of Education, these frameworks are required to focus on holding principals accountable for school-wide student learning gains and for effective principal practices. For example, the Wisconsin Framework for Educator Effectiveness 6 defines a framework for principal evaluation that includes 50% of the evaluation based on student outcomes and 50% based on professional practices. Professional practices are assessed through a state evaluation system or an equivalent process adopted by the school district. Student outcomes are assessed through a formula developed by the state, to include the state assessment results (15%), district assessment results (15%), classroom-level student learning objectives and school-wide reading for elementary and middle schools (15%), graduation rates for high schools (2.5%), and a locally determined optional criterion (2.5%).

Fonnative research or program monitoring procedures are di­ rected at answering questions about the actual implementa­ tion of operating programs. The primary purpose of carrying out such procedures, according to Epstein and Tripodi (1977* 56), is to enable administrators to make "rational and in­ formed decisions about program operations". This type of research assists administrators' in that it not only veri­ fies the decision-making system's understanding of the pro­ gram, but it is also likely to reveal difficulties that they face in actually conceptualizing these preconditions. Fur­ thermore, formative research can discover latent goals and unintended effects which may not be identified in an evalu­ ability assessment.

Table 3: Sample of Curriculum Pioneers involved in Strand 2 interviews June-July 2017 Consortium Curriculum Pioneers who joined in January 2017 Curriculum Pioneers interviewed in June an[r]