45
PRACTICE OF INSURING AND ASSESSMENT PERFORMANCE
INDICATORS AND QUALITY LEVELS IN HIGHER EDUCATION
Liliana PASCHIA, PhD Candidate „1st of December 1918” University of Alba Iulia
Abstract
If 20-30 years ago the graduate of the educational institution had at his disposal 3-5 years to adapt to the concrete conditions of the enterprise or organization where he was directed to work, today employers insist on a quick insertion of the graduates into the labour market. In this paper we examine practices and benchmarks for performance appraisal of higher education institutions in Romania. Based on the results of the evaluations carried out we conclude that the best practice of those examined is the European orientation on the implementation of the European-agreed recommendations, instruments and indicators. We also find that a sufficiently elevated level of standardization has not been reached until now, so doubts about the conformity of the assessment with reality are not yet fully eliminated. Also, the lack of statistical data and the lack of agreement on a fundamental set of indicators only amplifies the relativity of judgments at this stage.
Keywords: performance, university, quality, organizational management, teaching-learning-research
Introduction
The first document we submit to your attention is "The Recommendation of the Parliament and the Council on the European cooperation in quality evaluation in school education", which draws on the first disintermediate document, which establishes the clear guidelines in this area, concerning everything that is education and professional training. This recommendation follows a report on the quality of education in Europe, achieved by an international work group between 1997 and 199967.
The document, which appeared as a proposal in 200068, recommends that Member States
support the improvement of the quality of school education through specific measures aimed at introducing transparent quality systems, promoting self-evaluation, the essential tool for quality assurance, developing the exchange of good practice and specific tools.
Another important document on quality is the European Report on Quality of Lifelong Learning Indicators, published in 2002. This report proposes a number of 15 quality indicators for lifelong learning. In contrast to the previous one, which proposed only very general guidelines to be applied to quality assurance systems, the Report proposes concrete indicators grouped into several areas (Skills, Competences and attitudes, Access and participation, Lifelong learning resources, Strategies and development), which allow a comparison of the performances of the different Member States or candidate countries. On the other hand, the Report does not propose concrete targets or acceptable minimum levels because of the extreme diversity of situations in the respective school systems. The next step was when it was agreed upon the five "benchmarks" (targets for the development of European
67Evaluating quality in school education. A European pilot project. Final report (1999)
68ProposalforaRecommendationoftheEuropeanParliamentandoftheCouncilonEuropean Cooperation in
46 education systems and training up to 201069 (which we do not present in the present paper).
All the following European initiatives in this field have a consistent quality assurance part. Relevant and important for the implementation of quality schemes was the proposal to establish a European Qualifications Framework (EQF) 70, which insists on the link between the development of the
framework and the national qualifications, on the one hand, and the procedures and systems for the management and the quality of education and training, on the other. The documentary proposes a set of principles of quality assurance in education and training, which can be a binder for the correlation of quality systems at all its levels: PE, ITE, ECS, HE. We will present these principles in view of the previous steps taken by the joint conference of representatives of pre-university and higher education professional education at Graz in 2006 under the auspices of the Austrian Presidency of the European Union71:
• Quality systems are needed to ensure public accountability and improve education and training.
• Policies and quality assurance procedures will cover all levels of education and training systems.
• Quality assurance will become an integral part of the management of education and training institutions.
Quality assurance will include regular evaluation of scheduling institutions through external monitoring / evaluation structures or agencies. External quality monitoring / evaluation quality structures and agencies will, in turn, be monitored and evaluated. Quality assurance will be multidimensional, will refer to the context , "inputs", processes and "outputs", focusing on learning outcomes.
Quality assurance systems will include the following elements: - clear and measurable objectives and standards,
- implementation guidelines, including stakeholder involvement concerned;
- adequate resources;
- consistent assessment methods that combine self-evaluation with external evaluation; - feedback mechanisms and improvement procedures;
- Ensuring the wide accessibility of evaluation results.
Initiatives to ensure the availability of international, national and regional competences are going to be directed in such way to ensure the coherence, synergy and analysis of the system.
Assuring the quality of a collaborative processing the entire system of education and training, involving all relevant stakeholders, both in the Member States and in Europe.
The evolution of the 2000-year-old pressures in 2006 is evident. Also, the "EQQM model" as well as the "plan-do-check-act" PDCA is becoming visible in the process of improving and improving quality. In the field of higher education, on the basis of a recommendation of the European Council72,
the Education Ministers of 29 European countries signed on 19 June 199973, the Bologna Declaration
which triggered the process of creating a European Area of Higher Education (EHEA) make European education "mature and comparable, more competitive and more attractive to its citizens and those of other continents." In the Bologna process, the issue of quality assurance plays an increasingly important role. To this end, the European Association for Quality Assurance in Higher Education (ENQA74) was
created. LaBergen, in 2005, the ministers of education from the Bologna Declaration signatory countries
69EuropeanBenchmarksinEducationandTraining:Follow-uptotheLisbonEuropeanCouncil (2002). 70See the Recommendation of the European Parliament and of the Council of 23 April 2008 on
the establishment of the European Qualifications Framework for lifelong learning (2008/C111/01)(2008)
71Conclusions of the Conference „Quality Assurance in Higher Education andVocational Education
andTraining”, 11/12May, University of Graz(2006).
72COUNCIL RECOMMENDATIONof 24September 1998onEuropean cooperation in quality assurance
in higher education (1998)
47 adopted the standards and guidelines75 for quality assurance in the EHEA and committed themselves to encouraging the development of national agencies that will assess the quality of education provided by higher education institutions. It was decided to set up a European Register of Quality Assurance Agencies and to set up joint systems and procedures for the accreditation of higher education institutions based on the standards and guidelines proposed by ENQA. The Ministers for Education Conference in London, held in May 2007, specifically concerned the situation of the European Higher Education Area (EHEA) in the context of globalization. Obviously, the issue of quality assurance could not fail. Significant progress has been made in implementing the common guidelines (adopted at Bergen) in all EHEA member countries. It was decided to establish the European Register of Quality Assurance Agencies76 and to initiate the registration of external evaluation agencies in this register (based on transparent and independent evaluation procedures), and its mode of operation will be evaluated externally at two years with the involvement of all stakeholders.
Programs, Standards and Procedures
If 20-30 years ago the graduate of the educational institution had at his disposal 3-5 years to adapt to the concrete conditions of the enterprise or organization where he was directed to work, today employers insist on a quick insertion of the graduates into the labour market. In other words, the length of accommodation of graduates becoming employed is considerably reduced by employers. This, in turn, may affect the quality of specialist training.
Currently, in Romanian education more than 80% of the quality problems are system-dependent and, consequently, it is the duty of the managers to solve them. But, as a rule, they are happy to take action after the negative events have taken place, focusing on "extinguishing" action rather than on "fire prevention". This is usually due to the fact that they are not aware of the true price of lack of quality. Once this price is perceived, it becomes clear the effectiveness of prevention and the use of appropriate methods to improve quality, from the start and on a permanent basis.
One of the greatest achievements in this area was the development in 2004 of the Common Quality Assurance Framework (CQAF) as a common reference framework designed to support the development and reform of VET quality at system level and VET providers, while fully respecting the responsibility and autonomy of EU Member States to develop their own quality assurance strategies.
CQAF is the European reference framework for quality assurance and development of education and training, based on the key principles of the most relevant current models. It can be considered as a cross-cutting tool to help practitioners to better understand the functioning of existing quality assurance models, to determine the areas that need to be improved and to make decisions based on quantitative and qualitative common references.
CQAF also provides for the collection and classification of good practices within the EU. The CQAF process was based on contributions from EU Member States as well as two important quality management models: ISO 9001: 2000 and EFQM.
Progresses estimated through the use of CQAF are related to efficiency, transparency and mutual trust in VET systems across the European Union.
In 2005, the European Qualifications Framework was launched on the basis of 4 common principles: quality assurance, validation of non-formal and informal education, guidance and counseling and the promotion of key competences. The general principles promoted by CQAF are:
1. Quality Assurance is required to ensure accountability and improvement of VET. 2. Quality Assurance policies and procedures should cover all levels of VET systems. 3. Quality assurance should be an integral part of the internal management of VET institutions. 4. Quality assurance should include the regular assessment of institutions or programs by
external monitoring bodies or agencies.
5. External quality assurance monitoring bodies should also be subject to periodic analysis and evaluation.
48 6. Quality Assurance should include the context, inputs, process, and output dimensions, with
emphasis on outcomes and learning outcomes. 7. Quality Assurance Systems should include:
- Clear and measurable standards and objectives.
- Implementation guides, including stakeholders involvement; - Necessary resources;
- Valuable assessment methods, associating self-evaluation and external analysis; - Feedback mechanisms and procedures for improvement;
- Evaluation of results, accessible on a large scale. CQAF contains:
- a model that allows for the planning, implementation, evaluation and analysis of quality management systems of comparable levels;
- a methodology for assessing and analyzing systems, paying attention to self-evaluation combined with external monitoring; the methodology can be seen as a complementary horizontal step, which should be considered both in each of the stages of the model and in the quality management system as a whole;
- a monitoring system that is determined at national or regional level;
- a measuring instrument, represented by a set of indicators to allow monitoring and evaluation of the different quality management systems existing at national level in the EU.
This model has a set of characteristics that contribute to the development of the quality of vocational training in a number of different areas, namely:
- gives an overview of the different approaches to existing quality;
- identifies a small number of common criteria in European countries in terms of quality promotion;
- the common European criteria are in line with the main components of other quality management models, in particular the EFQM model and the ISO 9001: 2000 model; - it only mentions the provisions that are considered crucial to quality promotion programs and does not indicate how the system or providers should work;
- seek to cover the essential aspects of all existing practices; - can be used both at national VET systems and in providers;
- can ensure the promotion of quality development in vocational training.
All these aspects, cumulated, can turn the reference model into a valuable tool for promoting and developing the quality of vocational training, both at the level of the EU Member States and the other European countries.
Another obstacle is that, as a general rule, very few EU Member States use the same quality assessment system and the same criteria as regards the national system as well as the quality management systems of training providers.
At the planning stage the organization's policies, procedures, objectives and resources are established as a consequence of the organization's VISION, MISSION, VALUES, OBJECTIVES, POLICIES and STRATEGY.
The implementation stage involves the operationalization of the processes identified, described and related through the process map and process identification sheets, with the main purpose of developing and providing specific VET activities. All these processes must be related to the organization's policies and quality assurance requirements.
It is particularly important that, at this stage, educational and support processes must be carried out in line with Good Practices, identified mainly through Benchmarking techniques.
The evaluation consists of measuring the effectiveness of VET by specific instruments - analysis, audit, self-evaluation, external evaluation, measurement of stakeholders' satisfaction by VET participants (employers, community, supplier staff).
49 It is also extremely important for these results to show sustainability and positive comparability with other similar organizations.
Practices and benchmarks for performance appraisal
In the European education area, the issue of quality standards in education is currently in full dynamic. Education ministers from the EU Member States have entrusted the ENQA (European Network for Quality Assurance) with the task of developing harmonized standards and procedures specific to education so that all educational institutions in this area can adopt a common reference that facilitates both inter-comparability of institutions' performances and ensuring the free movement of people in the community space.
A hierarchy of possible practices and references to assess the performance of educational institutions in the European space, according to the level of these performances ("minimal", "standard" and "excellent") is presented in Figure 1.
Figure 1. Possible practices and benchmarks for performance evaluation (source: Nelu Cârneanu et al, Mangementul calităţii în organizaţia şcolară, Ed. Universităţii
din Piteşti, 2009, p. 41)
Common practices assume a minimum level of performance of educational institutions and consist of multi-criteria evaluation and authorization / accreditation of study programs. The periodic accreditation system determines whether a study program can be initiated under acceptable conditions in a particular educational institution or whether it should be wound up due to deterioration in the conditions for its development.
Good practice implies that, in the continuation of accreditation, to improve the performance of the educational institution, a quality management system should be designed and implemented, preferably in line with the requirements of a quality management standard with a wide international recognition. This QMS, in order to be credible outside the university, must also be certified by a third party accredited certification body.
Best practice implies the selection of those "good practices" which have led to higher performance - those that have the highest efficiency and effectiveness, able to determine the excellence of the educational institution that adopts them.
There are already three important "excellence models" - similar to the standards - that meet the criteria for awarding quality awards in Japan, the US and Europe.
COMMON PRACTICES
GOOD PRACTICES
BEST PRACTICES Performance
Level
EXCELLENT
STANDARD
MINIMAL
Purpose of the evaluation (benchmarks) THE QUALITY
AWARD (excellence models EFQM , PNAC-MB)
QUALITY ASSURANCE
(ISO 9000:2000)
LICENSING (CNEA standards and
50 In EU Member States, so-called "good practice codes" have been developed - in a number of areas of activity, including in education institutions - and numerous benchmarking actions are organized on a regular basis, aiming to know and study "best practices".
The Quality Management System model, according to ISO 9001: 2000, presented in the figure below, is mainly based on customer satisfaction and continuous improvement, using as input elements the customer's requirements and having as output elements the products / services that satisfy the same customers.
Figure 2. ISO 9001: 2000 model (as an example of a "virtuous circle")
(source: Nelu Cârneanu et al, Mangementul calităţii în organizaţia şcolară, Ed. Universităţii din Piteşti, 2009, p. 42)
The elements that add value for customers are:
a. Quality management system
a.1 General requirements: The institution creates, documents, implements, maintains and improves a quality management system (QMS). The processes necessary for the operation of the institution and the QMS are identified, described, implemented and continuously improved, including the appropriate criteria / indicators and methods for measuring / monitoring their performance. The necessary resources are provided.
a.2 Documentation requirements: The QMS documentation includes: the quality policy and objectives of the institution, a quality manual, QMS procedures and the main processes of the institution (education, research-consulting, internal-external communication), documents required by the institution to ensure the effectiveness of the planning, operation and control of its processes, as well as the records required to demonstrate the effective functioning of the institution and the QMS. The scope of the QMS documentation, the form and the support environment are decided by the institution. Procedures are in place to keep documents (drafting, issuing, approving, updating, distributing, accessing) and recordings.
b. The responsibility of the management of the institution
b.1 Responsible Involvement in Quality Promotion: The institution's leadership is responsibly involved in supporting, promoting and continuously improving the QMS through: communicating within the institution the importance of QMS and meeting stakeholder requirements, setting policy and quality objectives , conducting management analyzes and allocating the necessary resources.
b.2 Customer Guidance: The management of the institution ensures that the requirements of the client and stakeholders are determined and met.
b.3 Quality policy: The management of the institution shall establish and update, where necessary, the quality policy. It must be appropriate to the declared mission of the institution, contain a commitment to meet the requirements and continuously improve the effectiveness of the QMS, and provide an appropriate framework for setting the quality objectives. It must be known, understood and applied throughout the institution.
b.4 Planning: The management of the institution ensures that the quality objectives are set for the relevant functions and levels, that there is a plan to implement, maintain and improve the QMS.
b.5 Responsibility, authority and communication: The management of the institution ensures that responsibilities and authority are defined, communicated and function within the institution, that adequate
CUSTOMERS
(AND OTHER STAKEHOLDERS) SATISF
MANAGEMENT RESPONSIBILITY MANAGEMENTULUI
CONTINUOUS IMPROVEMENT OF Q.M.C.
RESOURCE MANAGEMENT
CUSTOMERS (AND OTHER STAKEHOLDERS)
MEASUREMENT, ANALISYS AND IMPROVEMENT
P ACHIEVEMENT
PRODUSULUI
OUTPUTS
RE
51 communication processes exist and function so that all staff are aware of the extent to which the quality objectives have been achieved . The management appoints a QMS representative.
b.6 Management analysis: The management of the facility analyzes, at planned intervals, the QMS operation and the processes considered relevant. Records of analyzes conducted by management should be maintained. The input elements of the analysis are information on: audit results, customer feedback, process performance and product compliance, corrective and preventive action status, tracking actions from previous analyzes, changes that could influence QMS, and recommendations for improvement. Output elements of the analysis include resource requirements, decisions and actions to improve the effectiveness of QMSs, relevant processes and products of the institution.
c. Resource management
c.1 Resource Assurance: The institution determines and provides the necessary resources for the continuous implementation and improvement of the QMS, the relevant processes and the products of the institution.
c.2Human Resources: The personnel involved in the relevant processes of the institution must be competent in terms of studies, training, skills and experience. The institution identifies training needs for staff, plans and performs regular training in appropriate areas. Training records should be maintained.
c.3 Infrastructure: The institution identifies, makes available and maintains the infrastructure (buildings, workspaces and associated utilities, equipment, software and support services) required to produce its products under appropriate conditions.
c.4 Work environment: The institution determines and maintains the work environment required to achieve compliance with requirements.
d. Product making
d.1 Product Product Planning: The institution plans and develops product manufacturing processes in accordance with requirements and consistent with other QMS processes. In the planning of product manufacturing, the institution determines, as appropriate: quality objectives, product requirements, the need to establish new processes and allocate resources.
d.2 Stakeholder Relationship Processes: The institution identifies and updates stakeholder requirements (through appropriate communication channels) regarding products, legal requirements and regulation, and examines whether it has the capacity to meet them. Requirements may refer to: the skills and aptitudes of graduates, the results and the impact of the research and direct assistance provided to organizations in the economic and social environment. Records of analysis results should be maintained.
d.3 Project management: In the case of the educational institution, the design refers to: study programs (the package of competences and abilities assumed, the curriculum, the analytical program specifications and the intermediate stages, the ways of examining and testing them, etc. .), research projects, assistance-consulting projects.
The institution identifies the stages of the design process as well as the modalities of analysis, verification and validation appropriate to each stage. It determines the responsibilities and authority for this process and controls the interfaces between the different groups involved.
Periodic review of project progress is required, verification at relevant moments of the ability of the results to meet the requirements, internal and external validation of the final design result. In the case of study programs, internal validation is done through graduation exams, and external through feedback from employers and graduates. In the case of research, consultancy-assistance projects, internal validation is carried out by testing (where possible) the results and the external ones by the impact generated in the economic and social environment.
d.4 Partnerships and External Entry Control: The institution ensures by appropriate means that admitted pupils / students / students meet the necessary conditions for quality teaching and learning. In this respect, the criteria, methods and procedures for assessing future admissions should be defined.
d.5 Managing product development processes: Achieving products (educational, research and advisory services) must take place under the conditions of maintaining processes, information flows, physical resources used (laboratories, equipments, facilities etc.), methods and tools used to monitor, evaluate and measure products.
52 In the case of measurement-monitoring methods (student examinations, evaluation of the partial or final results of the research, etc.), the issue of their periodic analysis and updating is compared, as well as their comparison with those considered as reference at national or European level.
e. Measurement, analysis and improvement
e.1 General: The institution identifies and plans appropriate methods, implements the necessary monitoring, measurement, analysis and improvement processes to demonstrate the compliance of its products, the effectiveness of the relevant processes and QMSs.
e.2 Monitoring and measurement: The institution identifies and applies appropriate methods to monitor stakeholder perception of meeting their requirements. Direct questioning of research, assistance, consultants, employers, graduates and learners about the competence acquired through education and their own employees on the work environment can be ways of determining the degree of satisfaction of stakeholders.
The educational institution shall ensure that an annual audit or internal evaluation program is developed and implemented in accordance with an appropriate procedure so that the QMS status can be identified, detected nonconformities and identified opportunities for improvement. Reporting of internal audits and their outcomes, as well as monitoring of actions to eliminate detected nonconformities and their causes, is required.
e.3 Control of Non-Compliant Products and Processes: Non-conformities found following process and product monitoring will be dealt with by appropriate procedures, including: ways to avoid continuing the process in the same form (changing the curriculum, the curriculum, analytical course, course owner) or use of the non-compliant product (trainees without proper training, inadequate research contracts); recording nonconformities and keeping a record of them; initiating corrective actions.
e.4 Data Analysis: The institution identifies, collects and analyzes data (resulting from measurement and monitoring activities or other relevant sources) needed to demonstrate the suitability and effectiveness of the QMS and to assess the opportunities to continually improve the effectiveness of the QMS.
e.5 Improvement: The institution shall ensure that a framework and mechanisms are in place to continuously improve the effectiveness of QMS and relevant processes through the use of quality policy, quality objectives, audit results, data analysis, corrective and preventive actions and analysis by management
Conclusion
The concrete results of the implementation of legislation and the methodology / procedures for quality assurance and assessment, as well as the functioning of the institutions that guide the quality management system at national level lead us to a series of conclusions. A first conclusion concerns the pursuit and application of some unanimously recognized principles of quality of education.
The Romanian Agency for Quality Assurance in Higher Education (ARACIS) aims to apply the recommendations and tools elaborated at the EHEA and ENQA level, without which ARACIS activity, on the one hand, and the evaluation of the Romanian higher education institutions, on the other hand, can not be integrated into the European educational space.
Even though self-evaluation is fundamental and many of the principles are common, the ARACIS Guidelines do not provide any explicit reference to quality models applied in other areas (including those promoted by ISO and EFQM). As a result, focus on the client is less obvious - for example, "peer review" has a much larger share than the student's views on teacher evaluation. Similarly, there is no reference to the "quality circle" or to another methodologically recognized model (obviously outside the one promoted by ENQA)
At the procedural level, it can be seen that the presented systems are presented, allotted to the decision-making de-valuation based on evidences. All judgments must be made (at least at the level of intent and regulations) based on evidence. On the other hand, however, a sufficiently high level of standardization has not been reached, so that any doubt about the conformity of the assessment judgment with reality disappears. In addition, the lack of statistical data and the lack of agreement on a fundamental set of indicators only amplify the relativity of value judgments at this stage.
53 received from these beneficiaries remains to be used to review the principles, procedures and tools for quality assurance and assessment and, implicitly, for continuous improvement.
With regard to external evaluation, important steps have been taken to ensure the independence and transparency of procedures and, in particular, the results of the evaluation. All external evaluation reports for ARACIS are developed by independent evaluators (acting according to their own ethical code) and are public, including access to appeals procedures. This is less obvious in the monitoring / validation system of self-assessment reports, since, on the one hand, the monitoring reports are not public and, on the other hand, the staff involved in the monitoring / validation visit of the self-evaluation results is recruited locally.
The importance attached to self-evaluation is another point shared by the three systems analyzed. Internal evaluation / self-evaluation reports are the lasses of any external evaluation or demonstrating judgment. They also support ongoing improvement processes.
Lastly, a final conclusion on the results of the evaluations carried out reveals that the best practice of those examined is the European orientation on the implementation of the European-agreed recommendations, instruments and indicators.
References
1. Cârneanu Nelu et al, Mangementul calităţii în organizaţia şcolară, Ed. Universităţii din Piteşti, 2009, p. 42
2. *** Evaluating quality in school education. A European pilot project. Final report (1999) 3. *** Proposal for a Recommendation of the European Parliament and of the Council on
European Cooperation in Quality Evaluation in School Education (2000)
4. *** European Benchmarks in Education and Training: Follow-up to the Lisbon European Council (2002).
5. *** Recomandării Parlamentului European şi a Consiliului din 23 aprilie 2008 privind stabilirea Cadrului european al calificărilor pentru învă area de-a lungul vie ii (2008/C111/01)(2008)
6. *** Conclusions of the Conference „Quality Assurance in Higher Education and Vocational Education and Training”, 11/12 May, University of Graz (2006).
7. *** COUNCIL RECOMMENDATION of 24 September 1998 on European cooperation in quality assurance in higher education (1998)
8. *** The Bologna Declaration of 19 June 1999 (1999)
9. *** www.enqa.eu
10. *** Standards and Guidelines for Quality Assurance in the European Higher Education Area
11. *** http://www.eqar.eu/
