Training Experts for Quality Assurance: Lessons Learned

!

Training Experts for

Quality Assurance:

Lessons Learned

!

Training Experts for

Quality Assurance:

Lessons Learned

The results of

the E-TRAIN Pilot Trainings

Copyright*©*2012*by*the*European)Consortium)for)Accreditation)in)higher)education* Author(s):*Mark*Frederiks,*Stephanie*Zwiessler,*Maria*E.*Weber* ISBN/EAN:)978H94H90815H07H3) All*rights*reserved.*This*information*may*be*used*freely*and*copied*for*nonHcommercial* purposes,*provided*that*the*source*is*duly*acknowledged** Additional*copies*of*this*publication*are*available*via*www.ecahe.eu.*

! !

Table of content

! 1.

!

Introduction,...,9

!

2.

!

Training,Practices,of,ECA,agencies,...,12

!

2.1.

!

Starting!point:!when!is!someone!an!expert!and!a!good!panel!member!...!13

!

2.2.

!

What!can!and!should!be!accomplished!instead?!...!16

!

3.

!

The,Pilot,Trainings,...,18

!

3.1.

!

The!pilot!trainings!for!European!experts!...!18

!

3.1.1.

!

The!first!pilot!training!of!European!experts!...!20

!

3.1.2.

!

The!second!pilot!training!of!European!experts!...!24

!

3.2.

!

The!pilot!trainings!for!QA!staff!...!28

!

3.2.1.

!

The!first!pilot!training!of!QA!staff!...!28

!

3.2.2.

!

The!second!pilot!training!of!QA!staff!...!30

!

4.

!

Conclusions,...,33

!

4.1.

!

The!training!of!European!experts!...!33

!

4.2.

!

The!training!of!QA!staff!(train!the!trainers)!...!35

!

5.

!

Annexes,...,37

!

Annex!1:!Detailed!overview!of!training!characteristics!of!QA!agencies! Annex!2:!Preparatory!material! Annex!3:!Programme!of!the!training! Annex!4:!Programme!of!the!first!pilot!training!of!QA!staff! Annex!5:!Recommendations!(final!draft)! Annex!6:!Programme!for!the!training!days! !

!

!

”

!

!

“

“European cooperation in higher

!

education calls for internationally

experienced and trained experts.”

*

!

!

1.

Introduction

!fundamental! part! of! external! quality! assurance! procedures! is! peer! review.! The! connection! between! the! quality! of! the! procedure! and! the! ‘quality’! of! the! team! members! is! apparent:! no! matter! how! competently! the! quality! standards,! frameworks! and! procedures! are! composed! –! if! the! experts! are! not! conversant! with! the! application!of!these!standards,!the!quality!of!the!procedure!might!suffer.!This!is!one!reason! why!the!European,Standards,and,Guidelines,for,Quality,Assurance,(ESG)!insist!that!“(…)!the! experts! undertaking! the! external! quality! assurance! activity! have! appropriate! skills.”! Additionally! the! ESG! point! out! that! Quality! Assurance! Agencies! should! provide! ‘appropriate! briefing!and!training!for!experts”.!!!

Within! the! higher! education! sector,! collaboration! across! regional! and! national! borders! is! more! and! more! the! rule! instead! of! the! exception! [! especially! within!the!European!Higher!Education!Area!–!(EHEA),! and! correspondingly! the! field! of! work! for! experts! is! no!longer!restricted!to!their!own!national!system(s).! This! fact! is! also! stressed! by! the! ESG’s! emphasis! on! “the!use!of!international!experts”!and!would!lead!to! the!conclusion!that!the!training!of!experts!should!not!

be! restricted! to! the! national! level,! either.! Drawing! on! international! expertise! can! help! to! ensure! the! independence! of! the! panel! activity! from! the! influences! of! national! conflicts! of! interests! and! foster! the! orientation! towards! European! and! international! developments! for! standards!and!procedures.!!

A

No matter how competently the quality standards, frameworks and procedures are composed – if the experts are not conversant with the application of these standards, the quality of the procedure might suffer.

!

10

!

Supported!by!EU,funds!the!European!Consortium!for!Accreditation!in!higher!education!(ECA)! launched!in!October!2010!the!“European!Training!of!QA!Experts!(E[TRAIN)”!project.!ECA’s!E[ TRAIN!project!intends!to!strengthen!the!European!dimension!in!quality!assurance!by!making!it! possible! for! Quality! Assurance! Agencies! to! identify! trained! and! experienced! experts! from! countries!within!the!EHEA.!

The!overall, aim! of! E[TRAIN! was! to! facilitate! the!sharing, of, trained,, knowledgeable,, and, internationally, experienced, experts! who! will! be! better! equipped! to! participate! in! quality! assurance! (QA)! procedures,! to! the! benefit! of! higher! education! institution! institutions! under! review.!Two!objectives!were!specified!in!order!to!ensure!that!the!overall!aim!of!E[Train!would! be!met:!! ! 1. to!develop!a!European!training!programme!for!experts!in!QA!procedures!and! 2. to!develop!a!database!to!share!experts!among!Quality!Assurance!Agencies!in!Europe.!! ! The!first!objective!is!the!most!important!one!for!this!handbook:!the!provision!of!training!for! staff!member!of!quality!assurance!agencies!and!for!training!of!experts!(peers/auditors)!who! will!be!part!of!panels!in!QA!procedures.!According!to!the!first!objective!two!sorts!of!trainings! have!been!conducted!since!E[TRAIN!was!launched:! ! 1. training!for!quality!assurance!agency!staff!–!the!train[the!trainer[session;!and!! 2. pilot!training!sessions!for!experts! ! The!focus!within!the!pilot!training!sessions!was!not!upon!specific!approaches!for!the!training! of! panel! chairs! and! panel! secretaries.! Both! types! of! trainings! delivered! were! open! for! participation! for! staff! of! ECA! member! agencies! but! as! well! for! experts! nominated! by! the! agencies.!Participating!agencies!and!experts!involved!in!the!pilots!received!practical!hands[!on! approaches!for!the!training!of!experts.!!

!

!

11

The! first! pilot! session! for! staff! members! of! quality! assurance! agencies! was! held! on! 3rd! November!2011!in!Vienna!and!the!second!was!held!on!10th!May!2012!in!Madrid.!The!first!pilot! session!for!experts!was!held!on!6th!October!2011!in!The!Hague!and!the!second!one!on!23!and! 24! April! 2012! in! Vienna.! In! advance! to! these! sessions! the! agencies! were! asked! to! nominate! experts!to!participate!in!these!pilot!sessions.!!

This! publication! includes! a! summary! of! the! experiences! and! achievements! acquired! through! the!various!pilot!sessions!during!the!E[TRAIN!project.!!

Chapter! 2! provides! a! general! overview! on! the! actual! status! quo! of! training! practices! within! agencies.!Chapter!3!summarizes!the!trainings!actually!delivered!within!the!E[TRAIN!projects.!! Since! the! handbook! first! and! foremost! deals! with! the! aspect! of! how! to! train! experts! undertaking!external!quality!procedures,!a!second!publication![!The!Handbook!for!Training!of! Experts! [! provides! guidance! for! preparing! trainings! for! expert! panel! members.! With! this! Handbook! the! author! Gillian! King,! who! was! also! a! trainer! in! the! pilot! sessions! for! quality! assurance! agency! staff! members,! provides! a! valuable! guidance! for! preparing! trainings! for! expert!panel!members.!!

!

12

2.

Training Practices of ECA agencies

he! following! information! is! derived! from! a! survey! amongst! QA! agencies! which! was! conducted! in! order! to! explore! which! training! practices! exist! among! the! quality! assurance! agencies! and! to! identify! what! the! agencies! consider! to! be! good! training! practices.! It! served! as! an! important! input! and! inspiration! for! the! design! of! the! European! training!course.!The!data!were!complemented!during!the!first!pilot!staff!training!in!November! 2011!where!staff!from!ten!different!European!agencies!elaborated!on!their!current!practices.!! The!survey!has!revealed!common!elements!of!the!agencies’!training!practices.!! Almost!all!the!agencies!carry!out!some!kind!of!training!experts.!The!focus!areas!of!the!trainings! are!comparable!and!involve!the!quality!assurance!process,!national!higher!education!system! and!practical!skills.!The!format!varies,!though.

Some! agencies! use! online! trainings,! some! arrange! seminars,! some! have! trainings! which! lead! to! certification,! and! some! combine! the! training! with! a! meeting!right!before!the!site!visit.!!

When!looking!at!what!is!considered!to!be!good!practice,! quite!a!few!agencies!find!it!appropriate!to!specialise!the! training! according! to! the! background! or! role! of! the! experts,! e.g.! distinguishing! between! newcomers! and! experienced! experts,! or! training! secretaries! and! chairs!

separately, or! having! separate! student! trainings.! Furthermore,! the! practical! elements! (interactive!elements!such!as!case!studies,!role[plays!etc.)!are!often!highlighted!and!this!is!very!

T

The national context is relevant for the needs and means of quality assurance and influence the outlook of trainings delivered; nevertheless the main focus of the trainings is laid on QA standards & criteria, the higher education system, practical skills.

!

!

13

much!appreciated!by!the!participating!experts.!This!is!one!reason!why!the!European!experts’! training!attaches!some!importance!to!integrating!practical!elements!into!the!training.!!

Considering! these! findings! of! the! survey! amongst! agencies! and! taking! into! account! the! European!dimension,!the!European!training!programme!of!experts!has!been!developed.!! The!table!in!ANNEX!1!gives!a!more!detailed!overview!of!training!characteristics!of!QA!agencies.!!

2.1.

Starting point: when is someone an expert and a good

panel member

Before!starting!to!create!a!training!programme!for!international!experts!two!very!basic! questions!had!to!be!answered:!! 1. What!constitutes!a!‘good’!expert?!And!can!the!skills,!qualities!and!competences!which! might!be!needed!by!such!an!expert!be!acquired!through!training!at!all?! 2. What!is!a!good!panel!member?!Of!course!there!is!no!clear!or!easy!answer!to!this! question:!the!‘quality’!of!an!expert!depends!on!a!variety!of!aspects.!! ! In!the!following!it!is!attempted!to!depict!a!short!set!of!characteristics!of!external!experts.!Due! to!the!experience!in!the!pilot!sessions!within!the!E[TRAIN!project,!the!following!outlines!are! solely!focussing!on!so!called!‘academic!experts’.!Nevertheless!various!agencies’!procedures!do! not!only!include!‘academic!experts’!but!also!experts!from!other!relevant!stakeholder!groups! (e.g.! students,! experts! with! relevant! professional! background).! Several! procedures! have! to! take!into!account!the!different!target!groups!and!based!on!that!the!needs!of!information!and! skills!of!panel!members.!!!

Hence,! if! the! expert! is! an! academic,! a! high! level! of! expert! knowledge! in! their! academic! discipline!or!field!should!be!a!given!–!she/he!should!be!up[to[date!on!developments!within!her! or!his!subject!area,!she/he!should!have!a!broad!scope!of!experience!in!research!and!teaching,! she/he! should! have! experience! with! the! development! and! implementation! of! study! programmes.!Where!the!expert!is!a!senior!HE!administrator!it!is!expected!that!he!or!she!has! been! responsible! at! a! senior! level! in! the! institution! for! educational! policy! and! processes.!!

!

14

She/he!should!have!been!involved!at!a!senior!level!in!their!institution!with!quality!assurance! processes!and!assessment!processes.!

Student! experts! will! be! expected! to! be! familiar! with! educational! and/or! quality! assurance! processes,! and! probably! to! have! (or! have! had)! some! representative! role! in! their! institution.!! Quality!assurance!agencies!can!check!these!basic!requirements!for!working!in!an!expert!panel! rather! easily! by! scrutinising! documents! such! as! the! curriculum! vitae,! list! of! publications,! personal!statements,!of!the!expert!candidate!in!question.!!

There! are! ‘practical’! skills! which! competent! experts! possess.! These! comprise! for! example! communication! techniques,! interview! techniques,! knowledge! about! how! to! write! reports! evidence! based,! knowledge! about! relevant! national! frameworks,! awareness! for! the! specific! role!in!the!group,!awareness!about!cultural!differences,!the!ability!to!be!good!team!workers!or! the! ability! to! accurately! analyse! strengths! and! weaknesses! of! the! study! programme/! institution!under!review.! ! ! ! ! What!experts!should!possess…! • Awareness!of!cultural!differences! • Ability!to!work!in!teams!and!groups! • Subject!specific!knowledge! • Analytical!skills!and!competences! • Communicational!skills!&!interview!techniques! • Knowledge!of!QA!standards!and!procedures!&!national!frameworks! !

! !

15

That!is!why!training!sessions!normally!include!a!practical!component:!! 1. Testing!and!enhancing!these!skills!e.g.!by!using!(parts!of)!the!higher!education! institutions’!self[evaluation!reports!to!be!analysed!in!case!studies.!! 2. Role[plays!in!which!interview[situations!with!representatives!of!the!institution!are! simulated!and!thus!possible!(problematic)!situations!can!be!anticipated.!Role[plays!are! also!a!good!training!method!to!focus!the!team!aspects!(various!roles,!diversity! aspects).!! ! Obviously!one!of!the!main!aims!of!(national)!training!courses!for!experts!is!to!guarantee!that! the!experts!are!well!acquainted!with!the!standards!and!guidelines!to!be!applied!in!the!review! process.!They!should!know!the!legal!framework!of!the!procedure,!their!role!in!the!review!and! the! limits! of! their! decision[making! powers.! This! is! why! a! training! programme! should! ensure! that!experts!are!familiar!with!the!framework!conditions!of!the!review!process.!

Developing!a!pan[European!training!course,!a!first!important!realisation!regarding!the!target! group! and! the! aims! of! the! pilot! training! session! was! that! it! could! not! possibly! cover! all! European! external! quality! assurance! systems! with! their! varying! standards,! criteria! and! procedures.!! 1. Firstly,!this!would!be!unfeasible!regarding!the!sheer!extent,!! 2. Secondly!not!every!potential!expert!is!interested!in!each!national!system,!! 3. Thirdly!it!is!the!core!task!of!each!regional/national!QA!agency!to!make!their!experts! acquainted!with!the!respective!standards.! ! A!European!training!programme!cannot!and!should!not!try!to!accomplish!this.! ! !

!

16

2.2.

What can and should be accomplished instead?

As!noted!above,!academic!subject!knowledge!is!an!important!prerequisite!for!experts!but!is! not!teachable!in!the!context!of!a!training!programme.!Always!bearing!in!mind!that!the!target! group!are!experts!who!already!have!some!experience!in!external!quality!assurance!procedures! at! the! national! and! international! level,! this! would! leave! practical! and! communication! skills.! Since!the!training!course!is embedded!into!a!European!dimension,!the!communicational!skills! may!have!to!be!complemented!by!input!on!intercultural!awareness!and!communication.!The! practical!skills!may!include!reflection!on!the!role!of!the!expert!within!the!procedure,!and!work! on!a!case!study.!

The!knowledge!dimension,!as!previously!discussed,!cannot!comprise!extensive!information!on! all! European! systems! but! could! focus! instead! on! a! brief! but! representative! overview! of! external! QA! approaches,! of! different! ‘philosophies’! across! Europe! (for! more! details! see! chapter!3).! What!a!European!Training!Programme!should!comprise!…! • Practical!/!Communicational!Skills!! • Analytical!skills!and!competences! • Knowledge!about!external!QA!approaches!in!Europe! • Intercultural!Awareness! !

3.

The Pilot Trainings

he! focus! of! this! chapter! is! to! summarise! the! experiences! from! the! trainings! delivered!for!European!experts!and!for!staff!of!QA!agencies.!First,!the!experiences! with!the!two!pilot!trainings!for!European!experts!are!described,!followed!by!the! two! pilot! trainings! for! QA! staff.! For! each! of! the! pilot! trainings! a! brief! description! of! the! target!group!(participants),!objectives,!learning!outcomes,!content!and!didactics!is!given.! This!is!followed!by!the!main!results!of!the!evaluation!of!each!of!the!pilot!trainings.!!

3.1.

The pilot trainings for European experts

Who!are!the!European!experts!that!should!be!trained!by!E[TRAIN?!The!E[TRAIN!proposal! indicates! that! experts! who! are! part! of! assessment! panels! in! national! external! QA! procedures!could!take!part!in!the!pilot!trainings.!!

In! its! meeting! on! 5th! May! 2011! the! E[TRAIN! Stakeholder! Group! (representing! the! QA! agencies)! specified! that! as! institutional! reviews! were! emerging! in! several! countries! it! would! be! most! useful! if! the! trainings! focused! on! experts! who! could! participate! in! institutional!reviews.!!

!

The! experts! trained! by! E[TRAIN! should! bring! international! experience! to! national! panels! and!also!help!to!legitimize!decisions!made!when!institutional!reviews!are!carried!out.!!

T

The! experts! trained! by! E[TRAIN! should! bring! international!experience!to!national!panels!and!also!help! to! legitimize! decisions! made! when! institutional! reviews! are! carried! out.! ! They! should! also! be! able! to! indicate! ways! in! which! the! institutions! can! improve! their! practices.!

!

!

19

They! should! also! be! able! to! indicate! ways! in! which! the! institutions! can! improve! their! practices.!It!was!argued!that!in!order!for!the!experts!to!reach!this!high!level,!they!need!to! have!solid!experience!from!QA!already!before!entering!the!E[TRAIN!pilot!training.!Hence,! the!focus!of!the!training!should!be!on!a!higher!level!than!basic!elements!of!QA.!However,! the!E[TRAIN!Focus!Group!(representing!experienced!experts)!had!warned!against!a!“new! class! of! elitist! QA! professionals”.! It! would! threaten! the! nature! of! peer! review! and! ultimately!the!legitimacy!of!the!review!process!itself!if!a!small!number!of!trained!experts! travelled!around!Europe!to!take!their!seats!in!review!panels!as!if!it!were!a!profession.!To! avoid!this!threat,!it!was!suggested!that!the!trainings!should!also!be!open!to!experts!with! little! or! no! experience! in! QA.! A! mediating! solution! was! found! in! the! agreement! by! the! Stakeholder!Group!that!less!experienced!experts!might!join!the!trainings!after!undergoing! on!line!E[TRAIN!training!modules!leading!to!the!entry!level.!Moreover,!it!was!decided!that! the!experts!for!the!pilot!trainings!should!be!proposed!by!the!national!agencies!(i.e.!the!ECA! members).!Nominated!experts!should!have!some!experience!with!the!national!procedures! in! the! country! concerned.! It! was! agreed! that! it! would! be! possible! to! nominate! student! experts.!The!trainer!should!receive!information!from!the!participants!on!their!background! and!expectations!in!order!to!customize!the!training!according!to!needs.!It!was!agreed!that! Professor! Nick! Harris! (independent! consultant! and! former! QAA! Director! of! Development! and!Enhancement!Group)!would!act!as!trainer!for!the!first!pilot.!With!regard!to!the!content! of!the!training!the!Stakeholder!Group!endorsed!the!distinction!made!by!the!Focus!Group! (and! included! in! the! “Guidelines! for! Training! of! Experts”)! between! the! awareness! dimension,! the! cognitive! dimension! and! the! interactive! dimension.! With! a! view! to! the! training! of! experts! for! institutional! reviews! the! Stakeholder! Group! emphasized! the! following:! ! 1. The!awareness!dimension;!there!should!be!additional!focus!on!intercultural! attitudes!(this!can!include!research!papers!on!international!communication).!! 2. The!cognitive!dimension;!here!additional!focus!should!be!on!the!European! dimension!and!on!variations!in!institutional!quality!management/!autonomy!in! higher!education!institutions!in!different!countries.!After!completing!the!training! the!experts!should!be!up!to!date!on!international!standards.!!!!

20

3. The!interactive!dimension;!the!experts!should!e.g.!learn!about!intercultural! communication,!effective!interviewing!skills,!and!other!“soft!skills”.! ! It!was!against!this!background!that!the!preparations!for!the!first!pilot!training!of!experts! started.!

3.1.1.

The first pilot training of European experts

For!both!budgetary!reasons!and!busy!agendas!of!experts!it!was!decided!to!limit!the!first! pilot!training!to!1!day!with!a!dinner!on!the!previous!evening.!The!trainer!and!the!project! partners! responsible! for! the! training! (ÖAR1_{! and! NVAO)! met! once! to! prepare! the! pilot!} training.! The! envisaged! content! for! the! training! was! derived! from! the! Guidelines!for! the! Training!of!Experts:! 1. What!should!experts!be!aware!of?! - Different!organisational!cultures,!different!quality!cultures!! - Being!biased!(prior!perceptions/prejudices)! - Intercultural!awareness/Intercultural!communication!! 2. What!should!experts!know?!! - Knowledge!of!the!European!dimension!of!QA!! - Knowledge!of!the!national!dimension!of!QA!! - Knowledge!of!cultural!differences!! 3. What!should!experts!be!able!to!do?!! - Audit!techniques/”tools!&!tricks”,!e.g.!how!to!ask!good!questions! - Team!work:!how!to!effectively!work!together!as!a!review!team! - How!to!write!a!(joint)!report!that!includes!the!opinions!of!all!! - “Do’s!and!Dont’s”!at!the!site!visit! - How!to!act!in!intercultural!situations! ! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! 1_{!With!the!legal!validity!of!the!HS[QSG!a!new!and!single!agency!(Agency!for!Quality!Assurance!and!Accreditation!Austria)!was!} established!which!is!responsible!for!external!quality!assurance!in!the!HE!sector!(Public!Universities,!Universities!of! Applied!Sciences,!Private!Universities).!The!‘Qualitätssicherungsrahmengesetz’!(QSRG)!entered!into!force!on!1st!March! 2012.!The!ÖAR!!persisted!until!31st!August!2012.!

!

!

21

It!was!quickly!agreed!that!this!agenda!was!too!much!for!a!one!day!training.!Instead,!the! trainer! had! made! a! proposal! which! included! several! dimensions! of! QA! training.! A! distinction!was!made!between!knowledge!and!understanding!which!could!be!provided!by! on!line!training,!and!skills!(including!roles!and!practice)!which!would!be!the!main!focus!of! the!training!day.!Each!of!these!dimensions!consisted!of!a!break!down!into!4!categories!on! an! introductory! level! and! 4! subsequent! levels! of! achievement.! This! resulted! in! a! matrix! which!would!have!to!be!further!developed.!Learning!outcomes!should!be!included!for!each! of!the!dimensions,!categories!and!levels.!It!turned!out!that!this!was!a!too!complicated!task! to! achieve! in! time! for! the! start! of! the! pilot! training.! Instead! the! trainer! would! ask! participants! at! the! start! of! the! training! what! they! expected! as! learning! outcomes! of! the! training.! The! first! question! in! the! evaluation! form! was! then! whether! these! learning! outcomes! had! been! achieved! in! the! view! of! the! participants.! The! other! questions! in! the! evaluation! form! dealt! with! opinions! on! the! strong! and! weak! points! of! the! training.! In! addition,! the! participants! were! specifically! asked! to! rate! the! following! aspects! from! 1! (excellent)!to!5!(poor):! ! - Clarity!of!session!goals! - Trainer’s!presentation!skills! - Relevance!of!information! - Clarity!of!information! - Scope!(coverage)!of!training! - Relevance!of!materials!presented!during!training! - Organisation!of!training! - Overall!rating!of!training.! ! The!knowledge!base!with!on!line!materials!for!experts!was!not!available!by!the!time!of!the! first! training! (this! was! meant! to! be! an! outcome! by! the! end! of! the! E[TRAIN! project).! Therefore,!it!was!decided!to!start!the!training!day!with!a!discussion!on!two!assignments! which!the!experts!were!asked!to!complete.!One!assignment!on!stakeholders!perspectives! was!to!be!accomplished!by!the!experts!prior!to!the!meeting,!and!another!assignment!on! placing!the!expert’s!own!national!QA!system!in!a!matrix!of!systems!would!be!included!as!a!

22

plenary!exercise!during!the!training!day.!The!attention!in!the!project!partners’!preparatory! meeting!then!turned!to!the!skills!dimension!and!the!role!playing!exercises!which!would!be! the!main!part!of!the!training.!First!the!training!programme!with!the!different!sessions!was! designed!and!then,!after!the!preparatory!meeting,!the!training!materials!were!finalised!by! the!trainer!and!sent!to!the!participants.!The!role!play!would!be!introduced!to!participants! during! the! dinner! on! the! night! before! the! training.! This! would! give! some! time! for! participants!to!prepare!the!roles!which!were!written!down!for!them.!Unfortunately!not!all! participants!were!present!at!the!dinner!and!so!the!preparation!did!not!take!place!then,!but! had!to!be!delayed!until!the!next!day!of!the!training.!!

Eight! experts,! including! one! student! expert,! were! proposed! by! the! agencies! and! participated! in! the! training.! All! experts! had! experience! with! national! QA! procedures! but! not! all! of! them! with! institutional! reviews.! It! turned! out! that! among! ECA! members! there! was! less! experience! with! institutional! reviews! in! practice! than! envisaged! by! the! Stakeholder! Group.! In! addition! to! these! eight! experts! to! be! trained,! four! experienced! experts!from!the!Focus!Group!(including!one!student!member)!participated!in!the!training.! These!experts!participated!fully!in!the!training!but!also!had!a!role!as!observers.!The!Focus! Group! met! immediately! after! the! training! to! discuss! their! observations.! The! Stakeholder! Group!had!decided!that!each!agency!could!send!an!observer!to!the!pilot!training.!This!led! to!an!additional!group!of!7!observers!from!agencies.!The!group!of!organisers!consisted!of! the!trainer!and!4!members!of!staff!of!ÖAR!and!NVAO!who!also!had!a!role!in!the!training! (particularly!in!the!role!play).!

The! training! was! held! on! 6th! October! 2011! at! the! NVAO! offices! in! The! Hague! (with! a! preceding!dinner!on!October!5th).!A!background!paper!introducing!the!programme!for!the! training!day!and!the!preparatory!material!was!sent!to!participants!in!advance:!ANNEX!2.!! The! pilot! training! was! evaluated! by! the! Focus! Group,! the! Stakeholder! Group! and! by! an! observation!report!of!the!responsible!project!partner!ÖAR.!The!main!conclusions!from!this! evaluation!were:! 1. The!overall!rating!of!the!first!pilot!training!was!3!on!a!scale!from!1!(excellent)!to!5! (poor).!The!ratings!of!the!different!aspects!and!comments!made!by!participants! made!it!clear!that!significant!improvements!were!needed!for!the!second!pilot! training.!

! !

23

2. The!training!focused!mainly!on!the!interactive!dimension!(role!play),!less!on! knowledge,!and!very!!little!on!awareness.!It!was!concluded!that!the!second!training! should!have!a!stronger!focus!on!the!latter!dimensions,!particularly!on!intercultural! awareness.! 3. More!structure!for!the!training!was!needed:!explicit!learning!outcomes!for!each!of! the!3!dimensions!should!be!developed.! 4. There!should!be!more!clarity!and!coherence!with!regard!to!didactics!and!materials! used.!! 5. The!role!play!needed!a!real!case[study.!! 6. A!clearer!profile!of!the!trained!experts!is!needed!(e.g.!programme!or!institutional! reviews).!! 7. There!were!too!many!observers;!there!were!as!many!agency!staff!present!as!there! were!experts,!and!within!the!whole!group!the!8!experts!to!be!trained!formed!only! a!third!of!the!group.! 8. One!training!day!is!too!short.! ! * Keeping!in!mind!the!critical!results!of!this!evaluation!the!E[TRAIN!Steering!Group,!ÖAR!and! NVAO!in!particular,!took!responsibility!for!designing!a!new!structure!for!the!second!pilot! training.! ! The!first!pilot!training!of!experts!focused!mainly!on!the! interactive!dimension!(role!play),!less!on!knowledge,!and! very!!little!on!awareness.!It!was!concluded!that!the! second!training!should!have!a!stronger!focus!on!the! latter!dimensions,!particularly!on!intercultural! awareness.!

24

3.1.2.

The second pilot training of European experts

It! was! decided! that! the! training! should! be! prolonged! to! 1! ½! days! to! enable! a! better! coverage!of!the!3!dimensions.!The!target!group!of!experts!to!be!trained!in!the!second!pilot! training! were! described! as! “experts! who! can! participate! in! institutional! external! quality! assurance! procedures! (audits/accreditations/evaluations)! in! different! national! systems”.! The! training! was! structured! along! the! lines! of! the! three! previously! agreed! dimensions! which!were!specified!as!the!following!aims!for!the!training:!

•! To! increase! experts’! knowledge! of! the! European! dimension! of! external! quality! assurance! and! of! the! characteristics! of! different! European! external! quality! assurance!systems;!

•! To! raise! intercultural! awareness! when! working! as! an! expert! in! distinct! (sub)national!external!quality!assurance!systems;!

•! To!enhance!the!audit!skills!needed!to!function!effectively!as!an!expert!in!various! (sub)national!external!quality!assurance!systems.!

The! training! was! divided! into! three! modules:! European! external! QA! approaches;! Intercultural! awareness;! and,! Audit! skills! for! diverse! national! settings.! Each! of! these! modules!corresponded!with!one!or!more!of!the!overall!aims!of!the!training.!For!each!of!the! modules!objectives!and!intended!learning!outcomes!were!drawn!up.!These!objectives!and! learning!outcomes!were!based!on!the!overall!aims.!!Based!on!the!learning!outcomes!the! didactical!approach!and!the!desired!profile!of!the!trainer!were!described.!Therefore,!the! description! of! each! of! the! modules! followed! the! same! format:! Objectives! –Learning! Outcomes! –! Didactical! Approach! –! Trainer’s! profile.! We! will! limit! ourselves! here! to! an! overview!of!the!learning!outcomes:!

! !

25

! ! A., European,external,QA,approaches, 1.! discuss!the!position!of!external!quality!assurance!in!the!European!Higher!Education! Area;! 2.! describe!how!higher!education!systems!and!their!specific!policy!choices!impact!the! external!quality!assurance!methodologies;! 3.! classify!(sub)national!models!of!external!quality!assurance;! ! B., Intercultural,awareness,

4.! position! one’s! own! national! culture! in! a! generic! frame! of! reference! for! cultural! differences!and!characteristics;!

5.! frame! questions! related! to! the! quality! of! higher! education! in! the! perspective! of! cultural!differences!and!characteristics;!

6.! perceive!culture!dependent!frames!of!reference!in!order!to!avoid!being!mislead!by! them;!

!

C., Audit,skills,for,diverse,national,settings,

7.! communicate! effectively! and! appropriately! with! all! actors! in! diverse! external! quality!assurance!settings;!

8.! employ! advanced! communication! techniques! for! successful! and! relevant! information!gathering!in!external!quality!assurance!procedures;!! It!was!decided!that!the!training!should!be!prolonged!to!1! ½!days!to!enable!a!better!coverage!of!the!3!dimensions.! The!target!group!of!experts!to!be!trained!in!the!second! pilot!training!were!described!as!“experts!who!can! participate!in!institutional!external!quality!assurance! procedures!(audits/accreditations/evaluations)!in! different!national!systems”.!

26

9.! assume! suitable! behaviour! for! particular! expert! roles! in! diverse! intercultural! settings!of!external!quality!assurance!procedures;!!

10.! cooperate! effectively! and! appropriately! in! decision[making! processes! of! an! audit! team.! The!trainers!had!to!meet!the!trainer’s!profile!and!were!carefully!selected.!A!preparatory! meeting!of!the!3!trainers!and!the!responsible!E[TRAIN!project!partners!(ÖAR!and!NVAO)! was!organised.!At!this!meeting!the!content!of!the!programme!was!agreed,!the!didactical! approach!was!discussed!and!the!assignments!and!materials!were!agreed.!The!pre[training! assignments!for!participants!consisted!of:!reading!a!selected!case[study!which!was!used!for! the!role!play;!a!text!on!European!QA;!short!profiles!of!3!national!agencies;!and,!a!matrix!on! which! experts! were! asked! to! position! their! own! national! QA! system.! This! matrix! would! then!be!discussed!during!the!training.!!!

The! programme! followed! the! sequence! of! the! modules! but! not! exactly;! to! increase! the! span! of! attention! during! the! day! it! was! decided! to! break! the! intercultural! awareness! module! in! two! sessions.! Intercultural! awareness! part! 1! was! scheduled! before! lunch! and! Part!2!before!the!end!of!the!day.!For!the!programme!of!the!training!see!ANNEX!3.!

!

The!training!was!held!on!23!and!24!April!2012!in!Vienna.!Ten!experts!participated!in!the! training.!As!in!the!first!pilot!training!they!were!nominated!by!the!ECA!member!agencies.!In! addition,! three! experienced! experts! from! the! Focus! Group! participated! in! the! training.! They!also!evaluated!the!training!immediately!after!the!end!of!the!training.!The!organising! team!consisted!of!the!three!trainers,!two!staff!members!of!the!Austrian!agency!(of!which! one!acted!as!observer)2_{,!and!two!staff!members!of!NVAO.!As!a!consequence!there!was!a!} mixed!group!of!people!from!Austria,!Belgium,!Germany,!the!Netherlands,!Norway,!Poland,! and!Spain.!! As!with!the!first!pilot!training!the!second!one!was!thoroughly!evaluated.!During!the!final! session! participants! were! asked! to! fill! in! an! evaluation! form! and! these! were! collected! before! going! to! lunch.! As! a! consequence! there! was! a! very! high! response! rate.! The! achievement!of!the!learning!outcomes!were!rated!and!a!number!of!other!questions!were!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! 2_!AQ!Austria!

! !

27

asked.!After!lunch!the!Focus!Group!shared!their!observations,!and!Notburga!Damm!from! AQ!Austria,!wrote!an!observation!report.!The!main!results!of!these!evaluations!were:! ! •! The!training!was!clearly!appreciated!by!the!participants;!the!overall!rating!was!7.8! on!a!scale!from!1!(poor)!to!10!(excellent).! •! Most!participants!felt!that!1!½!days!of!training!was!sufficient!and!feasible.! •! A!group!size!of!10!experts!worked!well,!especially!in!the!group!work!(2!groups!of!5! experts).!When!splitting!in!2!groups!for!a!role!play!or!discussion!(e.g.!when!making! judgements)! there! are! always! 2! trainers! needed! to! give! maximum! feedback! to! participants.!!

•! The! structured! approach! taken! worked! well;! the! content! structure! (e.g.! the! sequence!of!the!sessions)!could!even!be!further!refined!and!improved.!

•! The! case[study! and! role! play! generated! a! lot! of! enthusiasm! and! there! was! agreement!that!these!were!the!most!beneficial!parts!of!the!training.!

•! Intercultural!awareness!should!be!integrated!in!the!role!play/case[study.!

•! More! attention! is! needed! for:! the! classification! of! national! QA! systems;! the! role! and!attitude!of!the!auditor;!panel!chair!skills.!

!

! !

!

The! structured! approach! taken! in! the! second! pilot! training!of!experts!worked!well.!The!case[study!and!role! play! generated! a! lot! of! enthusiasm! and! there! was! agreement!that!these!were!the!most!beneficial!parts!of! the!training.!

28

3.2.

The pilot trainings for QA staff

Originally!in!the!E[TRAIN!project!proposal!the!staff!trainings!were!meant!to!train!QA!staff! members! to! become! trainers! of! European! experts! after! which! they! could! go! on! to! train! European! experts! themselves.! However,! it! turned! out! that! there! was! no! clear! profile! of! these!European!experts!which!was!shared!by!the!agencies!at!the!start!of!the!project.!As!a! consequence,! a! lot! of! time! and! energy! went! into! discussions! among! the! agencies! (Stakeholder! Group! and! Steering! Group)! and! experts! (Focus! Group)! on! reaching! an! agreement!about!who!should!be!trained!and!what!this!would!mean!for!setting!up!these! trainings.!Moreover,!it!gradually!became!apparent!that!agencies!preferred!so!called!‘train! the!trainers’!sessions!which!would!help!staff!members!to!become!better!national!trainers.! The!sequence!of!trainings!was!therefore!reversed:!first!the!pilot!training!of!experts!took! place,!followed!by!the!training!of!staff.!!

3.2.1.

The first pilot training of QA staff

The!Stakeholder!Group!agreed!that!the!first!pilot!training!of!QA!staff!training!should!take! place!after!the!first!pilot!training!of!experts.!This!would!allow!for!the!involved!parties!to! reflect!upon!the!experts!training!before!carrying!out!the!staff!training.!The!staff!training! would!function!as!a!“train!the!trainers”!meeting.!The!original!idea!was!that!observers!from! the!agencies!who!were!present!at!the!first!pilot!training!of!experts!could!participate!in!the! staff! trainings! so! that! they! could! act! as! future! trainers! of! European! experts.! However,! when!these!staff!members!were!asked!what!their!preference!for!the!first!pilot!training!was! it!turned!out!that!the!vast!majority!wanted!a!training!which!would!help!them!improve!as! national! trainers.! Therefore,! it! was! decided! that! the! first! pilot! training! should! have! the! format!of!a!workshop!of!trainers!where!they!could!share!how!they!train!experts!in!their! agencies.! Mutual! learning! and! sharing! good! practices! were! important! goals! for! this! workshop.!Moreover,!there!should!be!overarching!outcomes!for!the!group!of!agencies!in! the! form! of! an! overview! of! training! practices! in! agencies,! drafting! recommendations! concerning!national!trainings,!and!initiating!the!development!of!a!skills!profile!of!trainers.!! The!workshop!took!place!on!3rd!November!2011!in!Vienna.!For!the!programme!of!the!first! pilot!training!of!QA!staff!see!ANNEX!4.!

!

!

29

! !

There! were! in! total! 15! participants! in! this! workshop! representing! (prospective)! trainers! from! Austria,! Denmark,! Germany,! the! Netherlands! and! Flanders,! Spain,! Switzerland,! and! the! UK.! The! survey! of! good! practices! in! experts’! trainings! had! shown! that! the! Quality! Assurance!Agency!UK!had!a!long!lasting!experience!with!an!extensive!training!programme,! particularly!in!the!field!that!was!mostly!new!but!of!increasing!relevance!for!ECA!members,! namely! the! training! of! institutional! reviewers.! The! first! part! of! the! programme! was! therefore! reserved! for! a! presentation! by! Dr! Gillian! King! (consultant! and! former! Deputy! Director! of! the! Reviews! Group! at! QAA)! of! the! QAA! training! programme.! Then! the! participants!presented!the!training!practices!in!their!own!agencies!in!which!they!focused! primarily!on!the!following!aspects:! ! •! Description!of!experts!profile!(which!experts!are!trained)!! •! Content!and!programme! •! Preparation!and!materials! •! Conduction!of!the!training! •! Learning!outcomes! •! Didactic!approach!and!methods! •! Skills!of!trainers! !

The! first! pilot! training! of! staff! members! from! QA! agencies! should! have! the! format! of! a! workshop! of! trainers!where!they! could! share!how! they! train! experts! in! their! agencies.! Mutual! learning! and! sharing! good! practices!were!important!goals!for!this!workshop.!

30

For!an!overview!of!the!training!programme!characteristics!of!these!agencies!see!ANNEX!1.! Although!there!are!some!significant!differences!in!how!these!trainings!are!conducted!there! are!also!commonalities.!It!became!apparent!that!most!agencies!have:! ! •! The!same!training!or!briefing!for!all!experts;!! •! A!focus!on!the!review!framework;!! •! Legal!aspects!as!preparatory!materials;!! •! One!day!of!training!or!less;!! •! A!tendency!to!focus!didactics!on!presentations!and!discussions,!although!the!use!of! case!studies!is!increasing;!! •! Trainers!from!staff!or!the!management!of!the!agency.!! ! The!participants!also!discussed!ten!draft!recommendations!to!agencies!for!the!training!of! experts.!A!final!draft!of!these!recommendations!is!included!as!ANNEX!5.!

The! workshop! was! evaluated! which! resulted! in! an! overall! rating! of! 2! on! a! scale! from! 1! (excellent)! to! 5! (poor).! During! the! concluding! session! the! participants! expressed! their! preferences!for!the!second!pilot!training!to!include!guidance!on!the!structural!aspects!of! setting! up! a! training! and! for! opportunities! to! advance! their! communication! skills! as! a! trainer.!!

!

3.2.2.

The second pilot training of QA staff

All!staff!members!who!are!responsible!for!the!training!of!the!panel!members!within!their! agency! have! an! extensive! knowledge! of! the! subject,! i.e.! external! quality! assurance,! and! they! are! experienced! with! the! practicalities! of! external! quality! assurance! procedures! as! well.!At!the!same!time!the!majority!have!not!undergone!any!preparation!for!the!specific! task!of!converting!that!knowledge!into!a!training!course.!They!had!no!structured!training!in! how! to! train! panel! members! but! follow! more! or! less! a! “Do[it[yourself”! instead! of! a! didactic[methodical!approach.!This!is!why!it!was!decided!that!the!second!pilot!training!for! QA!staff!should!be!designed!as!a!“Train[the[trainers”!workshop,!conducted!by!professional! trainers.!The!staff!members!would!be!given!tools!which!they!can!use!when!designing!their!

! !

31

training!course!back!home.!On!10!and!11!May!2012!the!train[the[trainers!workshop!was! held!in!Madrid.!The!workshop!was!divided!in!two!parts;!on!the!first!day!Gillian!King!was!the! trainer!for!the!structural!part!of!the!training!and!on!the!second!day!the!Polish!consultants! Przemyslaw! Kotecki! and! Anna! Giniewicz! were! responsible! for! the! communication! skills! training.!The!learning!outcomes!and!the!content!of!the!two!training!days!were!developed! by!the!trainers.!

The!programme!for!the!two!training!days!is!included!as!ANNEX!6.!The!content!of!the!first! training! day! is! mainly! included! in! the! publication! ‘Handbook! for! the! training! of! experts’.! Fifteen!learning!outcomes!were!developed!for!the!two!training!days.!

!

Table 1. Learning Outcomes for the two days training of staff members Day,1, 1. be!aware!of!their!own!skills!as!a!trainer!and!have!identified!areas!for!development! higher!education!area! 2. be!aware!of!the!need!to!know!their!participants!and!the!material!to!be! communicated! 3. be!able!to!write!session!aims!and!outcomes! 4. be!able!to!choose!delivery!methods!which!suit!the!participants!and!the!material! 5. be!able!to!develop!training!materials!(notes,!visuals)!which!deliver!the!outcomes! 6. consider!assessment!tools,!if!required! 7. understand!the!importance!of!evaluating!the!programme.! Day,2, 8. be!able!to!identify!the!barriers!in!communication! 9. be!able!to!break!down!the!communication!barriers! 10. be!aware!of!different!aspects!of!communication! 11. be!able!to!work!with!the!group!dynamics!! 12. be!able!to!work!with!“difficult”!participants!! 13. understand!the!role!of!auto[presentation!in!delivering!the!training! 14. be!able!to!use!auto[presentation!techniques!for!delivering!of!the!training! 15. be!aware!of!methods!and!strategies!for!stress!management! ! There!were!10!participants!from!the!same!countries!as!in!the!first!pilot!training.!A!needs! questionnaire! was! sent! to! participants! asking! them! for! their! experience! as! a! trainer,! whether!they!had!been!trained!as!a!trainer!before,!their!strengths!as!a!trainer,!and!areas! for!development.!!

32

The! training! was! evaluated! by! participants! filling! in! an! evaluation! and! by! one! of! the! participants!who!had!acted!as!observer.!The!evaluation!led!to!the!following!results:! • The!overall!rating!on!a!scale!from!1!(poor)!to!6!(excellent)!was!:!5.2!for!day!1!and! 4.8!for!day!2.! • Participants!considered!day!1!to!be!particularly!useful!and!well[structured.!! • The!agendas!on!both!days!were!a!bit!overloaded;!the!time!pressure!was! noticeable.!! • A!possible!solution!could!be!that!the!first!day!would!be!stretched!into!2!days!and! some!of!the!objectives!of!the!second!day!integrated!in!the!first!day.! ! !

The! second! pilot! training! of! staff! members! from! QA! agencies!was!a!train!the!trainers!workshop.!It!was!valued! highly!by!the!participants.!!

!

!

33

4.

Conclusions

he! experiences! with! the! pilot! trainings! and! the! discussions! during! the! E[TRAIN! dissemination!conference!held!in!Madrid!on!14!and!15!June!2012!gave!very!useful!! feedback!on!a!possible!follow[up!to!the!project.!The!following!observations!may! guide!the!proposals!for!future!trainings!of!experts!and!staff!organized!by!ECA:!

4.1.

The training of European experts

• Expert!trainings!should!be!“tailor[made”!depending!on!the!target!group,!e.g.:!! o Experts!for!institutional!reviews!! o Experts!for!assessments!of!joint!programmes!(single!accreditation! procedures)! o Experts!for!the!assessment!of!HEIs!or!programmes!requesting!the!ECA! certificate!for!internationalisation!(CeQuInt!project).! • A!community!of!trained!experts!should!be!established!by!carefully!considering:! o Recruitment:!to!prevent!an!elite!corps!of!(retired)!experts!trainings!should! not!solely!rely!on!nominations!of!experienced!experts!but!be!open!to! newcomers!with!less!experience!in!external!QA.!Recruits!should!share!the! agencies’!values,!ethos,!and!ideas!of!professionalism.!! o Inclusion!in!the!European!database,!preferred!participation!in!panels,!and! maintenance!of!communication!(e.g.!newsletter,!providing!opportunities! for!updating!skills,!expression!of!appreciation)! o Performance!of!experts!should!be!reviewed! o Student!experts!should!be!included!in!trainings!and!in!the!community!of! experts!but!they!may!require!some!specialised!content.! • Structure!and!content!of!the!training:! o The!structured!approach!taken!in!the!second!pilot!training!should!be! maintained!and!refined! o The!content!of!the!training!should!be!adapted!according!to!the!profile!and! experience!of!experts!

T

34

o Knowledge!of!European!external!QA!approaches!should!be!enhanced!by! expanding!the!ECApedia!and!making!useful!references!to!materials!in!the! ECApedia!throughout!the!training!and!in!a!designated!assignment! specifically! o Intercultural!awareness!should!be!integrated!in!the!discussion!on! European!QA!systems!and!in!the!training!of!audit!skills!(including!a!case[ study!and!role!play)! o Improve!the!learning!effect!of!the!role!play!by!providing!country[specific! information!on!local!higher!education!! o More!attention!needed!for:!the!role!and!attitude!of!the!auditor;!panel! chair!skills;!enhancement[led!vs.!assessment[focused!reviews;!creating!an! atmosphere!of!trust!and!openness.! • Organisation!and!finances:! o Trainings!should!be!organized!on!an!easily!accessible!location,!if!possible!in! cooperation!with!other!networks!to!increase!European!support!and!the! funding!base! o Financial!sustainability!can!be!enhanced!by!sharing!costs!between!involved! networks,!agencies,!and!experts! o Organize!a!1!½!day!training!for!approximately!10!experts!who!can!be!split! in!2!groups!guided!by!2!trainers! o Trainings!should!be!organised!at!least!twice!a!year!and!more!often!when! the!need!occurs.! ! ! Expert! trainings! should! be! “tailor[made”! depending! on! the! target! group! and! a! community! of! trained! experts! should!be!established.!

!

!

35

4.2.

The training of QA staff (train the trainers)

• Train!the!trainers!workshops!of!2!days!should!be!organised!annually!for!trainers!of! national!agencies! • The!train!the!trainers!workshops!may!adapt!the!objectives!and!learning!outcomes! according!to!the!needs!of!the!participants! • Future!train!the!trainers!workshop!may!focus!on!the!development!of!a!community! of!experts!and!on!increasing!the!effectiveness!of!recruitment!and!performance!of! experts! • A!community!of!trainers!should!be!established!by!a!regular!exchange!of!trainers! among!national!agencies!and!maintenance!of!communication!(e.g.!newsletter,! providing!opportunities!for!updating!skills,!etc.)! • The!costs!of!the!training!should!be!shared!by!the!agencies!involved.! ! ! ! ! !

Train! the! trainers! workshops! of! 2! days! should! be! organised!annually!for!trainers!of!national!agencies.!

36

!

!

!

37

Annex 1:

Detailed overview of training characteristics of QA agencies

Q ua li ty As su ra n ce Ag en cy Tr ai n in g’ s ta rg et au d ie n ce Tr ai n in g co n te n t Pr ep ar at or y ma te ri al Tr ai n in g p ro ce ss Le ar n in g ou tc om es Di d ac ti cs Tr ai n er s’ ACE De n m ar k Sa m e tr ai ni ng f or rep res en ta ti ves o f al l s tak eh ol d er gr ou p s As se ss m en t fr am ew or k ; ac cr ed . c ri te ri a; ro les a n d t as k s of e xp er ts No e xc es si ve pr epa ra ti on (l eg al fr am e-wo rk , g u id es , exa m p le of a rep ort ) 1 d ay tr ai n in g; lu n ch a s a n im p or ta n t s oc ia l co m p on en t Kn ow as se ss m en t fr am ewo rk an d c ri te ri a/ in te rp re tat ion ; mi n ima l t h re sh ol d le ve l; k n ow ro les a n d t as k s of exp ert s; so ci al c om p et en ce s Pr es en ta ti on ba se d tr ai ni ng ; di sc us si ons ; lit tle c as e w or k , exp eri en ces o f exp ert s Tw o tr ai n te ac h in g exp eri en ce, ac cr ed it at exp eri en ces ANE CA Sa m e tr ai ni ng f or rep res en ta ti ves o f al l s tak eh ol d er gr ou p s + in te rn at io n al exp ert s Se le ct io n cr it er ia fo r ch oo si n g exp ert s; g en d er an d t er ri tor ial ba la nc e as sur ed Ev al u at io n c on - te xt + p ro ce ss ; to ol s; ac cr ed . cri teri a an d th ei r in te rp re ta tio n ; ANE CAs r ol e, pr ac ti ca l i ss ue s St anda rdi ze d ma te ri al ( le ga l as p ec ts , gu id el in es an d st an d ar d s, ev al u at io n pr ot oc ol s, c as e st u d y fo r sm al l gr ou p w or k s) 1 d ay tr ai n in g Pr ac ti ca l or ie n tat ion , w it h fe ed b ac k lo op s Kn ow le ga l a sp ec ts ; k n ow ev al u at io n p ro ces s; s k il ls t o an al ys e a d egr ee ; in te rp re ta tio n o f c rit er ia ; s of t sk il ls; e th ic s an d pr of es si ona li sm , t ea m pl ay ; em p at h y Le ct u ri n g ab ou t ev al u at io n pr oc es s; Wo rk sh op I : st an d ar d s an d gu id el in es ; in te rp re ta tio n o f cri teri a Wo rk sh op I I: sm al l g ro u p s, ca se st u d ies , si m u la ti on o f ev al u at io n Ex p er ts f ro ANE CAs te am w it h exp eri en ces ev al u at io ac ad em ic QA exp eri en ces exp eri en ces ma n ag eme de gr ee s AQ U Ca ta lu n ya Sa m e tr ai ni ng f or rep res en ta ti ves o f al l s tak eh ol d er gr ou p s; Ad d it io n al ly “ ad ho c” c our se s fo r st u d en ts Se le ct io n cr it er ia fo r ch oo si n g exp ert s Ev al u at io n p ro - ces s, co n text , ac cr ed . c ri te ri a, IT t oo ls Spe ci al c ont ent fo r stu d en t co u rs es : pur po se A Q U , st u d en ts ro le in co m m it tees , et c Ma n ag em en t of exp ert s is p art of in te rn al Q A sy st em 1 d ay tr ai n in g Ad h oc c ou rs es fo r stu d en ts 0 .5 da y Im p ro ve m en t or ie n te d a sp ec ts; ap p li cat ion of ac cr ed . c ri te ri a; te am w or k a n d s oft sk il ls Le ct u ri n g; di sc us si on of exa m p les ; ca se st u d ie s (P B L ) in clu d in g fe ed b ac k St ra te gi c ta u gh t b y ma n ag eme so ft s ki ll s, pr obl em ba im p ro ve m sk il ls b y p ma n ag er s

EV A Al l e xp er ts a re tr ai n ed (e xc ep t exi st in g exp ert s) Se le ct io n is ba se d on p ed agogi cal kno w ho w a nd Q A exp eri en ces EV A ’s c on ce p t; ac cr ed it at ion as me th od , a cc re d . cri teri a; ro les an d t as k s of exp ert s Le tt er w it h ta sk s of exp ert s; Le gi sl at iv e te xts ; L O ma te ri al 3 h ou rs t rai n in g Kn owl ed ge o f ac cr ed it at io n a s me th od ; ro le o f ex p er ts , k n ow as se ss m en t cr it er ia Pr es en ta ti on s an d p le n ar y di sc us si ons (c as e stu d ie s pl anne d fo r fu tu re ) EV A ma n ag eme n t + ext ern al co u n sel lo r w it h th or ou gh exp eri en ces in ac cr ed it at ion FH R 1 Wo rk sh op s fo r co u n ci l m em b ers an d ac ad em ic me mb er s Se le ct io n cr it er ia fo r ex p er t p an el me mb er s Pu rp os e of as se ss m en t; as se ss m en t fr am ew or k ; na ti ona l c ont ex t, ro les o f exp ert s an d s taf f, “good que st io ns ” No f or m al tr ai n in g ye t No f or m al tr ai n in g ye t Ne ed ed L O a cc or d in g to su rv ey s: K n ow h ow a b ou t as se ss m en t fr am ew or k an d cri teri a/ in terp ret at io n ; k n ow ro les a n d t as k s of exp ert s; kno w ho w t o as k sm ar t que st io ns ; kno w ho w t o w ri te a rep ort No f or m al tr ai n in g ye t No f or m al tr ai n in g y et FI B A A ac ad em ic s, pr of es si ona ls st u d en ts: r eg u la r tr ai n in gs o f n ew rev iew ers ( 2 p er an n o) an d se m in ar s fo r exp eri en ced exp ert s (2 p er an n o) Ne w co m er s: le ar n a b ou t as se ss m en t pr oc edur e, na ti ona l reg u la ti on s an d Eu ro p ea n st an d ar d s as se ss m en ts fr am ew or k . Ex p er ie n ce d exp ert s: n ew reg u la ti on s, st an d ar d s an d tr en d s, co n si st en cy o f pr oc edur es Ba ck gr ou n d do cum ent s (r eg u la ti on s); FI B A A as se ss m en t gu id e 1 d ay tr ai n in g; 4 tr ai n in gs p er ye ar or gan iz ed Ne w co m er s le ar n r ol es a n d ta sk s of ex p er ts ; di ff er enc es co n d it io n s/ reco m m en d at io n s; as se ss m en t m et h od s, n at ion al an d E u rop ean r egu lat ion s. Ex p er ie n ce d e xp er ts le ar n ab ou t ch an ge s of r egu lat ion s, in te rp re ta tio n o f r eg u la tio n s an d ab ou t sp ec ial it em s (i .e . jo in t p ro gr am m es ) Pr es en t ca se st u d ie s; d isc us s exa m p les , wo rk in g on te xts , r ep or ti n g of ow n exp eri en ces Ex p er ie n ce d st af f of a ge n cy ; exp eri en ces in QA p ol ic ie s an d pr oc edur es

1 "Wi th "th e" le ga l"v al id ity "o f"th e" H S2 QS G" a" ne w "a nd "s in gl e" ag en cy "(A ge nc y" fo r"Qu al ity "A ss ur an ce "a nd "A cc re di ta tio n" Au st ria )"w as "e st ab lis he d" w hi ch "is "re sp on si bl e" fo r"ext er nal "q ual ity" as su ran ce" in "th e" H E" sec to r"( Pu bl ic " Uni ve rs iti es ,"Uni ve rs iti es "of "A ppl ie d" Sc ie nc es ,"P riv at e" Uni ve rs iti es )." Th e" ‘Q ual ität ss ic he ru ng sr ah m en ge se tz ’"( Q SRG )"e nt er ed "in to "fo rc e" on "1s t"M ar ch "2012. " Th e" FH "Co un ci l"p er si st ed "u nt il" 31s t"Au gu st "2012. ""