Peer Review Feedback - What it is and How to Use It

Instructions for accessing Peer Review Feedback can be found on page 5 of this document. Making Sense of Your Peer Review Feedback

We have replaced the rubric-based feedback form used in previous years with a checklist-based one. Checklists are superior to rubrics when the reviewer’s objective is to identify the presence/absence of specific things.

The feedback form is divided into sections. There are scores for each section and a total score for each report. Peer reviewers were asked to provide a comment to every ‘no’ rating. It is our intention that you will be able to use this feedback/rating as a very specific resource to help make future projects better. Contact Michele Marden ([email protected]) if you would like to discuss your feedback with a LAC coach (or if you would like coaching help for this year’s projects). Michele will also collect any SAC complaints about feedback.

Feedback Form Sections

 Overall Mechanics of Reporting – these scores represent the degree to which reports

were complete (e.g., included essential attachments; identifying information removed; etc.).

 SAC Participation/Coordination – core outcome assessment requires significant

coordinated effort from SAC members: these scores indicate the degree to which that coordination was demonstrated.

 Assessment Process – these scores represent the degree to which assessments

demonstrated technical components that help to ensure valid, reliable results.

 Student Achievement - scores here represent the degree to which the reports

provide valid and reliable information about student attainment of core outcomes.

Low scores in a section indicate that your SAC’s core outcome assessments can benefit from increasing focus/attention in that area, so please review those

carefully. While low scores in a section can be informative, the best things to attend to are the specific ‘No’ evaluations. ‘No’ ratings mean that your peer review team determined that an important aspect of good assessment was missing from your report. As a rule, focus on correcting those aspects of your assessment projects. If you are not sure how to correct any aspects of your assessment projects, make use of our Help Guide (available from the LAC web page) and/or contact Michele Marden, who will be supervising the LAC Coaches this year.

There will be some changes to this year’s report forms, but, the specific expectations you have gotten feedback on this year will still be the focus of feedback next year. In addition, we will be adding three related, additional, explicit expectations to LAC reporting. These additions are based on our quantitative analysis of the reports and peer review feedback.

1. When possible, SACs are now being asked to identify an expected level of student

performance (sometimes called a ‘benchmark’) in their assessment reports. For

example, if a SAC is using a rubric to rate student papers, the SAC should indicate which score on the rubric (e.g., 2 or 3) satisfies their collective expectation for student performance in that course/assignment.

2. When reporting the results of your assessments (that used a rubric), do not

average the scores. Simply report the number (or frequency) of students who scored at each rating.

3. When reporting the results of your assessments, do not combine scores for

elements of a rubric that do not measure the same outcome. Report them separately. For example, if one assignment is being scored for written

communication and critical thinking, report those scores separately.

You will be able to access this year’s report forms, as well as the checklist reviewers will be using this year, at the Learning Assessment Council’s web page early fall term.

Main Findings from 2013-2014 Reports

32/80 possible LDC SAC assessment reports included enough information to

determine students’ outcome attainment rates. Since outcome attainment rates are central to assessing student learning, we want to increase this number. This is the reason for adding the ‘expected level of student performance’ information in this year’s assessment reports.

Using a model developed from the data in the 2013-2014 reports, we estimate that the typical outcome attainment rates in LDC courses at PCC is between 70-80%. While there is limited information about these outcome attainment rates at other institutions, our performance seems consistent with other similar institutions.

LDC Outcome Attainment Data

A preliminary, pilot study was conducted comparing outcomes that met the criteria for a level of proficiency consistent with associate’s degree level expectations (using the Degree Qualifications Profile). These more demanding outcomes are attained at the rate of 58-73%. “Easier” outcomes are attained more often (73-85%). This difference is likely to be real. This data suggests SACs should attend to the difficulty of the benchmarks they are establishing and keep in mind that students are likely to need increased levels of instruction in order to better attain more challenging, associate’s degree level outcomes.

Summer Peer Review Summary Data

Possible scores on ranged from 0-18. Typical LDC reports score 14-17; higher ones score 18; lower: 13 and below.

Subsection Comparisons

The feedback for summer peer review was divided into 4 sections. Use the following charts to see how LDC reports scored in each section. Each chart represents one subsection (described above on page 2). The numbers along the bottom represent the subsection score for a report; and the numbers in the left column represent the number of reports receiving that particular score.

LDC FIGURES (60 REPORTS) 0 10 20 30 40 1 2 3 4 5

OMR

SPC

APr

SA

 The SAC Participation and Coordination and the Assessment Process sections were less strong.

Item Comparisons/LDC

It will be useful to many SACs to see the higher and lower scoring items from the summer peer review feedback form across all LDC SACs.

Higher Scoring:

 Has all student identifying information been removed and does the process of this appear sound and, whenever possible, ensure the anonymity of the instructor?

 Are there plans to inform all relevant members of the SAC of the results of the assessment project and to provide the necessary training (if needed)?

 Was a direct assessment conducted or was a compelling rationale for using an indirect assessment presented?

Lower Scoring:

 Does the assessment use the population or a recommended sampling strategy?

 If faculty are assigning the ratings, are both full and part-time faculty evaluating student work?

 If a rubric was used, was inter-rater reliability at recommended levels? Other Findings

 Were all of the data shown in the report disaggregated in a way that supports specific and meaningful faculty analysis of student learning?

 Were any averages reported where the peer reviewers believe meaning of the results might have been “averaged away”?

We looked for the presence/absence of these two indicators of quality assessment, without rating any SAC reports for them. Our findings indicate that our LAC coaches and support documents should provide better assistance in these two areas.

If you have questions or want to arrange to work with one of the LAC coaches, contact Michele Marden.

Finding Your Learning Assessment Report Feedback

Each year, the Learning Assessment Council coordinates a peer review process so that PCC faculty can comment on other faculty’s core outcome assessment projects. The peer review feedback for each SAC has been posted to PCC Spaces.

Accessing Learning Assessment (LA) Report Feedback:

These instructions are intended for faculty use. If you follow the steps below and find you do not have access to your SAC’s page, contact Susan Wilson at [email protected] for assistance. There is a link to Spaces on the PCC Intranet home page. For those working offsite from a non-networked computer, it is easiest to enter Spaces via My.PCC using the steps outlined below:

1. Log into My.PCC

2. Click on the Employee tab

3. Go to the Tools channel and click on the Spaces link found under the Communication header

4. Log into Spaces using the same user name/password as your My.PCC login

5. At the Dashboard, type "Learning Assessment Report Feedback" in the search box at the top right of the screen

6. See the Learning Assessment Council Annual Report Feedback home page appear on the screen

7. Access a specific SAC page by going to the gray box at the left and clicking the "+" symbol in front of the alphabetic grouping to which your SAC belongs

8. When the gray box expands to reveal the names of each SAC in that range, click on the name of your SAC

9. At the department home page, hyperlinks will appear if LA report feedback is available to view

10. Make the page a "favourite" to avoid searching for it each time you go to Spaces. To do this, go to the Committees tab and see a list of Spaces pages appear. Scroll down the list until you find Learning Assessment Report Feedback and click the star icon that appears to the right. Once the star is highlighted, the targeted committee will pop up whenever the Favourite button is selected at the Dashboard.

If you're wondering why we spell ‘favourites, with the "u," it's because the software program comes from Australia. They’re not as conservative with their vowels as we are.