Unlocking learners' evaluative skills: a peer review perspecive

David Nicol

1. Unlocking learners’ evaluative skills
A peer review perspective
David Nicol
Emeritus Professor of Higher Education. University of Strathclyde
Visiting Professor, University of Ulster and Swinburne, Australia
Sheffield-Hallam University 15 Sept 2015

2. Peer review
Definition of peer review
Peer review is an arrangement whereby students
produce a written assignment then review and
provide comments on assignments produced by peers
in the same topic domain.
In peer review, students produce feedback reviews
for peers and receive feedback reviews from peers.

3. Plan for Session
Brief introduction to peer review
You experience peer review – produce assignment (5
mins) and review two peer assignments (10 mins)
Reflection and discussion
Drawing on case examples – engineering, sociology
bio-chemistry

4. Introduction
Research on peer review has been confounded by
three factors (i) an over-focus on peer assessment
rather than peer review (ii) a bias towards examining
the benefits of receipt of feedback reviews rather
than the production of feedback reviews and (iii)
studies examining general benefits of peer review
without distinguishing its component parts.
Today the focus is primarily on producing feedback
reviews

5. Your assignment
Write a convincing argument for having students review the
work of peers (the reviewing component only). Provide
evidence for your argument (from literature, logical, from you
own experience, convincingly anecdotal) and identify and
respond to any obvious counter-arguments.
Criteria for good argument are: convincingness of argument (ii)
evidence in support of argument (iii) identification and
responses to obvious counter-arguments.
Five minutes for this task Normally students can produce about
10-14 lines of text  Assume it is a draft or initial free writing

6. The peer review task
Review and provide feedback comments on the work
of two peers using the given criteria.

7. Argument from peer 1
I think that students would gain understanding of their own work in
the process of reviewing the work of peers. When reading another
student’s work, the reviewer would more likely be able to see areas
where improvements could be made. It is often the case that it is
easier to identify others’ weaknesses than one’s own. When
reviewing the work of others, the student would engage in a process
of comparison with their own work. This leads to a form of reflection
otherwise not available. However, it could be argued that students
are not well-qualified to comment on the work of others. They do
not have the knowledge of the subject or the pedagogical training to
make valuable comments. This I do not agree with. Students are
often close to each other in their level of knowledge and writing and
would therefore be able to give constructive criticism. At the same
time, giving criticism would heighten awareness of their own
performance. 

8. Argument from peer 2
Having students review the work of peers should be a regular
activity in higher education because if students do this they will see
the way others tackle the same assignment and they will learn and
get ideas from this. Also, when they review they will have to apply
some criteria and this will help them to understand these criteria
better. In my experience students often produce poor assignments
because they do not understand what is expected, not because they
cannot do the work. Indeed, when I organise peer discussion of
criteria before a task this results in better quality work. However, it
is clear that there might be problems of plagiarism as in reading
peers assignments it is likely that students will copy without owning
the ideas themselves. This could be tackled, however, by having
students review and just say what they would do to improve their
own assignment (if they had the opportunity) without actually
getting them to do it. In this way, they would provide evidence of
interpretation rather than copying.

9. Criteria/questions for the peer review
(i) Summarise the core of the argument written by each
peer in one sentence.
(ii) a. Identify any hidden assumptions in thisargument
OR
b. Identify and formulate a good feedback principle
from the argument presented by this writer
(iii) Make one suggestion that would strengthen the
argument. Give a reason for your suggestion in a
sentence or two.
Time = 10 minutes = 5 minutes each review

10. Discussion and reflection
You: Reflection on your experience – the learning
from peer review
Me: present of some recent research findings using
‘student quotes’

11. Reflection and discussion
What did you learn from this peer review exercise?
How did you go about the reviewing task?
What ‘mental processes’ are elicited through the
reviewing activity?

12. What was the most valuable aspect of the reviewing process?
Rate each of the following on the following scale (where 0 is not valuable, 1 is of
some value and 7 is very valuable).
1.Seeing how peers had approached this task
2.Engagement with the criteria/questions
3.Comparing the peer assignments with your own
4.Making evaluative judgements about others’ work
5.Writing the feedback commentaries
6.Comparing one peer’s work with the other

7. Thinking about changes to your own assignment
0 1 2 3 4 5 6 7

13. Research: what do students say?

14. Engineering Design
Peer Project case example
DM 100 Design 1: first-year class
Dr Avril Thomson, Course Leader, Design
Manufacturing and Engineering Management (DMEM),
University of Strathclyde
[email protected]
Caroline Breslin, Learning Technology Adviser,
University of Srathclyde
[email protected]
Funded by JISC: www.reap.ac.uk/PEERToolkit/Design.aspx

15. Engineering Design 1
82 first-year students
Design a product – ‘theme eating and resting in the
city’
Research in groups (in city, in library etc.)
Individually produce a Product Design Specification
(PDS) – detailed requirements for and constraints on
design (rationale, performance,standards, manufacturing etc)
Given a PDS exemplar from another domain to show
what’s required (stainless steel hot water cylinder)
Online learning environment: Moodle and PeerMark
part of Turnitin suite

16. DM 100: Design 1
Peer review task
Individually, each student peer-reviewed and provided
feedback on the draft PDS of two other students
Criteria: (i) completeness (ii) convincingness of
rationale (iii) specificity of values (performance) (iv)
one main suggestion for improvements with reasons
Students used experience, giving and receiving
feedback to update own PDS
Peer review not assessed directly but 10% marks for
professionalism which included participation in peer
review.

17. Evaluation
1.Online survey completed by 64 students
2.Focus group interviews
3.Peer review comments recorded online
4.Course work marks compared to previous years

18. Results 1
Which aspects of the peer review did you learn from?
Giving feedback 10.9%
Receiving feedback 26.6%
Giving and receiving feedback54.7%
Neither giving or receiving 7.8%

19. Results 2
Did you modify your initial submission as a result of the
peer review activity?
Yes, as a result of the peer review
given
23.4%
Yes, as a result of the peer review
received
25.0%
Yes as a result of the peer review
given AND received
28.1%
No 21.9%
N/A 1.6%

20. Results: student comments
If yes, please give specific examples of modifications (n=41)
[Comments are from different students]
I added a couple of paragraphs and improved existing
paragraphs, this added two full A4 pages to my work
I provided more specific numeric values and expanded my
rationale after seeing someone else’s PDS and after
receiving feedback
I added a legal and patents section
Improved the rationale, included more facts
I made some of my numeric points more specific to my
final design concept.

21. Results: learning from RECEIVING reviews
Please give examples of what you learned from RECEIVING peer reviews
from other students (n=54)
Specific content mentioned: Depth of analysis needed, more numerical
data and figures, stronger rationale, how to structure it better etc.
Receiving peer reviews gave me insight into what others thought of my
work and gave me a direction to improve (reader response)
Where the PDS was confusing to understand (reader response)
Parts that I had previously missed were brought to eye such as market
competition (noticing)
The person who peer reviewed my PDS gave me positive feedback
which helped me a lot (motivational)
Not much, they...[the peer reviews]...weren’t very good (no value)

22. Results: learning from PROVIDING reviews
Please give examples of what you learned from PROVIDING peer reviews
of other’s work (n=47)
How to look at work critically that isn’t your own [critical judgement]
Thinking from a critical point of view [critical judgement]
I was given a greater understanding of the level of the work the
course may be demanding [attention to expectations/criteria]
Allowed me to see from an assessor’s perspective
[expectations/criteria]
When giving advice to people on theirs, it gave me greater perception
when reviewing my own work by listening to my own advice for
example [reflection/transfer]
I had a chance to see other peoples work and aspects of their work
that I felt were lacking in my work, this helped me to improve my work
[reflection/transfer]

23. Results: How you carried out peer review
Could you make any comments about how you carried out the
peer review? How did you evaluate the quality of the work to
provide a response to the peer review questions? (n=37)
I compared it to mine and ...and said how I would improve it
Partly by comparing my work to theirs
I tried to think about what I wrote and whether this product
design specification was better or worse

24. Focus groups
How did you go about reviewing?
‘I read it through and compared it with what I had done to see if
they had put something I had not done and then I added it in if
they hadn’t. The four questions...[criteria provided by the
teacher]...were useful as they provided a framework for the
review. If we hadn’t had the questions it would have been
difficult. I did the reviews separately and then answered one then
the other. The first was a better standard than the other – so I
used the ideas from the better one to comment on the weaker
one. I also read the guidelines in class when I did the peer review.
There were ideas from the good one that I hadn’t even thought
of in mine’

25. Results: reviewing
In the focus groups the effects of the review questions (criteria)
was probed further. Typical comments were:
You compare it (the other student’s work) to the criteria but
then in the back of your mind you’re comparing it to our own at
the same time.
I went down the questions and compared it to my own..I was
trying to think what has this person done. Have they put in more
effort or knowledge than me.
I went through the questions keeping my own in mind
You’ve got what you’ve done at the back of your mind while
going over theirs so you see where you’ve gone wrong without
anyone pointing it out so you learn it yourself
‘Reviewing is grounded in comparisons with students’ own work’
(Nicol, Thomson and Breslin, 2013)

26. Benefits of reviewing (1)
Reviewing elicits multiple acts of evaluative judgement
1.Evaluate peer’s work against own
2.Evaluate one peer’s work against another (and own)
3.Evaluate work against given criteria to produce response
The pre-condition for these effects
1Students must first have produced an assignment in the ‘same
domain’ as those that they are asked to review
To what extent does your experience resonate with this finding?

27. Benefits of reviewing (2)
Students both create and apply evaluative criteria
1Create criteria as they compare work with own (holistic
judgements)
2Apply explicit criteria (analytic) to instances of practice when
the produce a written response (analytic judgements)
3Simulates what experts do when they make evaluative
judgements
‘Through reviewing students generate richer criteria than those
provided by the teacher but sounder criteria than those they might
be able to formulate on their own’ (Nicol, Thomson and Breslin,
2013)
Nicol, D., Thomson, A and Breslin, C. 2013. Rethinking feedback in higher education: a peer
review perspective. Assessment and Evaluation in Higher Education

28. Focus groups
What do you think is best for learning – giving or
receiving feedback?
‘For me it would probably be to give feedback because I think
seeing what other people have done is more helpful than getting
other people’s comments on what you have already done. By
looking at other people’s work you can see for yourself what you
have forgotten or not even thought about. When people give
feedback on yours they generally just talk about what is there.
They don’t say, well I did this on mine and you could put that in
yours.’

29. Focus groups
What do you think is best for learning – giving or
receiving feedback?
I think when you are reviewing...[the work of peers]...it’s more a
self-learning process, you’re teaching yourself; well, I can see
somebody’s done that and that’s a strength, and I should
maybe try and incorporate that somehow into my work.
Whereas getting...[teacher]... feedback you’re kind of getting
told what to do; you’re getting told this is the way you should
be doing it, and this is the right way to do it. You’re not really
thinking for yourself.... I think...[reviewing]... would help you
not need so much of teacher feedback, if there was more of
this. Whereas, I think if you’re not being able to do...
[reviewing]... then you will always be needing more...[teacher
feedback]...

30. Peer review: a new perspective on feedback
Students construct feedback ‘meanings’ for themselves
while they produce them for others
Puts feedback processes in the hands of the student
See many examples of the same work of different quality
Reduces their need for teacher feedback
Suggests another focus for teacher feedback – helping
students calibrate the quality of their own judgements
Nicol, D., Thomson, A and Breslin, C. 2013. Rethinking feedback in higher education: a peer
review perspective. Assessment and Evaluation in Higher Education. 39(1), 102-122

31. Feedback in professional and
workplace settings
1.In the professions, feedback never comes from a
single source: task is usually to evaluate, weigh up
and reconcile and respond to different and
sometimes contradictory feedback perspectives.
2.Professionals are not just ‘consumers’ of feedback
but also ‘producers’
Nicol , D. Thomson, A and Breslin, C. 2013. Rethinking feedback
practices in higher education: a peer review perspective.
Submitted to Assessment and Evaluation in Higher Education

32. Principles of good peer review design
1.Encourage an atmosphere of trust and respect
2.Use a range of different perspectives for the review tasks
3.Give practice in identifying quality and in formulating
criteria
4.Require well-reasoned written explanations for feedback
response
5.Facilitate dialogue around the peer review process
6.Integrate self-review activities into peer review designs
7.Encourage critical evaluation of received reviews
8.Provide inputs that help reviewers calibrate their
judgements
Nicol, D (in press) Guiding principles for peer review: unlocking learners’ evaluative skills. See
also http://www.reap.ac.uk/PEERToolkit/Design.aspx

33. Design decisions
1.Target task – factual or open-ended (design, computer
programme, essay, report etc)
2.Unit for task: individual, pair, group
3.Unit for review: individual, pair, group work
4.Matching reviewers: random, by ability, by topic
5.Number of reviews – more is better
6.Privacy: anonymous or known reviewer and/or author
7.Peer review criteria – not-given: guidelines: fixed
format
8.Review focus: holistic v analytic, content or process
9.Use of received reviews: drafts, self-review, new task
10.Requesting and responding to feedback
11.Grading: no marks, marks for participation, for
reviews, marks for self-review after peer review
Available at http://www.reap.ac.uk/PEERToolkit/Design.aspx

34. References
Boud, D. and Associates (2010) Assessment 2020: Seven propositions for assessment reform in higher
education. Sydney: Australian Learning and Teaching Council (ALTC) Available from
www.assessmentfutures.com
Cowan, J. (2010) Developing the ability for making evaluative judgements, Teaching in Higher Education,
15(3), 323-334.
Nicol, D. and Macfarlane-Dick, D. (2006), Formative assessment and self-regulated learning: A model and
seven principles of good feedback practice. Studies in Higher Education, 31(2), 199-218
Nicol, D (2010) From monologue to dialogue: improving written feedback in mass higher education,
Assessment and Evaluation in Higher Education, 35:5, 501-517
Nicol, D (2010) The foundation for graduate attributes: developing self-regulation through self and peer
assessment, QAA Scotland, Enhancement Themes. Available at:
http://www.enhancementthemes.ac.uk/resources/publications/graduates-for-the-21st-century
Nicol, D (2011) Developing students’ ability to construct feedback, QAA Scotland, Enhancement Themes.
Available at http://www.enhancementthemes.ac.uk/resources/publications/graduates-for-the-21st-century
Nicol, D (2013), Resituating feedback from the reactive to the proactive. In D. Boud and L. Malloy (Eds)
Effective Feedback in Higher and Professional Education: understanding it and doing it well, Routledge UK
Nicol, D., Thomson, A. and Breslin, C. (2014) Rethinking feedback practices in higher education: A peer review
perspective, Assessment and Evaluation in Higher Education, 39(1), 102-122
Nicol, D. (in press) Guiding principles of peer review: unlocking learners’ evaluative skills. In Advances and
Innovations in University Assessment and Feedback. Eds C. Kreber, C. Anderson, N. Entwistle and J. McArthur.
Edinburgh University Press.
Sadler, D.R (2010) Beyond feedback: Developing student capability in complex appraisal, Assessment and
Evaluation in Higher Education, 35:5, 535-550

35. 1. Atmosphere of trust and respect
Why
1.Requires commitment and collaboration to work
2.Changes power relations: teacher gives up place of
authority
3.Raises issues of academic integrity
How
1.Explain why important e.g. for learning and in
professions
2.Explain that purpose is formative not about marking
each other
3.Show how to give constructive feedback
4.Model review process in tutorial (Draper, 2009)
5.Engage students in discussing the merits themselves

36. 2. Use a range of different perspectives
for the review tasks
Why:
1.Elaborates existing, and builds new, knowledge
2.Experts can evaluate work from many perspectives
How:
1.Disciplinary perspective
2.Holistic perspective (evaluate work as a whole)
3.Stakeholder perspective (the nurse, doctor, patient)
4.Reader- response perspective (i.e. non-judgemental)
5.Graduate attributes perspective (e.g. ethical)
6.Contrastive perspective (e.g. another theoretical
vantage point)

37. 3. Give practice in identifying quality and in
formulating criteria
Why:
1.Develops holistic judgement
2.Helps students recognise and explain quality rather
than being told
3.More like professional practice – where criteria are
emergent, tacit and cannot be fully codified.
How:
1.Don’t always give criteria
2.Students review and externalise criteria for
judgements
3.And rank works and explain their ranking decisions

38. 4. Require well-reasoned written explanations
for feedback responses
Why:
1.Engages students in knowledge construction/ elaboration
and reflection
2.Learn to use the discourse of discipline
3.Multiple reasons for evaluations lost if only a grade or
rating provided
How:
1.Specify length of response required (sentence,
paragraph)
2.Extended rationale usually required
3.Not just about ‘critiquing’ (e.g. alternative idea/perspective)

39. 5. Facilitate dialogue around the peer
review process
Why:
1.Discussion will enrich the review process leading into
peer learning
2.Furthers reflection & knowledge elaboration: questions,
extended conversations, re-processing of ideas
How:
1.Pairs or groups – to produce the assignment, the reviews,
the criteria or the responses to reviews
2.Students formulate questions for the reviewer
3.Discussion with teacher in class about the task and/or
reviews

40. 6. Integrate self-review activities into peer
review designs
Why:
1.Facilitates transfer of skill to students’ own work
2.Strengthens development of evaluative judgement
3.Helps develop distance and objectivity
How:
1.Self-review after peer review on same assignment
2.Self-review then peer review (e.g. asking questions
for reviewers to answer about the work)
3.Responding to the reviewers’ comments/questions –
revisit own work

41. 7. Encourage critical evaluations of
received reviews
Why:
1.Simulates professional practice – develops evaluation
and critical skills
2.Ensures attention to feedback, deeper processing and
further knowledge building by students
How:
1.Have students respond to reviews (e.g. develop an
action plan for improvement, identify inconsistencies
in received reviews)
2.Encourage peer discussion of received reviews

42. 8. Provide inputs that help students calibrate
their judgements
Why:
1.Recognises and utilises teacher expertise
2.Provides high level feedback on self-regulatory abilities
of students (Hattie and Timperley, 2007)
3.Helps students develop their own evaluative expertise
How:
1.Teachers provide feedback on students review outputs
2.Teachers select and highlight examples of quality
reviews produced by students
3.Scaffold students’ reviewing activities by providing them
with a menu of questions teacher would ask about the
work.

44. Example 2: Biochemistry labs
Peer Project case study
Biochemistry: third-year class with 30 science
students (BM310)
Professor Peter Halling, University of Strathclyde
[email protected]
Dr Sue Barnes, Learning Technology Adviser,
University of Strathclyde
[email protected]

45. Aims of Biochemistry pilot
To ensure that students actively engage with, process
and use teacher feedback
To develop students own skills in making evaluative
judgements and in producing feedback
Study possible because teacher had been producing
feedback for many years on labs – saw this feedback
data as a potential reusable resource

46. BM310: Biochemistry labs
Assessment and feedback – a summary
Wrote three lab eports
Got feedback and mark from academic on one (FAT)
Got feedback from fellow students on other two (EK
and AKE)
Gave feedback on two reports written by fellow
students
Was given a mark for the quality of feedback on one
of these two reports

47. Scaffolded peer review task
Three conditions of support for peer review task:
Menu of Feedback Comments (MFC)
e.g. ‘You are confusing the meaning of retention times and
areas – retention times tell you nothing about whether a
reaction is proceeding. It’s only the areas that tell you whether
the concentrations are still changing’
Menu of Feedback Questions (MFQ)
e.g. ‘Have they understood the link with increased pH of the
borate promoting full ionization of nitrophenol to the yellow
nitrophenolate anion?’
No menu (NM) – must produce comments un-aided.

48. Findings
Analysis of students’ feedback on work of peers
Initial results show that the MFQ (feedback questions) led
to better quality feedback comments than the MFC
(feedback comments)
No Menu condition not significantly different from MFQ
or MFC conditions.
However, further research needed as effects might be
due to difficulty of questions and have not tested
pairings of students for reviews.

49. Findings
Survey responses: students’ reports
All reported that reviewing engaged them in comparing
peer’s work with own work and with the menu template
(MFQ or MFC). Most made a physical comparison.
Reported that MFQ made them think more than MFC and
were more helpful for their understanding
Were less happy about the NM condition

50. Findings: MFC versus MFQ
The teacher’s feedback comments help me understand the basic
point to the assignments, however the questions allowed me to
understand the assignment better as I had to research the answers
myself and this allowed me to gain a better understanding of the
topic
I prefer the questions provided from the AK equilibrium because it
made me think more about the right answers and it allowed me to
write my own comments using the questions as guidance. Having
feedback comments from the Enzyme kinetics made it too easy to
just copy and paste previous feedback comments in and I did not
understand what some of the feedback comments referred to.

51. Findings survey: No Menu (NM) condition
NM condition had mixed results as some students claimed
that without any external input difficult to go beyond the
ideas they had for their own work.
Without teacher support we only know what we answered – our
perspective
Without the teacher feedback we lose an important area of the
question we may not have known was even a requirement to answer
Highlights value of having an external reference point
when providing feedback

52. Interpretation
Students engage with teacher feedback while also
developing their evaluative skills to question others’
work.
Teacher’s feedback on the peer review helps develop
students’ own feedback skills.
Implementation in a science discipline where feedback
might differ - about accuracy as well as quality of
judgement
Addresses issue that use of teacher feedback leads to
‘scripted’ responses (Boud, 2012).
Enables feedback to become a reusable resource.

53. Cognitive benefits of feedback construction
1.High-level cognitive activity: students cannot easily be
passive
2.Students actively exercise assessment criteria from
many perspectives
3.Writing commentaries causes students to evaluate and
rehearse their own knowledge
4.Evaluate different approaches to same work and learn
that quality can be produced in different ways
5.Shifts responsibility to student – puts them in role of
assessor exercising critical judgement

54. Cognitive benefits of feedback construction
6.Learn to evaluate their own work – as exactly the same
skills are involved
Develops capacity to make evaluative judgements - a
fundamental requirement for life beyond university.
Also, this capacity underpins all graduate attribute
development (Nicol, 2010)
Nicol, D (2011) Developing students’ ability to construct feedback,
Published by QAA for Higher Education, UK
http://www.enhancementthemes.ac.uk/resources/publications/graduates-for-the-21st-century

55. Enhancing evaluative judgement
through peer review
Maximise the number of reviews within practical limits
Integrate peer and self reviews – would enhance
transfer
Embed opportunities for dialogue in reviewing process
Broaden the scope of the review criteria –beyond
‘critiquing’
Make reviewing a regular activity
Change focus of teacher feedback to commenting on
quality of peer reviews.