cogntive structure MEQs.pdf medical assessment

Modified Essay Questions (MEQs) are assessment items that present a clinical or case-based scenario followed by a series of short, sequential questions, aiming to assess higher-order cognitive skills like problem-solving, reasoning, and clinical decision-making, especially in fields like medical education.

1. Medical Education, 1982, 16, 326-331
The cognitive structure of the modified essay question
W. G. IRWIN AND J. H. BAMBER
Department of General Practice, The Queen’s University of Belfast, Northern Ireland
summary
This paper analyses and discusses the extent to
which the modified essay question (MEQ) in the
Final MB Part I1 examination of the Queen’s
University, Belfast, assesses the three levels of cogni-
tive ability described by Buckwalter
et al. (198 1). The
1978 and 1980 Final MB MEQs are analysed and
compared question by question using Bloom’s taxo-
nomy and Buckwalter’s cognitive levels. Scores
obtained for each question were factor analysed to
determine any underlying relationships between the
questions. The findings (Table 1) indicate an uneven
distribution of cognitive questions (Buckwalter’s
Levels
I, I1 and 111) in each MEQ and between the
1978 and 1980 MEQ papers. This reflects partly
variation in the type of problems presented, and
partly problems posed by having to mark large
numbers of paper by hand against an objective
marking schedule. The statistical analysis in Table
2
shows a consistent and strong correlation between
scores in the MEQ Final MB papers and the Final
Clinical examination. The factorial analysis shows
that comprehension and evaluation questions do not,
as perhaps expected, dominate the loading of any
factor. The 1980 MEQ paper emerges as a testing of
‘knowledge’ paper compared with the 1978 paper,
which tests much more analysis, synthesis and
evaluation of knowledge. The appropriate balance
between cognitive levels to be tested must be deter-
mined beforehand. The study shows however the
potential of the MEQ to measure the entire span
of
Bloom and Buckwalter’s levels.
Correspondence: Professor W. G. Irwin, Department of General
Practice, The Queen’s University of Belfast, Dunluce Health Centre,
1 Dunluce Avenue, Belfast BT9
7HR, Northern Ireland.
Key words: EDUCATIONAL MEASUREMENT/*methOdS;
*COGNITION; *EDUCATION, MEDICAL,
UNDERGRADU-
ATE;
PROBLEM SOLVING; NORTHERN IRELAND
Introduction
Buckwalter et al. (1981) state that while the optimum
practice of medicine requires the complex cognitive
abilities of problem-solving and interpretation of
data, as well as recall or recognition of isolated
information, yet systematic evaluation of students’
interpretation of information or problem-solving
may be difficult. Marshall (1977) supports this latter
point, arguing that the assessment of problem solving
ability is not only the most difficult but also the most
neglected area in the assessment of medical compe-
tence.
Currently two approaches which attempt to over-
come the difficulty of assessing problem-solving and
the interpretation of information in medical educa-
tion are the patient management problem (PMP)
(McGuire, 1974; Marshall, 1977) and the modified
essay question (MEQ) (Royal College of General
Practitioners, 1971 and 1972). Both the PMP and the
MEQ present the candidate with a clinical situation
and examine how he proceeds to tackle this situation.
The philosophy of education of the Department of
General Practice, Queen’s University, Belfast, in-
cludes overall learning aims (Irwin, Bamber
&
Henneman, 1976) which differentiate the diagnostic
and management skills to be acquired by students in
general practice in the clinical years. The method of
assessment used in continuous assessment and in the
Final MB Part I1 Examination to test these skills is an
MEQ type of paper.
0308-01 10/82/1100-0326 $02.00 0 1982 Medical Education The present paper attempts to demonstrate the
326

2. Cognitive structure of essay question 327
extent to which the MEQ paper set by the Depart-
ment of General Practice in Queen’s and employed
in Part I1 of the Final MB examination (the final
hurdle) evaluates the three areas of cognitive ability,
described by Buckwalter
et al. (1981), namely,
problem-solving, interpretation of data, and recall
and recognition of isolated data.
Method
Bloom (1956) specifies in his cognitive taxonomy five
cognitive process levels arranged in a hierarchical
order of complexity; knowledge, comprehension,
analysis, synthesis and evaluation. This hierarchical
order implies that the objectives at one level require
the behaviours found in the preceding levels.
Buckwalter
et al. (198 1) have telescoped cognitive
processes in the test situation into three levels:
Level Z: questions which attempt to test recognition
and recall of isolated information (Bloom’s knowl-
edge level).
Level ZZ: questions which attempt to test comprehen-
sion and interpretation of data (Bloom’s comprehen-
sion level).
Level ZZZ: questions which attempt to test the
application of knowledge to the solution of a specific
problem (Bloom’s levels of analysis, synthesis and
evaluation).
The Part I1 Final MEQs for the years 1978 and
1980 were analysed using Bloom’s taxonomy and the
MEQs were compared by employing Buckwalter’s
three levels. Each MEQ comprised four unfolding
clinical situations with a range of eight to eighteen
questions asked on each. Each question was classified
according to Bloom’s taxonomy and the scores
obtained on each question were factor analysed using
the principal components method and varimax rota-
tion. Examples of MEQ questions classified accord-
ing to Bloom’s taxonomy are shown in the appendix.
The following are brief summaries of the types of
case histories which were presented in the 1978 and
1980 MEQ papers. These are varied from year to year
across the various themes relevant to general prac-
tice, namely, general medicine and surgery, obstetrics
and gynaecology, mental health, paediatrics, geriatric
medicine, social medicine and therapeutics.
MEQ 1978
Case history I: A 26-year-old male with retrobulbar
neuritis and multiple sclerosis.
Case history 2: An 19-year-old unmarried student
with an unwanted pregnancy.
Case history 3: A 42-year-old married woman with
menorrhagia, fibroids and thrombocytopenia. She
undergoes
a hysterectomy.
Case history 4: An 18-year-old boy develops glandu-
lar fever.
MEQ 1980
Case history I: A married woman, 61 years of age,
has depression and later develops Parkinsonism.
Case history 2: A 41-year-old married man has
ischaemic heart disease and considerable anxiety. He
eventually has by-pass cardiac surgery and requires
rehabilitation.
Case history 3: A wife arranges an appointment for
the doctor to see her husband who has chronic
alcoholism, liver cirrhosis and develops delirium
tremens.
Case history 4: A married woman, 27 years of age,
has mild jaundice and anaemia. Investigations sug-
gest chronic hepatitis and haemolytic anaemia. She is
told by a doctor never to have children. She seeks
contraceptive advice and the doctor counsels about
adoption.
Results
1. Analysis of questions by taxonomy
Table 1 shows the percentage breakdown of MEQ
questions
in each of the four clinical situations. The
classification used in the breakdown relates Bloom’s
cognitive taxonomy to that of Buckwalter
et al. From
the table it can be seen that there is both variation
between clinical situations within MEQs and varia-
tion between MEQs. When the two MEQs were
compared according to Buckwalter’s three levels,
using the chi-square
hz) test, they were found to be
significantly different
(2 = 8.54, d.f. = 2, Pt0.02).
2. Correlation of MEQ total scores with ‘A’ level entry
qualification and other
MB examinations
Though moderately to fairly low, all coefficients are
significant at the
5% level. A Spearman rank order
correlation was calculated between the two rank

3. 328 W. G. Irwin and J. H. Bamber
TABLE I. Analysis of questions by taxonomy.
~~ ~~ ~~~ ~~ ~ ~~ ~~
MEQ Clinical Clinical Clinical Clinical
year Classification situation
I situation I1 situation I11 situation IV Total
1978 Knowledge (Level I) (%) 10 12.5 29.4 50 27.7
43
Comprehension (Level (11) (%) 10 12.5 17.6 8.3
(Level 111) (%) 80 75 53 41.7
50 53.3 42.9 61 52.4
Analysis
Evaluation
Total number of questions
(n) 10
Comprehension (Level 11) (%) 6.25 0.0 7.1 00 3.2
Analysis 1 15.9)
Synthesis (Level 111) (%) 43.75 46.7 50 38.9 28.6 44.5
Total number of questions (n) 16 15 14 18 63
8 17 12 41
1980
Knowledge (Level I) (a)
Evaluation 1 0.0)
TABLE 2. Product moment correlation coefficients between MEQ total scores and A level and MB examinations
Examination
1978 MEQ* Rank order of coefficients* 1980 MEQt Rank order of coefficients?
‘A’ Level
Physiology
Biochemistry
Anatomy
Behavioural science
Pathology
Therapeutics
Microbiology
Community medicine
Essay (Final Part
11)
MCQ (Final Part 11)
Clinical examination (Final Part 11)
0.260
0.460
0.412
0,397
0.301
0.354
0,369
0.4
15
0.280
0.227
0.430
0.418
11
I
5
6
9
8
7
4
10
12
2
3
0.209
0.345
0.238
0.287
0.368
0.349
0,388
0,435
0.286 0.503
0.360
0.4
16
12
8
I1
9
5
7
4
2
10
I
6
3
*n = 104.
tn= 106.
orders of coefficients. It produced a value of 0.084
indicating no relationship between the two ranks.
3. Factorial structure of 1978 and 1980 MEQs
Each MEQ was scored by awarding marks to the
answers given to each question in each of the four
clinical situations. As has been stated
in Section 1
above, each question was classified according to the
schemes proposed by Bloom and Buckwalter.
To
advance further the analysis of each MEQ, the scores
awarded to each question were factor analysed. This
statistical procedure was employed to determine
whether any underlying patterns of relationships
existed between the questions in each MEQ, such
that the data could be ‘rearranged’ or ‘reduced’ to
smaller sets
of factors (components) which could be
taken as source variables accounting for the observed
inter-relations
in the data. The method of factor
analysis was that
of varimax rotation, an orthogonal
method which seeks to achieve simpler and more
meaningful factor patterns. Applying this technique
to the scores achieved on each question in each MEQ,
ten coherent factors emerged from the data for each
MEQ. All of the questions in the 1978 MEQ were
represented in the ten factors derived from it, while
92% of questions were represented in the ten factors
derived from the 1980
MEQ.
1978 MEQ
The ten factors which were obtained can be classified
into two groups. Group
1 comprises four factors
dominated by questions classified as belonging to a
particular type of cognitive process according to
Bloom’s taxonomy.
Factor Z, an ‘analysis’ factor with 15% of the
questions classified under the cognitive process
of
‘analysis’.
Factors ZZ, ZZZ and ZV had respectively 83.3, 75 and

4. Cognitive structure of essay question 329
66.7% of their questions classified under the cognitive
process of ‘synthesis’.
Thus, out of the ten factors derived from 1978
MEQ, four factors emerged as being heavily
weighted with higher order cognitive process ques-
tions.
Group
2 comprises the remaining six factors. Each
of these factors is characterized by its embrace of
questions revolving around a particular clinical
theme. The questions within each factor were classi-
fied according to Buckwalter’s levels.
Factor V contained questions concerned with the
examination of retrobulbar-neuritis in the eye and its
outcome. Twenty-five per cent of these questions
belonged to Buckwalter’s Level
I and 75% to Level
111.
Factor VZ had questions dealing with the outcome of
a haematological report which showed anisocytosis
and polychromasia in the red cells. Here,
44.4% of the
questions were at Level
I and 55.6% at Level 111.
Factor VZI grouped questions associated with the
clinical examination of a women of
44 years com-
plaining of tiredness and heavy periods, of which
44.4% were at Level I and 55.6% at Level 111.
Factor VZZZ centred on questions dealing with the
management of the gynaecological problem of me-
norrhagia and the questions were evenly divided
between Levels
I, I1 and 111.
Factor ZX had questions centred on aspects of uterine
pathology following hysterectomy. Here the ques-
tions were evenly divided between Levels
I and 111.
Factor X grouped questions dealing with the unmar-
ried mother situation. Twenty-five per cent of the
questions belonged to Level
I and 75% to Level 111.
These factors, whose question composition stresses
some of the clinical themes contained within the
MEQ, vary with respect to the cognitive complexity
which their questions demand. The number of Level
I questions lies between 25 and 50% while that for
Level
I11 questions lies between 33.3 and 75%.
The ten rotated factors account for 56.3% of the
interrelations within the data.
1980 MEQ
As with the ten factors derived from the previous
MEQ, the ten derived from the
1980 MEQ were
classified into a group of
six factors each emphasizing
a particular cognitive process and a group comprising
the remaining four factors with questions centring on
clinical themes.
Group
1 factors include Factors I to IV each
weighted with questions at the ‘knowledge’ level of
cognitive process,
in the following respective percent-
ages
- 90.9, 60, 75 and 75%. Factor V contained 60%
of questions classified by cognitive process as ‘analy-
sis’, while Factor
VI contained 60% of ‘synthesis’
questions. Compared with 1978 MEQ, the present
MEQ is much more of a ‘knowledge’ centred MEQ in
relation to its factor structure.
Group
2 contains four factors with questions
centred on clinical themes. As before, these questions
have been classified according to Buckwalter’s levels.
Factor VII has questions dealing with the clinical
approach to liver disease and alcoholism. Sixty per
cent belong to cognitive Level
I, 10% to Level I1 and
30% to Level
111.
Factor VZZZ questions relate to treating a woman of
6
1 years complaining of constipation and weight loss,
of which 16.7% are Level
I questions and 83.3% Level
I11 questions.
Factor ZX questions deal with chronic hepatitis and
haemolytic anaemia and the personal consequences
for a 27-year-old female. These questions are evenly
divided between Levels
I and 111.
Factor X contains questions on the diagnosis and
management of Parkinsonism and Angina. Here,
44.4% of the questions are at Level I and 55.6% are at
Level
111.
The cognitive complexity of the above four factors
varies between 16.7 and 60% for Level
I and 30 and
83.3% for Level
111. This range is comparable to that
obtained for the similar group in the 1978 MEQ. The
ten rotated factors derived from 1980 MEQ account
for
50% of the interrelations within the data.
Discussion
The object of this investigation has been to determine
the extent to which MEQs used in the assessment of
final-year medical students evaluate the three major
areas of cognitive ability associated with good
medical practice. From the analyses which have been
described, it is clear that neither MEQ has an even
distribution of questions between the three cognitive
areas. More particularly, the questions in both MEQs
are lower on Buckwalter’s Level
I1 and higher on
Level
I11 than an expected frequency of 1/3 : 1/3 : 1/3.
When the discrepancies between observed and ex-
pected frequencies were analysed by the chi-square
test the following values were obtained:

5. 330 W. G. Irwin and J. H. Bamber
MEQ 1978 x’= 15.994 Pt0.001,
MEQ 1980 x’=26.381 PCO.001.
Nor, as Table
1 shows, are the MEQs similar in their
distribution of cognitive levels.
These differences require some explanation. Dif-
ferent types of unfolding clinical situations from
general practice are used in
MEQ construction. They
reflect the pattern of morbidity in the community.
They each offer a choice and variation in the
questions which can be posed. If, for example, one
attempts to test knowledge of the diagnosis and
management of acute short term illness, e.g. an 18-
year-old boy with glandular fever (Clinical situation
IV, 1978 MEQ, Table I), then a high percentage of
questions are not judged to pose an intellectual
challenge at Level
11, whereas the distribution
percentages are much higher in Levels
I and 111. Thus
half the questions are designed to test specific recall
of information related to glandular fever. By contrast
in Case history I of the same
MEQ (Table I), the
diagnosis and management of multiple sclerosis in a
young man, only 10% of questions concern recall of
factual information and
80% pose a challenge at the
Level
111 category. Thus the varying cognitive level
percentages within each
MEQ, and between different
MEQs can be largely explained by the type of
problem or morbidity chosen to be assessed.
Common clinical situations in general practice
constitute the core of the
MEQ paper. It tests factual
information of practical and/or vocational value.
A
further consideration which relates to marking large
numbers of
MEQ papers by hand against a set time
limit should be noted, because it affects the construc-
tion of the paper and the distribution of questions
over the three cognitive areas. The more one tends to
include specific factual questions with concise spe-
cific answers the easier it is beforehand to reach
agreement within
a peer group about a marking
schedule. The more one departs into constructing
interpretation of data and problem-solving questions,
and questions which test analysis, synthesis and
evaluation of knowledge, the ‘softer’ becomes the
marking schedule and the more difficult becomes the
process of marking
the paper objectively. A peer
group of markers has then to spend time discussing
and evaluating what they may identify as reasonable
alternatives to the answers provided beforehand. The
person who sets the questions is well aware of this
and his unconscious bias may therefore be to
construct more Level
I category questions. This,
added to what has been said, may help to explain the
gross differences in item percentages in Levels I,
I1
and 111 in Table 1, within each MEQ and between the
1978 and 1980 papers.
The factorial structure of both
MEQs shows a
division between ‘structure’ (i.e. factors dominated by
specific types of cognitive variable) and ‘content’ (i.e.
factors dominated by specific clinical themes). That
such ‘structural’ factors emerge provides confidence
in the discreteness of cognitive groups, i.e. knowl-
edge, analysis and synthesis. It is disappointing that
comprehension and evaluation questions do not
dominate the loading of any factor, because of their
weak representation in both
MEQs. The ‘content’
factors with clinical themes generally represent a
balance between Level I and Level
111 questions.
When this balance is upset the advantage seems to
fall in favour of Level I11 questions. The 1980
MEQ
is much more a ‘knowledge’-centred MEQ compared
with the 1978 paper. It is doubtful if the person who
set the papers was aware of this disparity, until the
analysis of the
MEQs question by question was
performed. The factor loadings and the analysis
of
questions by taxonomy (Table 1) show that the
choice of clinical situation or morbidity affects the
type of questions which can be posed.
The correlation coefficients obtained with other
examination variables (Table
2) for both MEQs
reveal three interesting points. First, despite the
variance in question distribution between cognitive
levels for each
MEQ, the range of coefficients is fairly
similar; the highest coefficient for each
MEQ being
0.46 and 0.503 respectively, and the lowest being
0.227 and 0209 respectively. Secondly, for both
MEQs the rank placings for Community medicine
and Clinical examination are identical, while those
for
‘A’ level and Pathology are adjacent. Thirdly, the
rank order correlation coefficient derived from the
two rankings of correlation Coefficients is virtually
0.0 due to a few wide discrepancies. These observa-
tions might suggest that, not withstanding variation
in the other examinations, variation in question
distribution between
MEQs operates selectively
rather than globally when compared with other
medical assessments.
Conclusions
As an educational tool for assessing the cognitive
processes essential to the optimum practise of medi-
cine, the
MEQ appears to have the potential of being

6. Cognitive structure of essay question 33 1
more than adequate in that it is capable of measuring
the entire span of Bloom and Buckwalter levels.
However, in order to utilize it at its full potential,
the constructors of MEQs should
(1) determine the balance between cognitive levels
which they regard as appropriate;
(2) construct items which reflect this appropriate
balance.
(1) and
(2) imply that the use of cognitive
taxonomies are essential if medical educators are to
develop assessment methods best equipped to mea-
sure the optimum practise of medicine.
APPENDIX 1
Examples of MEQ questions classified according to
Bloom’s taxonomy. (The year of the MEQ, the case
history number and the diagnosis/problem are
shown after each example.)
Knowledge
(1) What is the main risk in a patient with this
condition? (1978, Case history
3 - Thrombocyto-
penia)
(2) Name four routine investigations that you would
consider adviseable for the patient. (1980, Case
history 1
- Parkinsonism)
(3) The drug treatment of this condition falls into
three main categories. What are they? (1980, Case
history
1 - Parkinsonism)
Comprehension
(1) In what way might the social service department
be of help in this situation if Mrs Rowan became
severely disabled? (1980, Case history 1
- Parkinson-
ism)
(2) What are the three options open to Janet or to any
other young woman in this situation? (1978, Case
history
2 - unwanted pregnancy)
(3) What possible fears do you anticipate from your
patient? (1978, Case history
3 - Fibroids)
APPENDIX
2
Analysis
(1) What questions would you ask Mrs Rowan to
elucidate the problem? (1980, Case history
1 - Par-
kinsonism)
(2) Could the illness with which Mrs Rowan pre-
sented
6 months previously be associated with this
condition? (1980, Case history
1 - Depression)
(3) What should he look for in his examination of the
baby? (1980, Case history
4 - Adoption)
Synthesis
(1) What explanation would you give to John at this
stage? (1978, Case history 1
- Retrobulbar Neuritis)
(2) What treatment would you initiate? (1978, Case
history 1
- Multiple sclerosis)
(3) Outline the management of a patient severely
affected by this condition. (1978, Case history
1
- Multiple sclerosis)
Evaluation
(1) After discussion with her parents and the social
worker Janet decides to have the baby placed for
adoption. What are the advantages and disadvan-
tages of adoption for the child? (1978, Case history
2 - Unwanted pregnancy)
(2) Discuss the advantages and disadvantages of
marriage as a possible solution to an ‘unwanted
pregnancy’. (1978, Case history 2
- Unwanted preg-
nancy)
References
BLOOM, B.S. (ed.) (1956) A Taxonomy of Educational Objectives,
Handbook
I. Longman, London.
BUCKWALTER, J.A., SCHUMACHER, R., ALBRIGHT, J.P. & COOPER,
R.R. (1981) Use of an educational taxonomy for evaluation of
cognitive performance.
Journal of Medical Education, 56, 115.
IRWIN, W.G., BAMBER, J.H. & HENNEMAN, J. (1976) Constructing a
new wurse for undergraduate teaching of general practice.
Medical Education, 16, 302.
MARSHALL, J. (1977) Assessment of problem-solving ability. Medical
Education,
11, 329.
MCGUIRE, C. (1974) The role of evaluation and examinations in
Colleges
of General Practice. Journal of the Royal College of
General Practitioners, 24, 166.
ROYAL COLLEGE OF GENERAL PRACTITIONERS (1971) The modified
essay question.
Journal of the Royal College of General Practition-
ers, 21, 373.
ROYAL COLLEGE OF GENERAL PRACTITIONERS (1972) Explanatory
notes for the
MRCGP Examination. Journal of the Royal College
of General Practitioners, 22, 598.
Received 7 August 1981; accepted for publication 26
February 1982.