meta analysis

1. A Workshop on the Basics ofA Workshop on the Basics of
Systematic Review & Systematic Review &
Meta-Analysis Meta-Analysis
Philip C. Abrami, Robert M. BernardPhilip C. Abrami, Robert M. Bernard
C. Anne Wade, Evgueni Borokhovski, Rana Tamim, Gretchen C. Anne Wade, Evgueni Borokhovski, Rana Tamim, Gretchen
Lowerison & Mike SurkesLowerison & Mike Surkes
Centre for the Study of Learning and Performance Centre for the Study of Learning and Performance
and CanKnowand CanKnow
Concordia UniversityConcordia University

2. 02/25/11 2
What is a Systematic Review?What is a Systematic Review?
•A review of a clearly formulated question that
uses systematic and explicit methods to
identify, select and critically appraise relevant
research, and to collect and analyze data from
the studies that are included in the review.
•Statistical methods (meta-analysis) may or may
not be used to analyze and summarize the
results of the included studies.
•Other examples: Narrative review, qualitative
review, vote count, meta-synthesis.

3. 02/25/11 3
What is Meta-Analysis?What is Meta-Analysis?
•Meta-Analysis is a set of quantitative
research synthesis techniques and
procedures
•Meta-Analysis uses effect size as a
metric for judging the magnitude of
standardized difference between a
treatment and control condition

4. 02/25/11

5. 02/25/11 5
Purpose: Purpose: Explaining Explaining
Variability in Effect SizeVariability in Effect Size
Effect SizesStudy Features
Shared Variability
Unique Variability Unique Variability
Prediction

6. 02/25/11 6
2.Determine the research question
3.Develop terms and definitions related to the question
4.Develop a search strategy for identification of relevant
studies
5.Establish criteria for inclusion and exclusion of studies
6.Select studies based on abstract review (agreement)
7.Select studies based on full-text review (agreement)
8.Extract effect sizes (agreement)
9.Develop codebook of study features
10.Code studies (agreement)
11.Conduct statistical analysis and interpretation
10 Steps in Planning and 10 Steps in Planning and
Conducting a Systematic Conducting a Systematic
Review/Meta-AnalysisReview/Meta-Analysis

7. 02/25/11

8. 02/25/11 8
1. Determine the research question
The “big question” that guides the research. It
usually involves asking about the difference between
two conditions (i.e., usually treatment and control) or
the relationship between two measures.
10 Steps in a Meta-Analysis10 Steps in a Meta-Analysis

9. 02/25/11 9
Questions the Researcher Questions the Researcher
Should AskShould Ask
•Does the question have theoretical or practical
relevance (i.e., aids in practice and/or policy
making decisions)?
•Is the literature of a type that can answer the
question?
•Is there a sufficient quantitative research
literature?
•Do the studies lend themselves to meta-analysis?
•Is the literature too large given the resources
available?

10. 02/25/11 10
Example: Example:
Critical ThinkingCritical Thinking
Research Question: What instructional
interventions, to what extent, and under what
particular circumstances, impact on the
development and effective use of learner’s
critical thinking skills and dispositions?

11. 02/25/11 11
2. Develop terms and definitions related to
the question
This helps refine the research question and inform
the search strategies.
10 Steps in a Meta-Analysis10 Steps in a Meta-Analysis

12. 02/25/11 12
3. Develop a search strategy for the
identification of relevant studies
This involves the planning/implementation of search
and retrieval for primary studies
(e.g., electronic databases, branching).
10 Steps in a Meta-Analysis10 Steps in a Meta-Analysis

13. 02/25/11 13
Information Retrieval: Information Retrieval:
A Continuous ProcessA Continuous Process
Preliminary Searches
 Supports beginning steps: Definition of key concepts &
research question
 Use of standard reference tools and broad searches for
review articles and key primary studies
Main Searches
 Identification of primary studies through searches of
online databases, printed indices, Internet, branching, hand-
searches
 Most difficult given a number of challenges
Final Searches
 Occurs towards the end of the Review Process
 Refine search terms and update original searches

14. 02/25/11 14
Preliminary SearchesPreliminary Searches
Reference Sources:
Purpose: To obtain definitions for the terms; creativity, critical thinking,
decision making, divergent thinking, intelligence; problem solving, reasoning,
thinking.
Sources:
Bailin, S. (1998). Critical Thinking: Philosophical Issues. [CD-ROM] Education:
The Complete Encyclopedia. Elsevier Science, Ltd.
Barrow, R., & Milburn, G. (1990). A critical dictionary of educational concepts:
An appraisal of selected ideas and issues in educational theory and practice
(2
nd
ed.). Hertfordshire, UK: Harvester Wheatsheaf
Colman (2001). Dictionary of Psychology (complete reference to be obtained)
Corsini, R. J. (1999). The dictionary of psychology. Philadelphia, PA:
Brunner/Mazel
Dejnoka, E. L., & Kapel, D. E. (1991). American educator’s encyclopedia.
Westport, CT: Greenwood Press.
…… (see
handout)

15. 02/25/11 15
Main Searches: DecisionsMain Searches: Decisions
Selection of Primary Information Retrieval Tools
 Scope of search: Which fields should be searched
(including all related fields)?
 Availability of indexing tools: Which tools do we have
access to at our institution? Are there others who can
perform searches for us?
 Format of indexing tools: What format are they in (e.g.
online, print, web-based)?
 Date: How far back does the indexing go for each tool?
 Language: What is the language of the material that is
indexed? How can we locate non-English material?
 Unpublished work: How can we access dissertations,
reports, & other grey literature?

16. 02/25/11 16
Examples of DatabasesExamples of Databases
Education: ERIC, British Education Index, Australian
Education Index, Chinese ERIC, CBCA Education,
Education index, Education: A SAGE Full-text Collection
Psychology: PsycINFO, PubMed (Medline), Psychology: A
SAGE Full-Text Collection
Sociology: Sociological Abstracts, Contemporary
Women’s Issues. Sociology: A SAGE Full-text Collection
Multidisciplinary: EBSCO Academic Search Premier,
ProQuest Dissertations and Theses Fulltext, FRANCIS,
Social Sciences Index, SCOPUS, Web of Science

17. 02/25/11 17
Example: Critical ThinkingExample: Critical Thinking
To date, the following databases have been searched:
•AACE Digital Library (now known as EdITLib)
•ABI/Inform Business
•EBSCO Academic Search Premier
•ERIC
•EconLit
•PAIS International
•ProQuest Dissertations and Theses Fulltext
•PsycINFO
•Social Science Index
•Sociological Abstracts

18. 02/25/11 18
Main Searches: More DecisionsMain Searches: More Decisions
Preparation of Search Strategies
 What are the key concepts to be searched?
 How are these represented in each discipline?
 What are their related terms?
 How are these key concepts represented in the
controlled vocabulary within each database to be
searched? (See handout)
 Note: these decisions need to be made for each indexing tool used.

19. 02/25/11 19
Main Searches: Main Searches:
Yet More DecisionsYet More Decisions
Construction of the Search Statements
What terms should be searched as descriptors or as
“free text”?
What Boolean operators should be used?
Where should truncation characters be used? (e.g.
parent* will retrieve parent, parents, parental)
What limiting features are available to narrow
results? (e.g. use of Publication Type codes)?
What time period should be searched?

20. 02/25/11 20
Example: ERICExample: ERIC
Combining Keywords/Descriptors using Boolean
operators:
Searches and records below from: The ERIC Database
#5 #3 and #4 (1520 records)
#4 DTC = 142 or DTC = 143 or control group* (322893 records)
#3 #1 or #2 (7718 records)
#2 critical thinking in DE,ID (7562 records)
#1 thinking skills in DE and critical thinking (1269 records)

21. 02/25/11 21
Example from our Codebook:
ERIC (Date: September 21, 2003; AW)
Purpose: To retrieve the first set of abstracts to be reviewed by
team according to the current inclusive/exclusion criteria.
Result: Hit rate of 514/1520
Source code: ERIC1
Searches and records below from: The ERIC Database
(1966-2003, June)
#5 #3 and #4 (1520 records)
#4 DTC = 142 or DTC = 143 or control group* (322893
records)
#3 #1 or #2 (7718 records)
#2 critical thinking in DE,ID (7562 records)
#1 thinking skills in DE and critical thinking (1269 records)
Documenting Your SearchesDocumenting Your Searches

22. 02/25/11 22
Next Steps
Repeat these steps for each
database to be searched.
(see handout)

23. 02/25/11 23
Secondary Retrieval Strategies
 Locating the grey (unpublished) literature:
- Using the web, & Dissertations Abstracts
 Branching:
- Scanning the reference section of review articles
 Hand searches:
- Scanning the Table of Contents of key journals and
conference proceedings
 Personal contacts:
- Contacting key researchers in the field
Main Searches: Main Searches:
Yet Still More DecisionsYet Still More Decisions

24. 02/25/11 24
Information Retrieval: Information Retrieval:
Wrap UpWrap Up
“Shoestring-budget information retrieval is likely to introduce
bias, and should be avoided.” (IR Policy Brief, 2004)
Importance of information retrieval process
 Not a “one-shot”deal
 Requires expertise in the planning and implementation of
searches
 Library personnel are important members of the team
Use of bibliographic management software
 Reference Manager, EndNotes, RefWorks
Ability to replicate review
 Documentation of entire process, including search strategies
used for each database, decisions taken, etc.

25. 02/25/11 25
10 Steps in a Meta-Analysis10 Steps in a Meta-Analysis
4. Establish criteria for inclusion
and exclusion of studies
These are the criteria that guide the search for
literature and ultimately determine what studies
are in and out of the review.

26. 02/25/11 26
Inclusion/Exclusion: Questions
•What characteristics of studies will be used to
determine whether a particular effort was
relevant to the research question?
•What characteristics of studies will lead to
inclusion? exclusion?
•Will relevance decisions be based on a reading of
report titles? abstracts? full reports?
•Who will make the relevance decisions?
•How will the reliability of relevance decisions be
assessed?

27. 02/25/11 27
10 Steps in a Meta-Analysis10 Steps in a Meta-Analysis
5. Select studies based on
abstract review
This is the initial decision as to what
studies will be retrieved as full-text documents.

28. 02/25/11 28
10 Steps in a Meta-Analysis10 Steps in a Meta-Analysis
6. Select studies based on
full-text review
This is the second decision as to what studies
will be included in the review.

29. 02/25/11 29
10 Steps in a Meta-Analysis10 Steps in a Meta-Analysis
7. Extract effect sizes
Effect sizes extraction involves converting
descriptive or other statistical information contained
in studies into a standard metric by which studies
can be compared.

30. 02/25/11 30
What is an Effect size?What is an Effect size?
•A descriptive metric that characterizes
the standardized difference (in SD units)
between the mean of a control group and
the mean of a treatment group (educational
intervention)
•Can also be calculated from correlational
data derived from pre-experimental
designs or from repeated measures designs

31. 02/25/11 31
Characteristics of Characteristics of
Effect SizesEffect Sizes
•Can be positive or negative
•Interpreted as a z-score, in SD unitsSD units, although
individual effect sizes are not part of a z-score
distribution
•Can be aggregated with other effect sizes and
subjected to other statistical procedures such as
ANOVA and multiple regression
•Magnitude interpretation: ≤ 0.20 is a small effect
size, 0.50 is a moderate effect size and ≥ 0.80 is a
large effect size (Cohen, 1992)

32. 02/25/11 32
Effect Size ExtractionEffect Size Extraction
•Effect size extraction is the process of
identifying relevant statistical data in a study and
calculating an effect size based on those data
•All effect sizes should be extracted by two
coders, working independently
•Coders’ results should be compared and a measure
of inter-coder agreement calculated and recorded
•In cases of disagreement, coders should resolve
the discrepancy in collaboration

33. 02/25/11

34. 02/25/11

35. 02/25/11 35
Example of Example of ESES Extraction with Extraction with
Descriptive StatisticsDescriptive Statistics
Study reports: Treatment mean = 42.8 Control Mean = 32.5
Treatment SD = 8.6 Control SD = 7.4
n = 26 n = 31
SD
pooled=((26-1)8.6
2
)+(31-1)7.4
2
))/(57-2)
SD
pooled=(1849+1642.8)/55=3491.8/55=63.49=7.97
d=
42.8-32.5
7.97
=
10.3
7.97
=1.29
g=d1-
3
(4(N
E+N
C))-9
￦
￨
￨
￨
￨
￨
=1.291-
3
4(26+31)-9
￦
￨
￨
￨
￨
￨
=1.291-
3
219
￦
￨
￨
￨
￨
￨=1.27
Procedure: Calculate SD
pooled
Calculate d and g

36. 02/25/11 36
Extracting Effect Sizes in the Extracting Effect Sizes in the
Absence of Descriptive StatisticsAbsence of Descriptive Statistics
•Inferential Statistics (t-test, ANOVA, ANCOVA,
etc.) when the exact statistics are provided
•Levels of significance, such as p < .05, when the
exact statistics are not given (t can be set at the
conservative t = 1.96) (Glass, McGaw & Smith, 1981; Hedges,
Shymansky & Woodworth, 1989)
•Studies not reporting sample sizes for control and
experimental groups should be considered for
exclusion

37. 02/25/11 37
Examples of Alternative Examples of Alternative
Methods of Methods of ESES Extraction Extraction
d=
2t
df
=
2(2.56)
63
=
5.12
7.94
=.6448
• Study Reports: t (63) = 2.56, p < .05
• Study Reports: F (1, 63) = 2.56, p < .05
Convert F to t and apply the above equation:
t=F=1.6;df=63
d=
2t
df
=
2(1.6)
7.94
=
3.2
7.94
=.4030

38. 02/25/11 38
Zero Effect SizeZero Effect Size
ES = 0.00
Control
Condition
Treatment
Condition
Overlapping
Distributions

39. 02/25/11 39
Moderate Effect SizeModerate Effect Size
Control
Condition
Treatment
Condition
ES = 0.40

40. 02/25/11 40
Large Effect SizeLarge Effect Size
Control
Condition
Treatment
Condition
ES = 0.85

41. 02/25/11 41
Mean and VariabilityMean and Variability
Variability
ES+
Note: Results from Bernard, Abrami, Lou, et al. (2004) RER

42. 02/25/11 42
10 Steps in a Meta-Analysis10 Steps in a Meta-Analysis
8. Develop a codebook
Study feature coding involves describing the relevant
characteristics for each study (e.g., research
methodology, publication source).The codebook
details the study feature categories and their levels.

43. 02/25/11 43
Examining Study FeaturesExamining Study Features
•Purpose: to attempt to explain variability in
effect size
•Any nominal, ordinal or interval coded study
feature can be investigated
•In addition to mean effect size, variability
should be investigated
•Study features with small ks may be
unstable

44. 02/25/11 44
Examples of Study FeaturesExamples of Study Features
•Research methodology
•Type and nature of measures
•Direction of the statistical test
•Publication data
•Relevant aspects of the treatment
•Relevant aspects of the control condition

45. 02/25/11 45
10 Steps in a Meta-Analysis10 Steps in a Meta-Analysis
9. Code studies for study features
Coding study features is perhaps the most time-
consuming and onerous aspect of conducting a
meta-analysis.
However, it is arguably the most important step
because it provides the possibility for explaining
variability in effect sizes.

46. 02/25/11 46
10 Steps in a Meta-Analysis10 Steps in a Meta-Analysis
10: Analysis and interpretation
Analysis involves invoking a range of standard
statistical tests to examine average effect sizes,
variability and the relationship between study
features and effect size. Interpretation is drawing
conclusion from these analyses.

47. 02/25/11 47
Questions: Questions:
Statistical AnalysisStatistical Analysis
•What techniques will be used to combine results
of separate tests?
•What techniques will be used to assess and then
analyze the variability in findings across studies?
•What sensitivity analyses (i.e., tests of the impact
of such decisions on the results of the review) will
be carried out and how?
•What statistical procedures will be used to test
relationships between study features and effect
sizes (e.g., meta regression)

48. 02/25/11 48
Homogeneity vs. Homogeneity vs.
Heterogeneity of Effect Heterogeneity of Effect
SizeSize
•If homogeneity of effect size is
established, then the studies in the meta-
analysis can be thought of as sharing the
same effect size (i.e., the mean)
•If homogeneity of effect size is violated
(heterogeneity of effect size), then no
single effect size is representative of the
collection of studies (i.e., the “true” mean
effect size remains unknown)

49. 02/25/11

50. 02/25/11 50
Effect size and 95% confidence interval Test of null (2-Tail)
Number StudiesPoint estimateStandard errorVarianceLower limitUpper limitZ-valueP-value
168 0.34 0.01 0.00 0.31 0.36 23.28 0.00
Heterogeneity
Q-value df (Q) P-value
1816.71 167.00 0.00
Statistics in Comprehensive Statistics in Comprehensive
Meta-Analysis™ Meta-Analysis™
Comprehensive Meta-Analysis 2.0 is a trademark of BioStat®
Interpretation: Moderate ES for all outcomes (g+ = 0.34) in favor of
the intervention condition.
Homogeneity of ES is violated. Q-value is significant (i.e., there is too
much variability for g+ to represent a true average in the population).

51. 02/25/11 51
Examining the Study Examining the Study
Feature “Type of Research Feature “Type of Research
Design”Design”
g+ = +0.34
Overall
Effect
Pre-Post
Designs
Post-Only
Designs
Quasi-Exp.
Designs

52. 02/25/11 52
Groups
Group N of StudiesPoint estimateStandard errorLower limitUpper limitQ-value df (Q)P-value
one-group 27 0.16 0.04 0.09 0.24 181.30 26.00 0.00
post only 87 0.38 0.02 0.34 0.42 651.34 86.00 0.00
quasi-exp 54 0.35 0.02 0.31 0.40 957.45 53.00 0.00
Total within 1790.09 165.00 0.00
Total between 26.62 2.00 0.00
Overall 168 0.34 0.01 0.31 0.36 1816.71 167.00 0.00
Effect size and 95% confidence interval Heterogeneity
Tests of Levels of “Type of Tests of Levels of “Type of
Research Design”Research Design”
Interpretation: Small to Moderate ESs for all categories in favor of
the intervention condition.
Homogeneity of ES is violated. Q-value is significant for all categories
(i.e., type of research design does not explain enough variability to
reach homogeneity.

53. 02/25/11 53
Sensitivity AnalysisSensitivity Analysis
•Tests the robustness of the findings
•Asks the question: Will these results stand
up when potentially distorting or deceptive
elements, such as outliers, are removed?
•Particularly important to examine the
robustness of the effect sizes of study
features, as these are usually based on
smaller numbers of outcomes

54. 02/25/11 54
Selected ReferencesSelected References
Bernard, R. M., Abrami, P. C., Lou, Y. Borokhovski, E., Wade, A.,
Wozney, L., Wallet, P.A., Fiset, M., & Huang, B. (2004). How
Does Distance Education Compare to Classroom
Instruction? A Meta-Analysis of the Empirical Literature.
Review of Educational Research, 74(3), 379-439.
Glass, G. V., McGaw, B., & Smith, M. L. (1981). Meta-analysis in
social research. Beverly Hills, CA: Sage.
Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-
analysis. Orlando, FL: Academic Press.
Hedges, L. V., Shymansky, J. A., & Woodworth, G. (1989). A
practical guide to modern methods of meta-analysis. [ERIC
Document Reproduction Service No. ED 309 952].