Risk of Bias Assessment in Systematic Reviews

Enhance the credibility of your systematic reviews with our in-depth slides on "Risk of Bias Assessment." This presentation includes:

1) The significance of assessing risk of bias in systematic reviews
2) Key domains of bias (e.g., selection, performance, detection, attrition, reporting)
3) Tools and methodologies for assessing bias (e.g., Cochrane
4) Risk of Bias Tool, ROBINS-I)
Practical steps for conducting bias assessments
5) Interpreting and reporting bias assessment findings

Ideal for both beginners and seasoned researchers, these slides provide crucial knowledge to ensure the accuracy and reliability of your systematic reviews. Dive into the essential practices of risk of bias assessment and improve the quality of your evidence synthesis.

1. Systematic
Reviews: Risk of
Bias Assessment
Kwabena Kusi-Mensah
MBChB, FWACP (Psych), MSc.CAMH (Ib.)
Behavioural Sciences Department
School of Medical Sciences-KNUST
Kumasi, Ghana

2. Outline
Recap and
overview of the
Systematic
Review Process
Purpose and
principles of a
Quality
Assessment/
Risk of Bias
Different Tools
used in QA/RoB
Overview of
various kinds of
RoBtools
Take home
message
AuthorAid 2024

3. Learning
Outcomes
At the end of the lesson, participants should:
•Understand the reason for and place of a QA/RoB in
Systematic Review
•Know the Difference between the terms “Quality
Assessment” and “Risk of Bias assessment”
•Appreciate the approach to and principles underlying
a RoB
•Have a broad overview of various RoB assessment
tools
•Be familiar with at least 2 of these RoB tools
•Know how/where to find the appropriate RoB tool
AuthorAid 2024

4. Why this Presentation?
“Believe not every
spirit, but try the spirits
whether they are of
God.”-
I John 4:1
“In God we trust… all
others must bring data.”
W. Edwards Deming, American
Professor of Statistics
AuthorAid 2024

5. Recap: What is a “systematic review”?
“A systematic review attempts to identify, appraise and synthesise all the empirical
evidence that meets pre-specified eligibility criteria to answer a given research
question. Researchers conducting systematic reviews use explicit methods aimed at
minimising bias, in order to produce more reliable findings that can be used to inform
decision making” (Cochrane)
“Systematic reviews aim to find as much as possible of the research relevant to
theparticular research questions, and use explicit methods to identify what can
reliably be said on the basis of these studies. Methods should not only be explicit but
systematic with the aim of producing varied and reliable results”. (EPPI)
“A systematic review uses transparent procedures to find, evaluate and synthesize
the results of relevant research. Procedures are explicitly defined in advance, in
order to ensure that the exercise is transparent and can be replicated. This practice
is also designed to minimise bias” (Campbell)
Does not mean quick!
AuthorAid 2024

6. Hierarchy of
Evidence
•Systematic review of randomized trials/Meta-
analysis
•Single randomized controlled trial
•Systematic review of observational studies
addressing patient-important outcomes
•Single observational study addressing patient
important outcomes
•Physiological studies
•Unsystematic clinical observations; expert opinion
and
•clinical experience
AuthorAid 2024

7. Protocol: specific
objectives,
search strategy
etc.
Find eligible
papers, data
extraction
Critical appraisal
of methods and
results
Synthesis of
findings and
grading of
evidence
Write up of
results
(discussion) and
recommendation
s
The grand scheme of things: Systematic Review Process
QA / RoB
AuthorAid 2024

8. What are we seeking to do in a RoB?
•Can I trust these results?

9. What are we seeking to do in a RoB?
•Is the methodology sound?
•Are the reported results therefore trustworthy?
•How trustworthy are the results?

10. Difference Between ”quality assessment” and
“Risk of Bias assessment”
Risk Of Bias
Wether the result based on the
methodology quality of the study
are trustworthy
Specifically evaluates the potential
for systematic errors (biases) in
the study that could affect the
results
Quality Assessment
evaluates the overall quality and
reliability of the studies included
in the review. This process
involves looking at various aspects
of the study methods: study
design, conduct, and reporting to
determine the strength and
validity of the evidence provided.

11. Difference Between ”quality assessment” and
“Risk of Bias assessment”

12. Difference Between ”quality assessment” and
“Risk of Bias assessment”
Risk Of Bias
Key domains assessed for risk of bias
typically include:
•Selection bias: How participants were
selected and allocated to groups
•Performance bias: How interventions
were implemented and whether there
were deviations from the intended
intervention
•Detection bias: How outcomes were
measured and assessed
•Attrition bias: How incomplete data
were handled and reported
•Reporting bias: Whether all expected
outcomes were reported
Quality Assessment
Common criteria assessed include:
•Appropriateness of Study design (e.g.,
randomized controlled trial, cohort study)
•Sample size
•Methods of data collection
•Statistical analysis
•Consistency of findings
•Generalizability of results
•Adequacy of follow-up
•Reporting clarity and transparency

13. Difference Between ”quality assessment” and
“Risk of Bias assessment”
Quality Assessment
Outcome:
•Provides an overall judgement of
how well the study was conducted.
Risk Of Bias
Outcome:
•Identifies specific areas where bias
might have influenced the study
results.

14. Difference Between ”quality assessment” and
“Risk of Bias assessment”

15. Difference Between ”quality assessment” and
“Risk of Bias assessment”

17. Different RoB Tools and their Uses
Purpose/ Study Design Tool(s)Comment
RCTs ROB2 assesses RoB of randomized trials including Individually-
randomized parallel-group trial, Cluster-randomized parallel-
group trial, Individually randomized cross-over (or other
matched) trial
Other intervention studies (non-
RCT)
ROBIN-I assesses RoB in non-randomized intervention studies.
Patient Reported Outcome Measure
(PROMs) Validations
COSMINassessing studies that check the validity or adequacy of
measurement properties of a PROM (questionnaire) tool in
any particular setting
Predictive model development:
diagnosis/ Prognosis/monitoring
tool studies
PROBASTUseful for studies that build models that look at probability
that a condition or disease is already present (diagnostic
model) or will occur in the future (prognostic model) in
individuals e.g. polygenic risk score studies.
diagnostic test accuracy studiesQUADAS-
2
Useful for for diagnostic test accuracy studies

18. RoB done for
PROMs using
COSMIN

19. Beware “checklists” Vs “RoB Assessment Tools”
ONE LETS YOU SCORE THE
PAPER
THE OTHER TELLS YOU
WHAT TO LOOK FOR

20. Beware “checklists” Vs “RoB Assessment
Tools”
•Checklists ensure reporting
quality
•RoB actually assess the quality of
the study based on a set criteria

21. Checklists
https://www.equator-
network.org//reporting-guidelines/

22. Performing
QA/RoB of
RCTs
Flow of RCT
What can go wrong?

23. 1.Risk of Bias due to the
randomization process
•1.1. Was allocation sequence random?
•1.2. Was allocation concealed until all
patients enrolled and assigned?
•1.3. Do the baseline characteristics
suggest a problem with
randomization?

24. 2.Risk of bias due to intervention
•2.1. Were participants aware of the intervention they received
•2.2. Were caretakers and caregivers aware of the interventions
allocation?
•2.3. Were deviations from the interventions inherent to the study
design?
•2.4. Could deviations of the interventions have influenced the
outcome?
•2.5. Were deviations balanced between the arms?
•2.6. Was an appropriate analysis used to account for deviations?
(ITT)

25. Intention to Treat (ITT) Vs Per Protocol
Approach
Intention to treat: participants were analysed
in the same arm as they have been
randomized to, regardless of whether they
completed the treatment, adhered to the
protocol, or received a different intervention.
Per protocol: participants were
analysed according to which
intervention or comparison they
actually received

26. ITT scenarios
In an ITT analysis, participants
are analyzed in the groups
they were originally assigned
to (new drug or placebo),
regardless of these deviations.
Participants who stopped taking the new drug are still analyzedas
part of the new drug group.
Participants who switched from placebo to the new drug are still
analysed as part of the placebo group.
Efforts are made to include all participants in the final analysis,
potentially using methods to handle missing data.
Suppose an RCT is testing the
efficacy of a new drug for
hypertension. Participants are
randomly assigned to either
the new drug group or a
placebo group. During the
trial:
Some participants in the new drug group stop taking the
medication due to side effects.
Some participants in the placebo group switch to taking the new
drug.
Some participants are lost to follow-up and have no outcome
data.

27. Advantages of ITT Versus Per Protocol
Intention to Treat (ITT)
•Preservation of randomization
•Real-world applicability
•Conservative estimate of
treatment effect
Per Protocol
•More accurate measure of real
effect of treatment (since purity
of those receiving actual
treatment only is maintained).

28. ITT or Per Protocol: which is better?
•If the aim is to know the effect of actually recieving the
intervention (i.e. if there is NO effect, or what the true effect is)
then per protocol may be better.
•If aim is to know the effect of prescribing the intervention
(particularly in a real-life setting) then ITT is better
•Or: you do both analyses and see

30. PRACTICAL ACTIVITY: RoB Assessment of
Paper using ROB2 and COSMIN

31. PRACTICAL ACTIVITY: RoB Assessment of
Paper using ROB2 and COSMIN

32. Learning
Outcomes
At the end of the lesson, participants should:
•Understand the reason for and place of a QA/RoB in
Systematic Review
•Know the Difference between the terms “Quality
Assessment” and “Risk of Bias assessment”
•Appreciate the approach to and principles underlying
a RoB
•Have a broad overview of various RoB assessment
tools
•Be familiar with at least 2 of these RoB tools
•Know how/where to find the appropriate RoB tool
AuthorAid 2024

33. THANK YOU
FOR
LISTENING!