RISK OF BIAS assessmment in metaanalysis

Risk of bias is the likelihood that flaws in a study's design or execution will produce a skewed or misleading result. It is a critical aspect of assessing evidence quality, particularly in systematic reviews and meta-analyses, and helps to determine the validity of a study's findings.

1. Assessment of
Risk of Bias
M Jeeva Sankar
[email protected]

2. Overview
¡What & why?
–Definition; importance in systematic reviews
¡How?
–Cochrane Risk of Bias (RoB2) tool
–Using the tool kit

3. What & why?

4. Bias
Any systematic deviation
from ‘truth’Systematic error!

5. Bias
>100 biaseshttps://catalogofbias.org/biases/

6. Why in systematic reviews?
Who is/are ACCURATE?Syst review improves precision
but NOT accuracy!!Who is/are PRECISE?

7. Bias: Why in syst reviews?
Bias in included studies =
Biased results of review

8. Bias: Why in syst reviews?
Unclear vs. adequate randomizationBenefits exaggerated by
11%
Savovic 2012

9. Bias: Why in syst reviews?
No vs. double-blindingBenefits exaggerated by
13%
Savovic 2012

10. Bias: Why in syst reviews?
Small vs. large studiesBenefits exaggerated by
up to 50%
Dechartres2013

11. Bias: Affectsstudyresults
Affectsresults of systematic review
Certaintyofbody ofevidence
Strengthof recommendations
Bias: Why in syst reviews?
GRADE
profile

12. How?
Risk of bias tools

13. Assessing the risk of bias
https://sites.google.com/site/riskofbiastool/welcome/rob-2-0-tool

14. RoB2 tool: Principles
¡Assess risk of bias (RoB) for ALLkey resultsin
EACH study
¡Involves judgement; provide supporting
data –should be transparent
¡Sources of supporting datato be provided
¡Study-level(e.g., bias in randomization)
vs.
Review-levelbias (e.g., publication bias) Done by at least 2 people!

15. RoB2 tool: 5Domains
Bias in randomization
Bias due to deviations from intended
interventions
Bias due to missing outcome data
Bias in outcome measurement
Bias in selection of reported results

16. Signalingquestions
Judgement of risk
(algorithm-based)
Text boxes to justify
Predictlikelydirection ofbias
(optional)
RoB2 tool: For each Domain
Based on responsesResponses:
1.Yes;
2.Probably yes;
3.Probably no;
4.No;
5.No information

17. Domain 1: Randomization
Allocation sequence
was random?
Allocation sequence
was adequately
concealed?
Baseline differences
suggest problem with
randomization process?

18. Domain 1: Randomization
Allocation
sequence
random?
Computer-generated random
numbers; random number table;
coin tossing; shuffling cards;
throwing dice; or drawing lots
Allocation
sequence
adequately
concealed
Remote or centrally
administered method;
opaque, sequentially
numbered, sealedenvelopes;
sequentially numbered,
identical drug vials
Baseline
differences
?
Substantial differences
between group sizes;
substantial excess in significant
differences in baseline
characteristics; imbalance in
prognostic factorsunlikely to
be due to chance

19. Domain 1: Randomization

20. Domain 1: Randomization

21. Domain 1: Randomization

22. Domain 1: Randomization

23. Domain 2: Deviations from
interventions
Review to assess effect of
Assignmentto
intervention
(‘intention-to-treat’)
Adherenceto
intervention
(‘per-protocol’)

24. Domain 2: Deviations…
Participants aware of
assigned intervention?
Carers and people
delivering interventions
awareof assignment?
Deviations from
intervention because of
trial?
Deviations likely to have
affected outcome?

25. Domain 2: Deviations…
Deviations balanced
betweengroups?
Appropriate analysis used to
estimate effect of
assignment to intervention?
Potential for an impact of the
failure toanalyse participants
in the groups they were
randomized?

26. Domain 2: Deviations…
Participants
aware of
intervention?
Are participants aware of the
intervention administered?
Carers and
others aware
of
intervention?
Are carers andpeople
delivering the intervention
awareof the intervention
administered?
Deviations
because of
trial?
If trial context led to failure to
implement protocol interventions
or implementation of
interventions not allowed by the
protocol?
Changesfrom intervention
inconsistent with trial protocol?

27. Domain 2: Deviations…
Deviations
likely to affect
outcomes?
Will changes from assigned
interventioninconsistent with
trial protocol affect the
outcome?
Deviations
balanced
between
groups?
Are changes from assigned
interventioninconsistent with
trial protocol balanced
between thetwo groups?
Appropriate
analysis
used?
Were intention-to-treat (ITT)
analyses or modifiedintention-
to-treat (mITT) analyses used?
Anypost-randomization
exclusions?

28. Domain 2: Deviations…

29. Domain 2: Deviations

30. Domain 3: Missing outcome
Data availableforALL
participants?
Evidence that result was
NOT biased by missing
outcome data?
Could missing outcome
depend on its true value?
LIKELY that missing
outcome depended on its
true value?

31. Domain 3: Missing outcome
Data
available
for ALL?
Number of participants with
missing outcome is sufficiently
smallthat their outcomes could
have made no important
difference
Result NOT
biased by
missing
outcome?
Evidence may come from: (1)
analyses that correct for bias; or
(2) sensitivity analyses showing
that results are little changed
under a range of plausible
assumptions
Could
missingness
depend on
its true
value?
If loss to follow up, or withdrawal
from the study, could be related
to participants’ health status

32. Domain 3: Missing outcome
LIKELY that
missingness
dependson
its true
value?
Differencesbetween
groups in the proportions of
missing outcomes;
Reasons for missing data
provide evidence that
missingness depends on
true value;
Reported reasons differ
between the groups

33. Domain 3: Missing outcome

34. Domain 3: Missing outcome

35. Domain 4: Outcome
measurement
Method of outcome
measurement
inappropriate?
Measurementdiffered
between the groups?
Outcome assessors aware
ofintervention groups?
Assessment influenced by
knowledge ofintervention
received?

36. Domain 4: Outcome
measurement
Outcome
measurement
inappropriate?
Outcome unlikely to be
sensitive to intervention;
Instrument haspoor validity
Measurement
differed
between
groups?
Passive collection of data –
‘diagnostic detection bias’;
intervention involves additional
visits–more opportunities for
outcome measurement
Outcome
assessors
aware of
intervention?
If outcome assessors were
blindedor not to intervention
groups

37. Domain 4: Outcome
measurement
Outcome
assessment
influenced by
knowledge of
intervention?
Likely in ‘subjective’
outcomes and not in
‘objective’ outcomes like
mortality

38. Domain 4: Outcome
measurement

39. Domain 4: Outcome measurement

40. Domain 5: Selection of
reported result
Data analysed as per
the pre-specified
analysis plan?
Result selected from
multiple outcome
measurements?
Result selected from
multiple analyses ofthe
data?

41. Domain 5: Selection of
reported result

42. OVERALLrisk of bias

43. Toolkit
RoB2 Excel tool

44. Excel tool
https://drive.google.com/file/d/1uwAVr-
wKE3elEzcsVOBGLzJOVhbpf321/view

45. Excel tool

46. Excel tool

47. Excel tool

48. Thank you