Reducing waste from questionable research practices

Presentation from Professor Paul Glasziou, Bond University, to the AHMEN meeting on 19 June 2017, in Sydney, Australia.

1. Reducing waste from questionable
research practices
Paul Glasziou, Bond University
www.crebp.net.au

2. http://blogs.bmj.com/bmj/2016/01/14/paul-glasziou-and-iain-chalmers-is-
85-of-health-research-really-wasted/
Lancet Adding Value, Reducing Waste 2014
www.researchwaste.net
Five stages of waste in research
Annual waste in research is estimated to be 85% - from stages:
2. avoidable design flaws (50%),
4. non-publication (50%) and
5. unusable reports (50%)
– global total of over $140 Billion/year.
http://blogs.bmj.com/bmj/2016/01/14/
paul-glasziou-and-iain-chalmers-is-85-of-health-research-really-
wasted/

3. 50% of research is not published
Lancet 2014;383:257–66
But similar across countries, size, phase, …
WASTE

4. Non-Publication: a solution*
All Trials Registered; All Results Reported
*Simes RJ. Publication bias: the case for an international registry of clinical trials. J Clin Oncol. 1986

5. Non-reporting: lots of variation
Trials Tracker
Total (45% missing)
Kyowa (80% missing) Univ Ghent (84% missing)
Eli Lilly (5% missing)Johns Hopkins (24%
missing)

6. Trials registration rates: 2000-2016
International Committee of
Medical Journal Editors
requires registration
Food and Drug
Administration
Amendments Act of 2007

7. Posting of Summary Trial Results
10% of
registered
trials

8. Trials registration
Opportunities to check registration at:
1.Funding approval
2.Ethics approval
3.Final report to funder *
(+ post summary results)
4.Publication
(+ post summary results)
* NIHR’s HTA program requires, and has 98%
publication rate for their trials!

9. http://blogs.bmj.com/bmj/2016/01/14/paul-glasziou-and-iain-chalmers-is-
85-of-health-research-really-wasted/
Lancet Adding Value, Reducing Waste 2014
www.researchwaste.net
Five stages of waste in research
Annual waste in research is estimated to be 85% - from stages:
2. avoidable design flaws (50%),
4. non-publication (50%) and
5. unusable reports (50%)
– global total of over $140 Billion/year.
http://blogs.bmj.com/bmj/2016/01/14/
paul-glasziou-and-iain-chalmers-is-85-of-health-research-really-
wasted/

10. So what can journals do?

12. Improving Reporting: brief history
1930s: IMRaD structure
(Introduction; Methods;
Results and Discussion)
1987: Structured Abstracts
1996: CONSORT for Trials
2006: EQUATOR Network
Uptake of IMRaD.
Luciana B. J Med Libr Assoc. 2004

13. Missing elements in Reports of
Randomized Clinical Trials (PubMed)
13

14. Reproducible controlled trials
CONSORT checklist 2010 (25 items)
TITLE & ABSTRACT
INTRODUCTION
Background
Objectives
METHODS
Trial design
Participants
Interventions
Outcomes
Sample size
Randomization
Sequence generation
Allocation concealment
Implementation
Blinding (Masking)
Statistical methods
RESULTS
Participant flow
Recruitment
Baseline data
Numbers analyzed
Outcomes and Estimation
Ancillary analyses
Harms
DISCUSSION
Limitations
Generalisability
Interpretation
OTHER INFORMATION
Registration
Protocol Funding

15. Outcome switching
Ask for the protocol
at submission
Have open “rapid
responses” of use
PubMed Commons

16. Reproducible controlled trials
CONSORT checklist 2010 (25 items)
TITLE & ABSTRACT
INTRODUCTION
Background
Objectives
METHODS
Trial design
Participants
Interventions
Outcomes
Sample size
Randomization
Sequence generation
Allocation concealment
Implementation
Blinding (Masking)
Statistical methods
RESULTS
Participant flow
Recruitment
Baseline data
Numbers analyzed
Outcomes and Estimation
Ancillary analyses
Harms
DISCUSSION
Limitations
Generalisability
Interpretation
OTHER INFORMATION
Registration
Protocol Funding

17. What is the treatment?
The paper’s description of sodium reduction
"Individual and weekly group counseling sessions were offered
initially, with less intensive counseling and support thereafter,
specific to sodium reduction."

What would you tell a patient to do?
TOHP Study BMJ, Apr 2007; 334: 885

18. What is sodium reduction?
The paper’s description
"Individual and weekly group counseling sessions were
offered initially, with less intensive counseling and support
thereafter, specific to sodium reduction."
Description in earlier (paywalled) reference:
1.an individual session followed by 10 weekly group 90
minute sessions with a nutritionist, followed by a
transitional stage of some additional sessions
2.Topics in the weekly sessions included Getting Started,
sodium basics, the morning meal, midday sources of sodium,
the main meal, planning ahead, creative cooking, eating out,
food cues, and social support,
3.the sessions included sampling of foods, discussion of
articles on sodium reduction, and problem-solving,
4.patients kept diaries at least 6 days per week, and urine
sodiums were measured.

19. Poor reporting of non-pharmacological
interventions in 6 major medical journals
Of 133 trials in 2010
59% adequate after
contacting author
39% adequate in
primary sources
Hoffmann, Erueti, Glasziou. Poor description of non-pharmacological
interventions: A remediable barrier to evidence use in practice? BMJ 2013

20. The TIDieR checklist (1-7)

21. Methods, Materials, and Data

23. Fig 3. Reportedly available materials.
Kidwell MC, Lazarević LB, Baranski E, Hardwicke TE, Piechowski S, et al. (2016) Badges to Acknowledge Open Practices: A
Simple, Low-Cost, Effective Method for Increasing Transparency. PLoS Biol 14(5): e1002456.
doi:10.1371/journal.pbio.1002456
http://journals.plos.org/plosbiology/article?id=info:doi/10.1371/journal.pbio.1002456

24. Fig 2. Reportedly available data.
Kidwell MC, Lazarević LB, Baranski E, Hardwicke TE, Piechowski S, et al. (2016) Badges to Acknowledge Open Practices: A
Simple, Low-Cost, Effective Method for Increasing Transparency. PLoS Biol 14(5): e1002456.
doi:10.1371/journal.pbio.1002456
http://journals.plos.org/plosbiology/article?id=info:doi/10.1371/journal.pbio.1002456

25. What might journals do?
Study registration at submission
Use reporting guidelines
Ask & link protocol/materials
Assign someone to “ensure value”