LIDC_Data_Collection_Process.power point

Data collection process

1. The Lung Image Database Consortium (LIDC) The Lung Image Database Consortium (LIDC)
Data Collection ProcessData Collection Process
This presentation based on the RSNA 2004 InfoRAD theater This presentation based on the RSNA 2004 InfoRAD theater
presentation titledpresentation titled
“The Lung Imaging Database Consortium (LIDC) : “The Lung Imaging Database Consortium (LIDC) :
Creating a Publicly Available Database to Stimulate Research Creating a Publicly Available Database to Stimulate Research
in CAD Methods for Lung Cancer”in CAD Methods for Lung Cancer”
(9110 DS-i)(9110 DS-i)
November 29, 2004November 29, 2004
Michael McNitt-Gray (UCLA), Anthony P. Reeves (Cornell), Michael McNitt-Gray (UCLA), Anthony P. Reeves (Cornell),
Roger Engelmann (U. Chicago), Peyton Bland (U. Michigan), Roger Engelmann (U. Chicago), Peyton Bland (U. Michigan),
Chris Piker (U. Iowa), John Freymann (NCI) and Chris Piker (U. Iowa), John Freymann (NCI) and
The Lung Image Database Consortium (LIDC)The Lung Image Database Consortium (LIDC)

2. Principal Principal
GoalsGoals
To establish standard formats and processes To establish standard formats and processes
for managing thoracic CT scans and related for managing thoracic CT scans and related
technical and clinical data for use in the technical and clinical data for use in the
development and testing of computer-aided development and testing of computer-aided
diagnostic algorithms.diagnostic algorithms.

3. Principal Principal
GoalsGoals
To establish standard formats and processes To establish standard formats and processes
for managing thoracic CT scans and related for managing thoracic CT scans and related
technical and clinical data for use in the technical and clinical data for use in the
development and testing of computer-aided development and testing of computer-aided
diagnostic algorithms.diagnostic algorithms.
To develop an image database as a web-To develop an image database as a web-
accessible international research resource for accessible international research resource for
the development, training, and evaluation of the development, training, and evaluation of
computer-aided diagnostic (CAD) methods for computer-aided diagnostic (CAD) methods for
lung cancer detection and diagnosis using lung cancer detection and diagnosis using
helical CT.helical CT.

4. The The
DatabaseDatabase•The database will contain:The database will contain:
1)1) A collection of CT scan imagesA collection of CT scan images
2)2) Technical factors about the CT scanTechnical factors about the CT scan
•Non-patient information in DICOM headerNon-patient information in DICOM header
3)3) For Nodules > 3 mm diameterFor Nodules > 3 mm diameter
•Radiologist drawn boundariesRadiologist drawn boundaries
•Description of characteristicsDescription of characteristics
4)4) For Nodules < 3 mmFor Nodules < 3 mm
• Radiologist marks centroid, no characteristicsRadiologist marks centroid, no characteristics
5)5) Pathology results or diagnosis Pathology results or diagnosis
information whenever availableinformation whenever available
6)6) All in a searchable relational All in a searchable relational
databasedatabase

5. How to do this?How to do this?
The LIDC Data Collection ProcessThe LIDC Data Collection Process

For nodule detection, recent research For nodule detection, recent research
has demonstrated that the results from a has demonstrated that the results from a
single reader are not sufficientsingle reader are not sufficient

6. How to do this?How to do this?
The LIDC Data Collection ProcessThe LIDC Data Collection Process

At least two and perhaps four readers At least two and perhaps four readers
may be required.may be required.

Not practical to do joint reading sessions Not practical to do joint reading sessions
across five institutionsacross five institutions

LIDC Will NOT do a forced consensus LIDC Will NOT do a forced consensus
read. We won’t force agreement on read. We won’t force agreement on
location of a nodule nor its boundary.location of a nodule nor its boundary.

7. Truth – DetectionTruth – Detection
LIDC – Initial ApproachLIDC – Initial Approach

Multiple Reads with Multiple ReadersMultiple Reads with Multiple Readers

First Read – 4 readers, each reads First Read – 4 readers, each reads
independently (Blinded)independently (Blinded)

Compile 4 blinded reads and distribute to Compile 4 blinded reads and distribute to
readersreaders

Second Read – Same 4 readers, this time Second Read – Same 4 readers, this time
unblinded to the results of the other readers unblinded to the results of the other readers
from the first reading.from the first reading.

Still, no forced consensus on either location Still, no forced consensus on either location
of nodules nor on their boundaries.of nodules nor on their boundaries.

8. Blinded Reads – Each Reader Reads Independently
(Blinded to Results of Other Readers)

9. Reader 1
Blinded Read for Reader 1 – Marks Only One Nodule

10. Reader 2
Blinded Read for Reader 2 – Marks Two Nodules
(Note: One nodule is same as Reader 1)

11. Reader 3
Blinded Read for Reader 3 – Marks Two Nodules
(Note: Again, One nodule is same as for Reader 1)

12. Reader 4
Blinded Read for Reader 4 – Did Not Mark Any Nodules

13. 2
nd
Round - UnBlinded Reads
Readings in Which Readers Are Shown Results of Other Readers
Each Reader Marks Nodules After Being Shown Results From
Their Own and Other Readers’ Blinded Reads
(Each Reader Decides to Include or Ignore).

14. Reader 1
Unblinded Read for Reader 1 – Now Marks Two Nodules
(Originally only marked one)

15. Reader 2
Unblinded Read for Reader 2 – Still Marks Two Nodules
(No Change)

16. Reader 3
Unblinded Read for Reader 3 – Now Marks Three Nodules
(Originally only marked two)

17. Reader 4
Unblinded Read for Reader 4 – Now Marks Three Nodules
(Originally did not mark any)

18. 4/4 Markings
2/4 Markings
2/4 Markings
Results of Unblinded Reads from All Four Readers
We will capture one aspect of reader variability in this way

19. Radiologist Review & Reconcile- V2
•4 radiologist (blinded) – R1B, R2B, R3B, R4B
Radiologist Review & Reconcile- V2
•4 radiologist (blinded) – R1B, R2B, R3B, R4B
•Submit to Requesting Site; This site compiles markings and re-sends case
Radiologist Review & Reconcile- V2
•4 radiologist (blinded) – R1B, R2B, R3B, R4B
•Submit to Requesting Site; This site compiles markings and re-sends case
•4 Radiologists see all (anonymized) markings
Radiologist Review & Reconcile
•4 Radiologists Perform Blinded Read – R1B, R2B, R3B, R4B
•Submit to Requesting Site; This site compiles markings and re-sends case
•4 Radiologists see all (anonymized) markings
•4 Radiologists Perform Unblinded Read (R1U, R2U, R3U, R4U)
Database (will contain Blinded AND
Unblinded reads)
R1U R2U R3U R4U
Nodules for each condition: (R1B, R2B, R3B, R4B, R1U, R2U, R3U, R4U)
• Location
• Outline (where appropriate)
• Label (where appropriate)

20. Case 5, Slice 19Case 5, Slice 19

21. Radiologist 1 - Method 1Radiologist 1 - Method 1

22. Radiologist 1 - Method 2Radiologist 1 - Method 2

23. Radiologist 1 - Method 3Radiologist 1 - Method 3

24. Radiologist 2 - Method 1Radiologist 2 - Method 1

25. Radiologist 2 - Method 3Radiologist 2 - Method 3

26. Radiologist 3 - Method 1Radiologist 3 - Method 1

27. Radiologist 3 - Method 2Radiologist 3 - Method 2

28. Radiologist 3 - Method 3Radiologist 3 - Method 3

29. Radiologist 4 - Method 1Radiologist 4 - Method 1

30. Radiologist 4 - Method 2Radiologist 4 - Method 2

31. Radiologist 4 - Method 3Radiologist 4 - Method 3

32. Radiologist 5 - Method 1Radiologist 5 - Method 1

33. Radiologist 5 - Method 3Radiologist 5 - Method 3

34. 
For each voxel, sum the number of For each voxel, sum the number of
occurrences (across reader markings) that occurrences (across reader markings) that
it was included as part of the noduleit was included as part of the nodule

Create a probabilistic map of nodule voxelsCreate a probabilistic map of nodule voxels

Higher probability voxels are shown as Higher probability voxels are shown as
brighter; lower probability are darkerbrighter; lower probability are darker

Can use apply a threshold and show only Can use apply a threshold and show only
voxels > some prob. Value if desired.voxels > some prob. Value if desired.
How to Represent This Variability? How to Represent This Variability?
Create a Probabilistic Description of Create a Probabilistic Description of
Nodule BoundaryNodule Boundary

35. Probabilistic Description of Probabilistic Description of
BoundaryBoundary

36. Apply Threshold if DesiredApply Threshold if Desired

37. Challenge: Define the Boundary of a Challenge: Define the Boundary of a
NoduleNodule

Do we need to have agreement between Do we need to have agreement between
radiologists on boundaries?radiologists on boundaries?

LIDC’s answer is no.LIDC’s answer is no.

LIDC Approach will be to:LIDC Approach will be to:

Construct a probabilistic description of boundaries Construct a probabilistic description of boundaries
to capture reader variabilityto capture reader variability

Use a threshold value (50% centile or 1% centile) to Use a threshold value (50% centile or 1% centile) to
give fixed contours.give fixed contours.

38. Pathology InformationPathology Information

In those cases in which pathology is In those cases in which pathology is
available, we will extract from reports:available, we will extract from reports:

Whether histology or cytology was Whether histology or cytology was
performedperformed

If histology, try to establish the cell type If histology, try to establish the cell type
according to WHO classificationsaccording to WHO classifications

If cytology, establish whether it was benign If cytology, establish whether it was benign
or malignantor malignant

39. Pathology InformationPathology Information

If no pathology, other diagnostic If no pathology, other diagnostic
information may be substituted when information may be substituted when
available (such as 2 years Dx F/U with no available (such as 2 years Dx F/U with no
change in radiographic appearance).change in radiographic appearance).

If neither is available, then case will be If neither is available, then case will be
used for detection purposes only.used for detection purposes only.

40. Database ImplementationDatabase Implementation
How to capture and collect all of this data?How to capture and collect all of this data?
5 Phases of Data Collection5 Phases of Data Collection
1.1.Initial ReviewInitial Review

review case for inclusion in database; review case for inclusion in database;

anonymize case; anonymize case;

Index case, e.g. Full Chest/Limited Chest, Image Quality.Index case, e.g. Full Chest/Limited Chest, Image Quality.
2.2.Blinded ReadBlinded Read

identifying and drawing nodules independentlyidentifying and drawing nodules independently
3.3.Unblinded ReadUnblinded Read

confirming using an overread, labeling nodules (characteristics)confirming using an overread, labeling nodules (characteristics)
4.4.Subject infoSubject info

demographics, smoking history, pathology.demographics, smoking history, pathology.
5.5.Export Data to NCI-hosted database (public)Export Data to NCI-hosted database (public)

41. Database ImplementationDatabase Implementation
How to capture and collect all of this data?How to capture and collect all of this data?
We have developed an internal standard for We have developed an internal standard for
representing a representing a region of interest (ROI)region of interest (ROI) that is 3- that is 3-
D based on xml. This is portable across D based on xml. This is portable across
software drawing tools.software drawing tools.
We are also using xml to capture radiologist We are also using xml to capture radiologist
interpretation of interpretation of nodule characteristicsnodule characteristics (shape, (shape,
subtlety, etc.) by using a limited set of subtlety, etc.) by using a limited set of
descriptorsdescriptors

42. Database ImplementationDatabase Implementation
How to capture and collect all of this data?How to capture and collect all of this data?
We have designed and tested a We have designed and tested a communication communication
protocolprotocol to send image data and xml messages to send image data and xml messages

Read Read RequestRequest messages (with a messages (with a
code/mechanism to distinguish blinded from code/mechanism to distinguish blinded from
unblinded read request)unblinded read request)

Read Read ResponseResponse messages (with a messages (with a
code/mechanism to distinguish blinded from code/mechanism to distinguish blinded from
unblinded read response)unblinded read response)

43. Database ImplementationDatabase Implementation
How to capture and collect all of this data?How to capture and collect all of this data?
Designed and implemented database for Designed and implemented database for
each host site for all case data.each host site for all case data.
Designed and are implementing the Designed and are implementing the
central NCI hosted database. central NCI hosted database.

44. Database ImplementationDatabase Implementation
Communication ModelCommunication Model

Each Site Plays Dual RolesEach Site Plays Dual Roles

As a Requesting SiteAs a Requesting Site

Identify Case and collect dataIdentify Case and collect data

Phase 1- Initial ReviewPhase 1- Initial Review

Manage it through blinded and unblinded read processManage it through blinded and unblinded read process

Create database entry for caseCreate database entry for case

Phase 4 – Demographics, PathologyPhase 4 – Demographics, Pathology

Phase 5 – Export to NCIPhase 5 – Export to NCI

NOTE: Site does not READ/MARK its own casesNOTE: Site does not READ/MARK its own cases

As a Servicing SiteAs a Servicing Site

Perform blinded (Phase 2) and unblinded (Phase 3) readsPerform blinded (Phase 2) and unblinded (Phase 3) reads

45. A
B,C,D,E
Initial
Review,
Anonymize
Send Image Data
XML Reading
Assignment message
XML Reading Response Message,
Compile
Responses
Nodule
Marking
Tools
SSH or SCP for transfer
Other Subject data fields
Linked to case
1
5
11
6
3
2
4
LIDC Message System
X
7
9
8
10
Requesting Site
Servicing Site

46. Access to LIDC DatabaseAccess to LIDC Database

Cases Exported to NCICases Exported to NCI

NCI hosts DatabaseNCI hosts Database

Publicly AvailablePublicly Available

Query Based on Data Elements CollectedQuery Based on Data Elements Collected

Imaging Data such as Slice Thickness, etc.Imaging Data such as Slice Thickness, etc.

Pathology or F/U DataPathology or F/U Data

Other FieldsOther Fields

ObtainObtain

Image Data including DICOM headersImage Data including DICOM headers

Serial Imaging when availableSerial Imaging when available

Radiologists’ Identification, Contours and Characterization of NodulesRadiologists’ Identification, Contours and Characterization of Nodules

Diagnosis Data (Path, Radiographic F/U, etc) whenever availableDiagnosis Data (Path, Radiographic F/U, etc) whenever available

Case Demographics whenever availableCase Demographics whenever available

Currently Implementing MIRC model (see infoRAD exhibit for Currently Implementing MIRC model (see infoRAD exhibit for
demo)demo)

47. Database ImplementationDatabase Implementation
TASKS COMPLETED (see reports on website):TASKS COMPLETED (see reports on website):

Specification of Inclusion Criteria:Specification of Inclusion Criteria:

CT scanning technical parametersCT scanning technical parameters

Patient inclusion criteriaPatient inclusion criteria

Process Model for Data collectionProcess Model for Data collection

Determination of Spatial "truth" Using Blinded Determination of Spatial "truth" Using Blinded
and Unblinded readsand Unblinded reads

Development of Boundary Drawing ToolsDevelopment of Boundary Drawing Tools

Development and implementation of xml Development and implementation of xml
standard for ROIsstandard for ROIs

48. Database ImplementationDatabase Implementation
TASKS COMPLETEDTASKS COMPLETED

Defined Common Data Elements for LIDCDefined Common Data Elements for LIDC

Database design – tables and Database design – tables and
relationships between tablesrelationships between tables

Communication protocolCommunication protocol

Establishing Public Database and Access Establishing Public Database and Access
Mechanism at NCIMechanism at NCI

49. Other Products Other Products
Publications/PresentationsPublications/Presentations

LIDC Overview manuscriptLIDC Overview manuscript

Radiology 2004 Sep;232(3):739-748. Radiology 2004 Sep;232(3):739-748.

Assessment Methodologies manuscriptAssessment Methodologies manuscript

Academic Radiology April 2004 Academic Radiology April 2004

((Acad Radiol 2004; 11:462–475)

Special Session SPIE Medical Imaging Special Session SPIE Medical Imaging

Sunday evening session at SPIE, 2005Sunday evening session at SPIE, 2005

50. SummarySummary

LIDC mission – to create public LIDC mission – to create public
databasedatabase

Current understanding of problem dictated Current understanding of problem dictated
multiple readersmultiple readers

Multi-Institutions dictated distributed, Multi-Institutions dictated distributed,
asynchronous readsasynchronous reads

51. SummarySummary

LIDC developed:LIDC developed:

Process Model for Blinded and Unblinded Reads Process Model for Blinded and Unblinded Reads
w/Multiple Readersw/Multiple Readers

Infrastructure to Communicate Radiologist Expert Infrastructure to Communicate Radiologist Expert
InformationInformation (Markings, Contours, Labelings) (Markings, Contours, Labelings)

Data Elements –image, meta data (DICOM), radiologist Data Elements –image, meta data (DICOM), radiologist
markings, contours and labels, pathology, demographicsmarkings, contours and labels, pathology, demographics

Data Representation Scheme (xml)Data Representation Scheme (xml)

Communication (messaging) protocolCommunication (messaging) protocol

Database DesignDatabase Design

Mechanism to handle reader disagreement/variabilityMechanism to handle reader disagreement/variability