2
App Name: PREVENTION ABSTRACT CLASSIFICATION TOOL (PACT)
Access URL: N/A
Docs URL: (if any) None
IC: OD/ODP
Submitter Name: Patricia L. Mabry, Ph.D.
Email Address:
[email protected] Phone (301) 402-1753
Functional Area
Where Used:
Extramural How Acquired? Custom Development
App IT Contact: Rich Panzer
Operating Sys: Windows DBMS SQL
Front End: .NET, HTML, CSS
We intend to: Just an FYI / other
FUNCTIONALITY PROVIDED
The Prevention Abstract Classification Tool (PACT) facilitates coding of grant applications used in
portfolio analysis. ODP is working with NIH Office of Portfolio Analysis (OPA) and Center for Information
Technology (CIT) to create a machine-learning based, computerized portfolio analysis tool to
characterize the Prevention Research Portfolio. This requires “gold standards” (a training set) for the
machine to learn from. “Gold standards” are grant abstracts that have been accurately classified
(manually) according to ODP’s Prevention Research Taxonomy. Many thousand gold standards,
generated by manual classification (aka coding )
, are needed for the machine to “learn.” Accurate coding
requires a team approach: each abstract is read and coded by three individuals who then discuss and
reach a “consensus code” that becomes the gold standard. A 23-page protocol dictates the rules for
coding based on the Taxonomy. For a subset of coded abstracts, a second team of three coders recodes
the abstracts following the same procedure. The two consensus codes are reconciled to yield a revised gold standard, if indicated. This double coding provides quality control.
PACT was requisitioned by ODP to facilitate the coding of the large number of gold standards needed.
PACT is a software tool containing a large database of grant abstracts imported from NIH RePORTER.
PACT provides a user interface for coding abstracts in accordance with the Taxonomy and allows coders
to seamlessly access relevant parts of the protocol in real time while coding, enhancing accuracy and reliability. PACT captures and stores data on individual and consensus codes. Because there are multiple
categories within the Taxonomy to be coded, multiple coders, and multiple teams of coders, PACT
calculates 96 inter- rater reliability statistics (Kappas) for each abstract.
A training and testing environment in PACT parallels the “production” environment. Would- be coders
use PACT in training mode to learn the Taxonomy and protocol by coding a large number of abstracts
selected for their pedagogical value. After training is completed, would-be coders code a test set of
abstracts. Kappas are reviewed to ensure that each coder’s test coding is at or above a predetermined
threshold for inter-rater reliability when compared to the correct codes. During production, Kappas are
used to identify abstracts that need coding by a second team, to identify problem coders, and as
indicators of quality control.
ODP and IQ Solutions, Inc. are showing this app to other NIH staff in order to demonstrate the utility of
PACT for supporting abstract coding for portfolio analysis. We think others who are doing any type of
portfolio analysis that relies on manual coding will be interested in our approach and the software that
facilitates this process. Interested parties may contact ODP or IQ Solutions about adapting the approach