AI for Event Monitoring and Risk Mitigation

Presentation at the 4th Patient Safety Congress 12 Sept 2024, Crowne Plaza Galleria Manila.

1. E)! @endocrine witch CCBYNCND

E
Artificial
Intelligence

FOR EVENT MONITORING AND RISK MITIGATION IN HEALTHCARE

Iris Thiele Isip Tan MD, MSc
Professor 12, UP College of Medicine
Director, UP Manila Interactive Learning Center

Chief, UP Medical Informatics Unit

vo a9ve

2. ¡(E)! @endocrine_witch CC BY NC ND

NO

Conflict
of Interest
to disclose with regard

to the subject matter of
this presentation

3. 'EJ' @endocrine-witch CC BY NC ND

ARTIFICIAL
Intelligence

the ability of a computer or health care
device to analyze extensive health care
data, reveal hidden knowledge, and
enhance communication

Choudhury, A, & Asan, O. (2020). Role of Artificial Intelligence in
Patient Safety Outcornes: Systematic Literature Review. JMIR medical
informatics, 8(7), e18599.

4. @endocrine_witch CC BY NC NO

Patient Safety

| | i Patient Safety /
Outcome

Intervention Influence

Influence

5. Role of Artificial Intelige

@endocrine_witch CC BY NC ND

PATIENT SAFETY
mms)

ADVERSE DRUG EVENT bi CLINICAL
DRUG SAFETY SENICALRERORTS ALARMS/ALETRS

- L..

MISSED INCIDENT | INCOMPLETE ANALYSIS IHEALTHICLINICAL | ERROR ALERTS |,

REPORT OF SAFETY REPORT aes

DRUG-DRUG
INTERACTION / OVERDOSE
REACTION

FALSE ALARMS

WRONG POOR IGNORED

CLINICAL VISIT

REPORT NOTES

SPECIMEN
LABELING

MEDICATION RECONCILIATION |_| PATIENT FEEDBACK EHR REPORTS =

6. E @endosine wich CC BYNEND

Artificial

Intelligence

ADVERSE DRUG
EVENTS/SAFETY

Natural language
processing to
detect ADRs in
hospital discharge
summaries

CLINICAL REPORTS

Real-time early
warning system to
monitor inpatient
mortality using
machine learning

CLINICAL
ALARMS/ALERTS

Machine-learning-
based clinical
decision support
system

7. E @endosine wich CC BYNEND

Natural Language
Processing

Building on a computer's ability to understand human
language and consecutively transform text to machine-
readable structured data, which can then be analyzed by
machine-learning techniques

&

Choudhury A, & Asan, O. 2020) Role of Arial Intelligence in Patient Safety,
Gutcomes, Systematic Literature Review, IMIR medical informatics, 8(7), 618209.

8. @endocrine_witch CC BY NC ND

Reference
sources
SR Natural
rug Language
gazetteer Processing Drug-AE Discharge

detection
summaries
a rules
gazetteer Tokenization

N (Rute Group 1)

‘Common nue = em
English words detection Rue Group 2)

Negation (Rule Group 3_)
terms es

Connecting gr EEE | TA

words (“Rule Groups )) {Predicted drug-AE
ae CRE relations

pairs

Vaccine-related Expert assessment
words using Readpeer-HSA

Readpeer for Active Pharmacovigilance (REAP)

ang, Y. etal (2019)
Summnanes of elec

adverse drug reactions in discharge

of Electronic Medical
Records Using
Readpeer

9. £3) endocrine. witch cc BY NEND

Relation rule

drug Cause AE

AE AttributeTo drug
AllergyTo drug

{drug StopAfter or StopBe)

drug) + word_d is(drug, AE) < 12

{AEBefore AE or AE AEAfter} + word _d is
(drug, AE) < 12

AttributeTo: {attributed to, secondary to, related to,
AllergyTo: {da to, allergic to, drug allergy, ...}

StopAfter: {stop, held off, interrupt, discontinue, take ol
...}; StopBefore: {stop, discontinue, take off, switch, change,
not to start, ...}

AEBefore: {in view of, following, noted, develop, likel
complain of, \fter: (develop, a: }

10. @endocrine_witch CC BY NC ND

I Observed
O Predicted

# Drug AE Pairs

2 3
Sentence Distance

c

5
5

0 12 345 6 7 8 910111213141516171819202122232425 2627

word distance

18 24 30 36 2 48 54 60 Gs ca. Gs ©

Word Distance

# Drug AE Pairs

Fig. 2. Distribution of drug-AE pairs.

11. n discharge
international journal

@endocrine_witch CC BY NC ND

Interface

12. EE): @endocrine witch CC BY NC ND

20111000 5835617
Creator: Machine
Context:
Left LL poer was 4, 5, subsequently progressed to 1, 5 on second day, Likely

progression of stroke, aspirin changed to clexane 40mg BD.- lower limb strength
subsequently improved in the ward to 3, 5, contin

User vote result:

Annotator 1: Wrong

Annotator 2: Correct

Annotator 3: Wrong

Tang, Y.etal 2019) Detecting a
international journal of

13. international journal

@endocrine_witch CC BY NC ND

Table 3
Drug and AE name recognition results.

Trial no. No. of Drug Drug AE precision AE recall
records precision recall

0.928 0.902 0.912

0.925 0.938 0.957

0.927 0.905 0.984

0.920 0.953 0.932

0.912 0.919 0.958

0.928 0.965 0.949

0.923 0.930 0.949

14. @endocrine_witch CC BY NC ND

Table 5
Results from applying REAP on the i2b2 dataset consisting of 100 records, with
129 drug-AE associations.

Type Precision Recall F-score

Drug 0.936 0.924 0.930
AE 0.811 0.979 0.887
Drug-AE 0.641 0.457 0.534

15. Role of Artificial Intelige

@endocrine_witch CC BY NC ND

PATIENT SAFETY
mms)

ADVERSE DRUG EVENT bi CLINICAL
DRUG SAFETY SENICALRERORTS ALARMS/ALETRS

- L..

MISSED INCIDENT | INCOMPLETE ANALYSIS IHEALTHICLINICAL | ERROR ALERTS |,

REPORT OF SAFETY REPORT aes

DRUG-DRUG
INTERACTION / OVERDOSE
REACTION

FALSE ALARMS

WRONG POOR IGNORED

CLINICAL VISIT

REPORT NOTES

SPECIMEN
LABELING

MEDICATION RECONCILIATION |_| PATIENT FEEDBACK EHR REPORTS =

16. Ye, Cet al (2019) A Real-Time Early Warning System for Monitoring
Inpatient Mortality Rske Pr Ve Study Using Electronic Medical
Record Data, Journal of mi net tesearch, Att) CISnS.

@endocrine_witch CC BY NC ND

for Monitoring Inpatient Mortality Risk

1. Retrospective cohort (2015.01.01 ~ 2016.12.31, n=42,484)

Controls Cases Retrospective model construction
Sea EMR data (4) (various machine learning methods)
- Demographics;
+ Medical history;
| Vital signs;
Discharge by 1 Lab tests Training ROC and determine thresholds

+

_ death Clinical utilizations; for high-risk early warning

(Treat each inpatient-day as the
observation window; exclude the
expired patients‘ last inpatient
day)

17. ]' @endocrine_witch CC BY NC ND

Real-time Early Warning System
for Monitoring Inpatient Mortality Risk

2. Prospective cohort (2017.01.01 ~ 2017.09.30, n=11,762)
rest each inpatient-day profiles as | Prospective apply the constructed early

the observation window (x) warning systems, and choose the best model

Real-time monitoring of mortality Po Evaluate the model's ability of

7 predicting encounter's inpatient
expiration risk in a real-time
surveillance

High-risk group:
Y PPV

Y Specificity
Admission Discharge alive Y Sensitivity

18. @endocrine_witch CC BY NC ND

Figure 3. The observed survival curves of the 3 risk eat

es (encounter level) statfied by the real-time early waming system in the prospective
validation cohort. HR: hazard ratio

Low-risk category

Intermediate-risk category

risk category

low-risk category (HR: reference)

Intermediate-risk category (HR: 25.78 [95%C1 19.24-34.54])
— High-risk category (HR: 93.65 [95%CI 68.75-127.57])

Time (days)

19. Real-Time Early Warning System fc
Fiske Pros study Using Ek
‘medical internet research, Zi

@endocrine_witch CC BY NC ND

Figure 4. The median real-time risk score curves of the patients who passed away stratified by 3 risk categories of the prospective cohort

The high-risk category

The high-risk cutoff

The intermediate-risk
category

The intermediate-risk
cutoff

3
&

The low-risk category

T
3

Time to death

20. | @endocrine_witch CC BY NC ND

Inpatient mortality real-time monitoring system

Hospitals encounter-level EMR data Notification of high-risk encounters

| Mortality risk category

à : à Y Warning high-risk patient of in-hospital
= 171 mortality before their death.

Y Anchoring individualized risk factors:

) @ Al

‘Abnormal Utilization Abnormal Diseases.
voisins cost Tabs dlagnotes

+
+ Sm

reviewing by
physicians

21. Role of Artificial Intelige

@endocrine_witch CC BY NC ND

PATIENT SAFETY
mms)

ADVERSE DRUG EVENT bi CLINICAL
DRUG SAFETY SENICALRERORTS ALARMS/ALETRS

- L..

MISSED INCIDENT | INCOMPLETE ANALYSIS IHEALTHICLINICAL | ERROR ALERTS |,

REPORT OF SAFETY REPORT aes

DRUG-DRUG
INTERACTION / OVERDOSE
REACTION

FALSE ALARMS

WRONG POOR IGNORED

CLINICAL VISIT

REPORT NOTES

SPECIMEN
LABELING

MEDICATION RECONCILIATION |_| PATIENT FEEDBACK EHR REPORTS =

22. E)! @endocrine witch CC BY NC ND

Probabilistic, machine-
learning based clinical
decision support system
Drug prescription
errors and ADEs

P: All drug prescriptions in EMR in a single
ward in a tertiary medical center
I: Novel outlier system in existing EMR

O: Accuracy, validity and clinical
usefulness of medication error alerts

‘Segal, Get a. 2019). Reducing drug prescription errors and adverse dru
of picbabliatic maine sang based crcl decison suppor yt
(ea Informatics Associaton JAMA, 261

Setting. Journal of the American

23. E)! @endocrine witch CC BY NC ND

Non-Al Clinical Decision Support

Systems

High incidence Rule-based Need for systems

of false alerts that could actively

EEE Miss error types monitorand |
IMSS that have not been identify emerging

small fraction of anticipated or ADEs

Senos programmed

‘Segal, Get al. (2019). Reducing drug prescription errors and adverse drug events by application
fa probabilistic, machine-leatning based clinical decision support system In an inpatient
Setting. Journal of the American Medical Informatics Association JAMIA, 26(12), 1560-1508

so a0ve

24. Segal, ©. et al. 2018). Reducing drug prescription errors and adverse drug events by application
ST pioosblst machine les based cinical dacsion support ayan an inpatient
Setting Journal 6f the American Medical Informatics Association AMIA, 26118, 1560-1565.

E)! @endocrine_witeh CC BY NC ND

MedAware

Clinical Decision Support System

Identifies prescriptions that are significant statistical
outliers given the patients’ clinical situations

SYNCHRONOUS ASYNCHRONOUS
ALERTS ALERTS
Pops up during the Generated after the
prescribing process medication order was already

entered, following a relevant
change in the patient's profile

25. Segal, ©. et al. 2018). Reducing drug prescription errors and adverse drug events by application
ST probable machine les based einen dec on support ayan an inpatient
Setting Journal 6f the American Medical Informatics Association JANA, 2611, 1560-1565.

E)! @endocrine_witeh CC BY NC ND

MedAware

Clinical Decision Support System

Identifies prescriptions that are significant statistical
outliers given the patients’ clinical situations

SYNCHRONOUS ASYNCHRONOUS
ALERTS ALERTS
+ Time-dependent Time-dependent irregularities

irregularities
« Clinical outliers
+ Dosage outliers
« Drug overlap

26. EI @endocrine witch CC BY NEND
Validation of

ALERTS

ACCURACY CLINICAL CLINICAL
VALIDITY USEFULNESS

Were there any

data-related issues Were there clinical Was the alert

that caused a false justification for this clinically useful to

alarm? medication to be the physician?

prescribed?

27. 3) @endocrine witch CC BY NC ND
Validation of

ALE R | S Was the alert
clinically useful to

the physician?

ACCURACY CLINICAL CUNICAL
USEFULNESS
VALIDITY

Naraiheio any eier 0: Alert irrelevant

related issues that Were there clinical 1: No clinical relevance

caused a false alarm? justification for this 2: Clinically relevant even if

YES = 0 medication to be MD overrides

OST prescribed? 3: Clinically relevant and MD

YES = 0; NO=1 should modify treatment

Segal, ©, et al. (2019). Reducing drug prescription errors and adverse drug events by ap accordingly

of probabilistic, machine learning based chnical accion support system
Sect Journal ofthe American Medical Informatles Association: JAMÍA 2

28. @endocrine_witch CC BY NC ND

Alert Type
‘Time dependent alerts (synchro)
Hypercarbi
Hyponatremia
Disrupted liver function
Bradycardia
Hyperkalemia
Thrombocytopenia

Hypotension

Disrupted coagulation tests (prolonged PT/INR)

Hypokalemia
Hypoglycemia
Dosage alerts

High dosage

Low dosage

Rare unit

Clinical alerts

Abbreviations: DIC, disseminated intravascular coa

‚mon Clinical scenario

Respiratory failure, sedation
SIADH

Hepatitis, Sepsis and septic shock
Cardiac arrhythmia

‘Acute kidney injury

Sepsis and DIC

Sepsis and septic shock, sedation
sepsis

sepsis

‘Sepsis and septic shock

ation; SIADH, syndrome of inappropriate antidiuretic hormo

Most common medications flagged

Sedatives, opioid narcotics
thiazide diuretics and SSRI's
Statins

Beta blockers, calcium channel blockers

Potassium sparing diuretics, Angiotensin-Converting-Enzyme inhibitors

Anticoagulants, anti-aggregates
Vasodilators

Anticoagulants

Diuretics, potassium associating resins

Insulin, oral hypoglycemic drugs

e secreion; SSRI, selective serotonin reuptake inhibitor.

29. @endocrine_witch CC BY NC ND

‘Time-Dependent Alert Typea A al a Most common medications flagged Percent
Hypercarbia Respiratory failure, sedation Sedatives, opioid narcotics 11.52%
Bradycardia Cardiac arrhythmia Beta blockers, calcium channel blockers 10.29%
Hypotension ‘Sepsis and septic shock, sedation Vasodilators 4.90%
‘Thrombocytopenia Sepsis and DIC Anticoagulants, anti-aggregates 4.66%
Disrupted liver function Hepatitis, sepsis, and septic shock Statins 441%
Hypoglycemia Sepsis and septic shock Insulin, oral hypoglycemic drugs 4.17%
Hyperkalemia Acute kidney injury Potassium sparing diuretics, Angiotensin-Converting-Enzyme inhibitors 2.94%
Disrupted coagulation tests (prolonged PT/INR) Sepsis. Anticoagulants 245%
Hypokalemia Sepsis Diuretics, potassium associating resins 1.72%

Hyponatremia SIADH Thiazide diuretics and SSRI's 0.49%

Acute kidney injury (elevated creatinine) epsis and septic shock Angiotensin-Converting-Enzyme inhibitors 025%

Hypercalcemia Malignancy calcium and vitamin D derivatives 025%

Elevated CPK levels Hepatitis, convulsions, rhabdomyolysis Statins 025%

Abbreviaions: DIC, disseminated intravascular coagulation; SIADH, syndrome of inappropriate antidiuretic hormone secretion; SSRI, selective serotonin reuptake inhibitor

"Asynchronous event/change in the patients profile necessitating certain medications to be flagged.

30. @endocrine_witch CC BY NC ND

causing physician behavior change

Alert Type Most common Clinical scenario Most common medicati Percent
High dosage
Low dosage

Bradycardia Cardiac arrhythmia Beta blockers

Disrupted liver function Hepatitis, sepsis and septic shock Statins

Hypotension Sepsis and septic shock, sedation Vasodilators

31. 5) @ondocrine witch CC BY NENB

Legacy CDS The System
Alert Burden (% of prescriptions)
Clinically relevant (% of alerts)

Caused a change in practice (% of alerts)

Post prescribing surveillance (% of alerts)

Clinical outliers

Time-dependent

Rare dosage unit

Drug-drug interaction

The legacy CDS generated
a high alert burden with a
high dismissal rate by
physicians.

32. Role of Artificial Intelige

@endocrine_witch CC BY NC ND

PATIENT SAFETY
mms)

ADVERSE DRUG EVENT bi CLINICAL
DRUG SAFETY SENICALRERORTS ALARMS/ALETRS

- L..

MISSED INCIDENT | INCOMPLETE ANALYSIS IHEALTHICLINICAL | ERROR ALERTS |,

REPORT OF SAFETY REPORT aes

DRUG-DRUG
INTERACTION / OVERDOSE
REACTION

FALSE ALARMS

WRONG POOR IGNORED

CLINICAL VISIT

REPORT NOTES

SPECIMEN
LABELING

MEDICATION RECONCILIATION |_| PATIENT FEEDBACK EHR REPORTS =

33. ¡(E)! @endocrine_witch CC BY NC ND

We are physicians.

We take this oath to do
no harm. That needs to
be the first way that we're
assessing any of these
tools.

-Nina Vasan, MD, MBA

Clinical Assistant Professor of Psychiatry
Founder & Executive Director of Brainstorm:
The Stanford Lab for Mental Health Innovation