Dev Dives: Measure the success of your document processing automation

Watch this Dev Dives session and learn how to measure the success of your document processing automation.
Dive into the key metrics that provide insights into your projects' success and find out how to understand and improve them for better automation rates.

📕 This Dev Dives webinar explores the following key topics:
- Understanding Success in Document Processing automations:
- Understand key metrics and how they tie into ROI to drive business opportunities;
- Gather actionable insights from data points;
- Use data points to improve your automation and make data-driven decisions.

🗣 Speakers:
Monica Luca, Sr. Product Manager @UiPath
Nathan Ness, Principal Sales Solutions Architect @UiPath

Download these tips&tricks of how to improve metrics
https://view.highspot.com/viewer/58a3192dc5a5147b21d48b237f53e948

⏩ Register for our upcoming Dev Dives October session: Unlock powerful data extraction with Semantic Activities
EMEA&APJ: https://bit.ly/Dev_Dives_October_EMEA_APJ
AMER: https://bit.ly/Dev_Dives_October_AMER
This session was streamed live on September 26, 2024.
Check out all our upcoming Dev Dives 2024 sessions at:
https://bit.ly/Dev_Dives_2024

1. Dev Dives
Go deeper, automate smarter

2. 2
Measure the success of your
document processing
automation
UiPath Dev Dives Webinar Series

3. 3
About today’s meeting
•Enjoy the next 60 min packed with useful guidance, demos,
andlive Q&As.
•You’ll receive the recording, useful links, and deck via your
email.
•Bonus content (PDF handout):
https://view.highspot.com/viewer/58a3192dc5a5147b21d
48b237f53e948
•Get answers to your questions and challenges. Please use
the chat box for Qs during the presentation. Live Q&A
session at the end.
•You're encouraged to network and share your
LinkedIn/Twitter in the chat.
•Have fun! Feedback is welcome.

4. 4
Monica Luca
Senior Product Manager
UiPath
Meet today’s speakers:
Nathan Ness
Principal Solutions Architect
UiPath

5. 5
Agenda slide
01
02
03
Defining what matters: Key metrics for evaluating success
Monitoring the numbers: Understanding the impact of your automations
Improving the numbers & taking data-informed decisions

6. Defining what matters
Key metrics for evaluating success
The UiPath word mark, logos, and robots are registered trademarks owned by UiPath, Inc. and its affiliates.
UiPath® is a registered trademark in the United States and several countries across the globe. See TME906. ©2024 UiPath. All rights reserved.

7. How do you evaluate
success today?
What metrics do you track?

8. 8
Proposed North Star: Estimated time saved
= Time (Number of hours) saved by having Document Understanding automations in place
where:
•no. of documents processed = total number of documents processed via automation
•x = time a user would need to process one document if automation was not in place
•validation time = total time the user spent in the Classification or Validation Station, validating all processed documents; considered 0 for
documents which did not require validation/went straight-through-processing
Example: 10mins = 2docs * 5min (manual processing) -0min validation

9. 9
Secondary Metrics
Metric Description Use Cases
No. of documents
processed
Total number of processed documents 1.How many documents have I processed
via automations?
2.How many documents have I processed
via automations in comparison to the
manual process?
Validation time Total Time spent validating the classification &
extraction results.
1.How much time is still required by a
human-in-the-loop?
2.Is validation improving with automation
and model updates?

10. 10
Secondary Metrics
Metric Description Use Cases
Average handling time Average time for processing a document
(including validation time, considered 0 for the
documents which have been straight-through-
processed).
= sum of (time spent processing documents)/no.
of docs processed
1.How much time do my employees need
to process a document, now that we have
the automation in place?
Field corrections trendTotal number of corrected fields, by field
modification (edited value, edited box, marked as
missing, etc.).
1.Are more fields modified over time?
2.Are newer model versions requiring more
human input or less? (has the model
improved?)
3.Why is validation taking so long?

11. Monitoring the numbers
Understanding the impact of your automations
The UiPath word mark, logos, and robots are registered trademarks owned by UiPath, Inc. and its affiliates.
UiPath® is a registered trademark in the United States and several countries across the globe. See TME906. ©2024 UiPath. All rights reserved.

12. 12
Project Performance Dashboard
Modern Projects > Monitor

13. 13
Because somebody needs to pay...
Insights > AI Units consumption
dashboards

14. Improving the numbers
… and taking data-informed decisions
The UiPath word mark, logos, and robots are registered trademarks owned by UiPath, Inc. and its affiliates.
UiPath® is a registered trademark in the United States and several countries across the globe. See TME906. ©2024 UiPath. All rights reserved.

15. 15
Reporting/monitoring
Identify questions you need answered
Time savings/
increased throughput
•What are my key value drivers?
•How do I deem this project
successful?
•What %of my documentsare not
approved?
•What are the top reasonsthey’re
being rejected?
•How longis it taking to validate an
exception?
•How many exceptions are due to
external influencevs. system
processing?
•What fieldsare being extracted?
•What doc typesare being
classified?
Model output
Extracted field
predictions
Doc type prediction
Automated validation
Business rule
validations
Confidence thresholds
System of record
lookups
Human validation
Average handling time
Doc type/field
validations
Exception reason
Data export
Output accuracy Model accuracy
Submission
exceptions
Cost/penalty
avoidance
Faster customer
response
•What business ruleexceptions
could be fired?
•Do business rule exceptions
have different priorities?
Q&A
Metrics &
Calculations
Log Metrics Build

16. 16
Determine metrics and calculations that
answer questions
Q&A
Metrics &
Calculations
Log Metrics Build
What are the most frequent reasons a
document is rejected?
Most Frequent Rejection
Reasons
Reason Frequency
Rejection Reason
(Extraction)
Rejection Reason
(Classification)

17. 17
Time Savings
Total Time Spent
(Automated
Process)
Number of
Documents
Manual
Intervention Rate
Number of
Documents
Processed
Filename
Number of
Exceptions
Rejection Reason
(Extraction)
Rejection Reason
(Classification)
Average Handling
Time
Number of
Documents
Validation
Duration
Validation
Duration
(Extraction)
Validation
Duration
(Classification)
Total Time Spent
(As-Is Process)
Number of
Documents
Manual
Intervention Rate
Average Handling
Time
Determine metrics and calculations that
answer questions
“How much time is my department saving each week?”
Q&A
Metrics &
Calculations
Log Metrics Build

18. 18
Document Lifecycle Dashboard –
Understanding what happened with eachtransactionin my
business process
How long did it take to process a
specific document?
False Positive
Document Type
(Pre-Validation)
Document Type
(Post-Validation)
Rejection Reason
Reviewer
(Classification)
Field Values (Pre-
Validation)
Field Values (Post-
Validation)
Reviewer
(Extraction)
Validation Time
Submission TimeNumber of Errors
Document Lifecycle
Total Processing
Time
**Not exhaustive
Q&A
Metrics &
Calculations
Log Metrics Build
Total Processing Time
Digitization Duration
Classification Duration
Extraction Duration
Validation Duration
Validation Duration
(Extraction)
Validation Duration
(Classification)
Data Export
Duration

19. 19
Business rulesConfidence scores
Generative
validation
Reducing Total Processing Time
•Primary mechanism fordeciding
when to trigger human in the loop
(HITL)
•Reduce needfor confidence
score thresholds
•Indicator of a model’s certainty
or uncertainty
•Understand impactof an
incorrect value to definefield
thresholds
•Boost confidence scores
of extracted data to reduce
human-in-the-loop requirements.
Optimize ML model
training & performance
>Reduce effort of initial
deployment
>Get direction on optimizing
performance of your models.

20. 20
Business rules when you can, and
human in the loop when you must
Basic math
Cross-reference
predictions
Total = Line items + Tax
Document fields = Email entities
No business rule
Use confidence score
External verification
Detect duplicate documents
Is human in the loop
required?
PO
Number
Invoice
Number
Status Payment
Date
Amount
Paid
5928452 5727346 paid 6/7/2022 760.00
2938572 3758292 unpaid
Subject: Please pay overdue invoice No. 5727346
From: Dylan James <[email protected]>
Created: June 29, 4:03pm

21. 21
What does confidence score look like in
practice?
Correct field
Field submitted
correctly
Field submitted
incorrectly
Review field
Incorrect
Correct
Low
confidence
High
confidence
ML
confidence %
in predicted
value
Human in the loop
Automation
Correct
Incorrect
Your
threshold
What is the impact of incorrect data?
What if we could increase the confidence
scores for these predictions?

22. 22
Minimize manual intervention
Data
destination
Document
source
Customers /
vendors
Document
specialist
Guided (efficient)
validation
1
min
Human
validation
needed
(HITL)
Automated
validation
System(s) of
record
Extract data
with ML
Business
rules
Confidence
scores
Generative
validation

23. 23
A “2
nd
opinion” to maximize your automation potential
Boosting confidence scores with
Generative validation
Predicted Value
5928452
Field Name
PO Number
Specialized
ML Model
Generative
Validation
“2
nd
opinion” on
extracted data
59%
Confidence Score
Predicted Value
5928452
Field Name
PO Number
85%
New Confidence Score
Match =
Increase
Confidence %
If predictions are equal, boost my confidence
value to X%

24. 24
Demo

25. How would you improve
the metrics you watch out for?
The UiPath word mark, logos, and robots are registered trademarks owned by UiPath, Inc. and its affiliates.
UiPath® is a registered trademark in the United States and several countries across the globe. See TME906. ©2024 UiPath. All rights reserved.

26. 26
Live Q&A

27. 27
Date/Time Topic
October 31st
9:00 AM EDT /
16:00 PM CET
Next steps
Bonus content:
https://view.highspot.com/viewer/58a3192dc5a5147b21d48b237f53e948
Join the next Dev Dives sessions:
https://bit.ly/Dev_Dives_2024
Connect with Monica and Nathan on LinkedIn:
https://www.linkedin.com/in/monica-luca-636069110
https://www.linkedin.com/in/nvpnathan/
Unlock powerful data extraction with Semantic Activities
Sign up:
AMER: https://bit.ly/Dev_Dives_October_AMER
EMEA&APJ:https://bit.ly/Dev_Dives_October_EMEA_APJ

28. Thank you
for your participation!
Join us: community.uipath.com