Experimental Performance Evaluation of RPA Bots

Recent years have seen the ascendance of Robotic Process Automation (RPA), offering significant potential for process automation even to non-IT specialists, termed as citizen developers. This technology has repeatedly demonstrated its capacity to deliver time savings, quality enhancements, and cost reductions when applied to appropriately chosen processes. Citizen developers, equipped with a range of tools and software libraries, are central to implementing RPA bots. Our research involved conducting an experiment to explore how varying tools and libraries impact the performance metrics of bot runtime and result quality. Utilizing a typical RPA use case, we observed and logged the execution of these bots in a controlled environment. The findings of our experiment reveal substantial disparities, primarily attributable to the performance of Optical Character Recognition (OCR) libraries. Based on these insights, we propose strategic recommendations for bot development and the future-oriented restructuring of business processes.

1. Experimental Performance
Evaluation of RPA Bots
Prof. Dr. Michael Groeschel
Mannheim University ofApplied Sciences, Germany
[email protected]
12.09.2024
Paper #313
1

2. Abstract
•Recent years have seen the ascendance of Robotic Process Automation
(RPA), offering significant potential for process automation even to non-IT
specialists, termed as citizen developers. This technology has repeatedly
demonstrated its capacity to deliver time savings, quality enhancements,
and cost reductions when applied to appropriately chosen processes.
Citizen developers, equipped with a range of tools and software libraries,
are central to implementing RPA bots. Our research involved conducting
an experiment to explore how varying tools and libraries impact the
performance metrics of bot runtime and result quality. Utilizing a typical
RPA use case, we observed and logged the execution of these bots in a
controlled environment. The findings of our experiment reveal substantial
disparities, primarily attributable to the performance of Optical Character
Recognition (OCR) libraries. Based on these insights, we propose
strategic recommendations for bot development and the future-oriented
restructuring of business processes.
2

3. Agenda
•RPA: Robotic Process Automation
•Research Question and Approach
•Related Work
•Our Experiment: Idea, Realization, Measurement
•Restrictions and Recommendations
•References
3

4. RPA: Robotic Process Automation
•Software Robots
•Allows to automate repetitive tasks, traditionally performed by
humans
•Goals: Increased efficiency, reduced labor costs
•Increasing popularity over the last years
•Citizen Developer should create their own bots
•Non-IT-trained employees
•User-friendly tools
•Pre-built components
•Low-code/No-code-Approach
4

5. RPA: Robotic Process Automation
•Danger of unmanaged shadow IT
•Governance needed
•Hard to measure efficiency and
effectiveness
•Performance of tools may differ
•Tool market still in development
•UiPath and Microsoft are leaders
5

6. Research Question and Approach
•What differences exist in terms of runtime performance and
quality with different RPA tools and different approaches?
•Conducting an experiment that implements a real-life scenario
several times and analyzes differences
6

7. Related Work
•Comparison of tools based on features [2, 7]
•Focus on functionality [8, 7, 17, 18, 4]
•Identification of meaningful use cases [1]
•Quality measurement and performance comparison largely
missing
•Bad performance decreases productivity, potential overload of the IT
systems
•Good performance saves system resources
7

8. Our Experiment
•Real-life scenario
8

9. Our Experiment: Setup and Projects
•Environment: Standard-PC
•119 equal-structured documents
•Tools: 2 Market leaders
•4 Projects
•1: UiPath‘Read PDF Text‘-Activity
•2: UiPath‘Read PDF With OCR’-Activity, Microsoft OCR Engine
•3: UiPath‘Read PDF With OCR’-Activity, Tesseract OCR Engine
•4: Microsoft Power Automate Desktop
•Ten-fold execution of the run with time measurement
9

10. Our Experiment: Results
•4 Projects
•1: UiPath‘Read PDF Text‘-Activity
•2: UiPath‘Read PDF With OCR’-Activity,
Microsoft OCR Engine
•3: UiPath‘Read PDF With OCR’-Activity,
Tesseract OCR Engine
•4: Microsoft Power Automate Desktop
•Very different results
•Fastest runtimes comparable to manual
programming
•Quality without OCR perfect
•OCR significantly slower (as expected)
•OCR quality often poor
10

11. Restrictions and Recommendations
•Restrictions
•All documents have the same format
•No AI used to improve the quality (but requires training effort)
•Recommendations
•Think about the process first!
•Provide templates, if possible
•Provide training for the citizen developers
•Keep in mind: RPA Development is software development
•Establish a RPA Governance
11

12. References
12