Proven Approaches to AI-Powered E2E Testing.pdf

Explore the evolution of AI-powered end-to-end testing and which approaches have been proven to work in this session. See how Applitools Visual AI and AI-powered testing have revolutionized functional, visual, cross-device, cross-browser, and regression testing with over a decade of training data. Session recording and more information at applitools.com.

1. Proven Approaches to
AI-Powered E2E Testing
The Applitools Intelligent Testing Platform
1
Shiran Sidis Weiss
EMEA Customer Success, Applitools
August 2024

2. 2
FINANCE
RETAIL
MEDIA
B2B TECH
HEALTHCARE
400+
Enterprise
Customers
10+ years
developing AI
algorithms
2B +
Tests ran to train
test models
200k
Community
Members
7/10 top banks
7/10 tech companies
2/3 top retailers
4/7 Top
Pharmaceuticals
Leading Media &
Telecommunication
Applitools at a glance

3. 3
Poor Test Coverage
Both manual testing and traditional test
automation suffer from validating a
finite amount of explicit scenarios in a
finite amount of time.
Slow Release Cycles
Rapidly changing UIs in today’s
multi-screen environment require a
large number of end-to-end tests that
need to be constantly rewritten.
Expensive QA Costs
To keep up with modern development,
organizations require an ever-growing
number of expensive headcount
dedicating time to testing software.
Traditional testing can’t keep up
Traditional testing
scales linearly
with more resources
2000 2005 2010 2015 2020 2025
Release
Frequency
x
Application
Complexity
x
Device
Proliferation
Increasing complexity
of software quality
Traditional test
automation cannot
meet the demands of
the business

4. More than half of engineering teams
are responsible for the development
and maintenance of automated tests.
2023 The State Of Dev Ecosystem 2022 DevOps Survey 2022: Cost Of Poor Software
58% #1 50%
PRODUCTIVITY LOSS
Testing was the #1 reason for the loss
of velocity in deployment and
launching new features.
VELOCITY LOSS
Over a 25-year life of a large software
system, almost fifty cents out of every
dollar will go to finding & fixing bugs.


INCREASED COST
Engineering & QA leaders are faced with
terrible tradeoffs

5. “Organizations are serious about automation”
67% of respondents said their software development lifecycle is mostly or completely automated

5
Roadmap
development

Roadmap development
+ Automation testing
Automation testing
development
Manual procedures Lack of coverage
2024 DevOps Survey

6. 6
The shift left approach
6
5X
10X
15X
20X
25X
30X
Design/
Architecture
Coding
Integration/
Component testing
E2E testing Production/Post-release
Relative bug fix cost per time of detectionRelative cost
Development
lifecycle

7. 7
More and more stakeholders care
about quality
Development
lifecycle
Organizational
lifecycle
Zoom out

8. 8
With an exponential growth in complexity and
scale, software teams need a new way to test
8
VOLUME
Surging amount of interfaces,
states, and scenarios to verify
COMPLEXITY
UIs are more complex and
dynamic than ever before
VELOCITY
Escalating frequency of new
features and code deploys.
RESOURCES
Limited team budgets = “do
more with less.”
Toolchain consolidation
SKILL SET
Engineers from all over the
development lifecycles are
expected to automate
ORGANIZATION
CULTURE
Rapid organization changes

9. 9
INTELLIGENT INFRASTRUCTURE
Functional
Testing
Visual
Testing
Accessibility
Testing
Parallel
Execution
Self-Healing
Locators
AUTHOR
VALIDATE
EXECUTE
ANALYZE
VISUAL AI
Automated
Maintenance
Root Cause
Analysis
Dashboards
& Reporting
Insights &
Optimizations
The Applitools Intelligent Testing Platform
The only platform with AI powering every stage of the testing lifecycle, delivering
superior testing for businesses no matter where they are on their testing journey.
PREFLIGHT
Codeless
Testing
Data
Generation
AUTONOMOUS
Web
Crawler
NLP Based
Testing
EYES SDKs
Regression
testing
Dynamic content
testing
Real device farm
Standalone
Chrome
extension

10. 10
Our Mission - Unlocking Automation Challenges
Automated test maintenance
Employs DNNs to find similar
differences across different OS,
devices and browsers (e.g.,
different fonts).
Self healing element locators
Employs algorithms and ML to
find UI elements with broken
locators based on previous
successful test runs.
Automated test data generation
Employs DNNs to detect input
forms in apps and the data types
of their fields to facilitate proper
test data generation.
Automatic test suite creation
Employs Generative AI
algorithms to generate entire test
suites for websites and
multi-page web applications.
Author tests in plain English
Employs LLMs for language
understanding and for translating
business level test instructions to
detailed UI interactions.
VisualAI: human vision modeling
Employs dozens of algorithms, and
DNNs to model color similarity, text
vs image, picture vs graphics,
pattern detection, etc.
* ML – Machine Learning, DNN – Deep Neural Network, LLM – Large Language Model
By AI powering every stage of the testing lifecycle

11. 11
INTELLIGENT INFRASTRUCTURE
Functional
Testing
Visual
Testing
Accessibility
Testing
Parallel
Execution
Self-Healing
Locators
AUTHOR
VALIDATE
EXECUTE
ANALYZE
VISUAL AI
Automated
Maintenance
Root Cause
Analysis
Dashboards
& Reporting
Insights &
Optimizations
The Applitools Intelligent Testing Platform
PREFLIGHT
Codeless
Testing
Data
Generation
AUTONOMOUS
Web
Crawler
NLP Based
Testing
EYES SDKs
Regression
testing
Dynamic content
testing
Real device farm
Coder for
non-coding
engineers
Let’s zoom in

12. Autonomous Is Here! (Live demo)

13. 13
Autonomous testing for websites and multi-page web apps
Easy one click
setup
Just point Autonomous at your
website and you are done.
Everything you need is available
out of the box.

Automatic website /
app discovery
Automatically create self-adjusting
test suites that detect new, missing,
changed or faulty pages and
components
Natural language
test builder
Describe complex end-to-end flows
using nothing more than plain English.
No coding or element locating skills
required
Cross device and
browser testing
Test your public and internal apps on
any device, browser and OS using
the world’s most modern test
infrastructure available out of the box
Flexible test
orchestration
Run tests on demand, from your
CI/CD, a webhook, or use our
built-in test scheduler. No DevOps
skills required
Self-heal broken locators, avoid
repetitive maintenance activities
and group together similar UI
changes and issues
AI assisted test
maintenance
13
vision of Autonomous Testing platform (I)

14. 14
14
AUTHOR
Seamlessly switch between recording test steps in real time to
writing them in plain English in the same flow.
INTERACTIVE TEST AUTHORING
Automatically discover new, changed, missing or faulty pages
and components on your website or multi-page web app
WEB CRAWLER
Build custom end-to-end tests using nothing more than plain
English
NLP BUILDER
Reduce the time it takes to build automated tests
Record your actions with a powerful web recorder to build
complex end-to-end tests without coding
UI RECORDER

15. 15
Use Visual AI to validate hundreds of elements across
different browsers instantly with a single line of code
AUTOMATED VALIDATION
Use Visual AI to easily test dynamic, content-heavy
websites without having to rewrite and maintain tests.
DYNAMIC ASSERTIONS
Increase test coverage with AI assertions
FUNCTIONAL DATA-DRIVEN TESTING
Easily perform data-driven and dynamic data testing
with variables, parameters, and non-visual assertions.
Validate

16. 16
Increase release velocity with faster, resilient tests
Run tests in parallel at massive scale on ultrafast test
infrastructure in the cloud.

FAST INFRASTRUCTURE
Automatically heal element locators that break your
pipelines even for the smallest UI change.

SELF-HEALING TESTS
Run tests on demand, from your CI/CD, a webhook, or
use our built-in test scheduler.
FLEXIBLE ORCHESTRATION
Execute

17. 17
Get deep insights into every test run
Get comprehensive reports and insights for all your test
runs and easily search and filter test results


INSIGHTFUL REPORTS
Quickly surface what code changes caused each UI
change or issue with Root Cause Analysis

PINPOINT WHAT WENT WRONG
Automatically group together similar changes and
issues to help accelerate test maintenance and analysis

AUTOMATED MAINTENANCE
Analyze

18. How can manual tester write code?
18
Roadmap
development

Roadmap development
+ Automation testing
Automation testing
development
Manual procedures Lack of coverage
Let’s zoom in

19. 19
TestGenAI for Cypress
Record your actions with a powerful web recorder to build
complex end-to-end tests and export them to Cypress.
UI RECORDER
Generate more reliable tests that require less time to maintain
with integrated self-healing.
SELF-HEALING TESTS
Generate synthetic data like names, emails, addresses and more
and then reuse them throughout tests
DATA GENERATION
Automatically generate tests with Industry best practices built in.
INDUSTRY BEST PRACTICES
Reduce the time it takes to build high quality
Cypress tests.

20. 20
Bonus! POM Generator

21. 21
INTELLIGENT INFRASTRUCTURE
Functional
Testing
Visual
Testing
Accessibility
Testing
Parallel
Execution
Self-Healing
Locators
AUTHOR
VALIDATE
EXECUTE
ANALYZE
VISUAL AI
Automated
Maintenance
Root Cause
Analysis
Dashboards
& Reporting
Insights &
Optimizations
The Applitools Intelligent Testing Platform
PREFLIGHT
Codeless
Testing
Data
Generation
AUTONOMOUS
Web
Crawler
NLP Based
Testing
EYES SDKs
Regression
testing
Dynamic content
testing
Real device farm
Coder for
non-coding
engineers
Let’s zoom in

22. 22 22
Brittle assertions & locators need constant maintenance
and lack proper coverage.
Limited Coverage, and Maintenance never ends..

23. 23
Protect your brand

24. 24
Protect your brand

25. 25
False positives are exhausting

26. 26
~20% false positive reduction

27. 27
What about Dynamic content?

28. 28
Here you go!

29. 29
INTELLIGENT INFRASTRUCTURE
Functional
Testing
Visual
Testing
Accessibility
Testing
Parallel
Execution
Self-Healing
Locators
AUTHOR
VALIDATE
EXECUTE
ANALYZE
VISUAL AI
Automated
Maintenance
Root Cause
Analysis
Dashboards
& Reporting
Insights &
Optimizations
The Applitools Intelligent Testing Platform
PREFLIGHT
Codeless
Testing
Data
Generation
AUTONOMOUS
Web
Crawler
NLP Based
Testing
EYES SDKs
Regression
testing
Dynamic content
testing
Real device farm
Coder for
non-coding
engineers
Let’s zoom in

30. 30
Visual AI accelerates
cross browser/device testing
Capture the entire UI with
a single snapshot
DOM & CSS sent to
Applitools
Ultrafast Test Cloud
Applitools Eyes
Visual AI analysis
Parallel DOM
Rendering
.
Firewall
18.2x faster than traditional cloud execution grids.
Applitools
Ultrafast Test Cloud
HTML CSS

31. 31
We know how to heal your test

32. Q&A
32

33. 33
Root cause analysis