Streamlining End-to-End Testing Automation

Streamlining End-to-End Testing Automation with Azure DevOps Build & Release Pipelines

Automating end-to-end (e2e) test for Android and iOS native apps, and web apps, within Azure build and release pipelines, poses several challenges. This session dives into the key challenges and the repeatable solutions implemented across multiple teams at a leading Indian telecom disruptor, renowned for its affordable 4G/5G services, digital platforms, and broadband connectivity.

Challenge #1. Ensuring Test Environment Consistency: Establishing a standardized test execution environment across hundreds of Azure DevOps agents is crucial for achieving dependable testing results. This uniformity must seamlessly span from Build pipelines to various stages of the Release pipeline.

Challenge #2. Coordinated Test Execution Across Environments: Executing distinct subsets of tests using the same automation framework across diverse environments, such as the build pipeline and specific stages of the Release Pipeline, demands flexible and cohesive approaches.

Challenge #3. Testing on Linux-based Azure DevOps Agents: Conducting tests, particularly for web and native apps, on Azure DevOps Linux agents lacking browser or device connectivity presents specific challenges in attaining thorough testing coverage.

This session delves into how these challenges were addressed through:

1. Automate the setup of essential dependencies to ensure a consistent testing environment.

2. Create standardized templates for executing API tests, API workflow tests, and end-to-end tests in the Build pipeline, streamlining the testing process.

3. Implement task groups in Release pipeline stages to facilitate the execution of tests, ensuring consistency and efficiency across deployment phases.

4. Deploy browsers within Docker containers for web application testing, enhancing portability and scalability of testing environments.

5. Leverage diverse device farms dedicated to Android, iOS, and browser testing to cover a wide range of platforms and devices.

6. Integrate AI technology, such as Applitools Visual AI and Ultrafast Grid, to automate test execution and validation, improving accuracy and efficiency.

7. Utilize AI/ML-powered central test automation reporting server through platforms like reportportal.io, providing consolidated and real-time insights into test performance and issues.

These solutions not only facilitate comprehensive testing across platforms but also promote the principles of shift-left testing, enabling early feedback, implementing quality gates, and ensuring repeatability. By adopting these techniques, teams can effectively automate and execute tests, accelerating software delivery while upholding high-quality standards across Android, iOS, and web applications.

1. @BagmarAnand
Streamlining
End-to-End
Test Automation
Anand Bagmar
Software Quality Evangelist

2. @BagmarAnand
Anand Bagmar@BagmarAnand
About Me

3. Ground
Reality
Distributed teams
Hybrid working
Different network setup and speeds
Many teams
(>100)
Mac, Windows and Linux laptops
Different Software versions
Certificates, Policies and multiple VPNs
Inconsistent
Developer &
SDET experience
Test execution environment setup is tedious
System Tests (e2e & component UI): teswiz (Appium, Selenium, Applitools, JDK 17)
•Emulator/Simulator setup (Android SDK, xcode)
API & API Workflow tests: karate
Contract tests: Specmatic
Unit tests, Sonar code quality checks
Test automation
toolset
Many environments
Test data
Branch
Configuring appropriate test execution in build and release pipelines
Complex path to
production
ADO agents: Windows Server & Linux agents
Firewall restrictions download dependencies
Direct access prohibited to CI agents
Multiple node & JDK versions
Connectivity issues to application-under-test
No browsers/devices on CI agents
CI execution
@BagmarAnand

4. Path to Production
@BagmarAnand

5. Getting a simple automated
test to run consistently for
all Developers and Testers
and in CI (ADO) is painful!
Setup
Execution (full or specific tests)
@BagmarAnand

6. Challenges of
End-2-End
Test
Automation
Ensuring Test
Environment Consistency
Coordinated Test
Execution
Test setup & execution on
CI Agents
@BagmarAnand

7. Solutions
Implemented
@BagmarAnand

8. Consistent
Environment Setup
#1
@BagmarAnand

9. •Setup important applications on Mac
•https://gist.github.com/anandbagmar/92b9f92298b1e17fa32c3404ad115
871
•Script to setup Android SDK on Mac
•https://github.com/anandbagmar/AppiumJavaSample/blob/master/setup
AndroidSDK.sh
•Script to setup Android SDK on Linux
•https://github.com/anandbagmar/AppiumJavaSample/blob/master/setup
_linux.sh
Test Authoring Environment Setup
@BagmarAnand

10. •Node script to install all dependencies (for system tests):
•https://github.com/znsio/getting-started-with-
teswiz/blob/main/package.json
•npm install – and you are ready!
Test Execution Environment Setup
@BagmarAnand

11. Test Automation
Framework support
#2
@BagmarAnand

12. •Setup should be simple – Ex:
•git pull
•./gradlew build
•No code change required for
•Running tests against any environment (local, dev, qa, staging, prod, etc.)
•Test data and environment configurations are separately maintained
•Running all or subset of tests
•Tests should run from command-line
Test Automation Framework Criteria
@BagmarAnand

13. Your Ultimate Open-Source Solution to
Automate Real-User Scenarios!
@BagmarAnand

14. Architecture
@BagmarAnand

15. Run tests
from CLITest Authoring
Execution Setup
1 2
2
3
4
5
6
6
Execution Reports
CI Tool
Feature coverage
@BagmarAnand 16

16. •Web browsers
•Mobile-web browsers
•Android apps
•iOS apps
•Windows desktop apps
•Electron apps
Platform support
@BagmarAnand

17. •Open source framework to automate real-user scenarios
•Multi-user
•Multi-device
•Multi-app
•Setup a HARD-GATE for your functional tests!
Unique capabilities of teswiz
@BagmarAnand

18. •Cloud device farm integrations
•Applitools AI for validations
•Comprehensive reports with trend analysis, feature
coverage, failure analysis using AI-ML
•CLI
•Configurable
Unique capabilities of teswiz
Defaults
Property
files
Environment
Variables
@BagmarAnand

19. CI Execution
@BagmarAnand

20. Node Setup
#3
@BagmarAnand

21. Use the right node version
@BagmarAnand

22. Use the right node version
@BagmarAnand

23. Script for
downloading artifacts
#4
@BagmarAnand

24. •For System Tests, the artifact (apk/app) could have been
generated from another pipeline
•This artifact needs to be available in local or cloud device
before tests can start execution
Script for downloading artifacts – Why?
@BagmarAnand

25. •Understand the CI tool APIs
•Script downloads the android/iOS artifact for:
•Specific branch
•Latest successful build, or a specific build number
Script for downloading artifacts
@BagmarAnand

26. •Script uploads the android/iOS artifact to your device farm
OR
•Teswiz can upload it automatically for you
Script for downloading artifacts – Bonus!
@BagmarAnand

27. Proxy handling
#5
@BagmarAnand

28. •Understand what dependencies in your framework need proxy
information. Ex:
•Gradle/maven
•Downloading newer version of browser drivers
•Any external connectivity
•Framework should be configurable to pass this at test execution
time. Ex:
•No proxy required from local laptop execution
•Proxy required when running tests from CI
Proxy Handling
@BagmarAnand

29. @BagmarAnand

30. updateGradlePropertiesForDevOps.sh
@BagmarAnand

31. Downloading
dependencies - Uber jar
#6
@BagmarAnand

32. •To reduce the number of dependencies to be downloaded,
teswiz is built as a uber jar.
•Specify only “teswiz” as a dependency in your test framework
Uber jar
@BagmarAnand

33. •Run as a java process
•./gradlew run
Uber jar
@BagmarAnand

34. Run browser in
docker
#7
@BagmarAnand

35. •CI agents may not have browsers installed
•The installed browser may be an older version
Run browser in docker – Why?
@BagmarAnand

36. •Should allow support for any os/architecture
•Should allow choosing the browser (ex: firefox, chrome, etc.)
•Should allow starting the containers with specific project
names and dynamic ports to prevent conflict with multiple
test executions
•Support specifying proxy information
•Can be used on local laptops as well as in CI executions
Run browser in docker
@BagmarAnand
https://github.com/znsio/teswiz/blob/main/dockerContainers.sh
https://github.com/znsio/teswiz/blob/main/docker-compose-v3.yml

37. Template for running
tests in build pipeline
#8
@BagmarAnand

38. Path to Production
@BagmarAnand

39. •Create templates
•Reuse with appropriate configuration parameters
Running tests in Build Pipeline
@BagmarAnand

40. @BagmarAnand

41. Task Groups for running
tests in release pipeline
#9
@BagmarAnand

42. Path to Production
@BagmarAnand

43. •Create Task Groups
•Include in each relevant stage of Release pipeline
Running tests in Release Pipeline
@BagmarAnand

44. @BagmarAnand

45. @BagmarAnand

46. @BagmarAnand

47. @BagmarAnand

48. @BagmarAnand

49. @BagmarAnand

50. Hard Gate
Make your tests valuable!
#10
@BagmarAnand

51. @BagmarAnand
What is a Hard Gate! Why is it required?
https://github.com/znsio/teswiz/blob/main/docs/HardGate.md
•Automated tests should allow you to take decisions on
product quality

52. @BagmarAnand
What is a Hard Gate! Why is it required?
https://github.com/znsio/teswiz/blob/main/docs/HardGate.md
•For every test execution cycle:
•Passing tests are expected to pass
•Known Failing tests are supposed to fail, unless:
•The product (bug) is fixed, OR
•The test is fixed/updated
If either criteria is not met, the build should fail!

53. @BagmarAnand
Hard Gate - Make your tests valuable!
https://github.com/znsio/teswiz/blob/main/docs/HardGate.md
Build passes if Hard Gate criteria is met.
Build fails if
•one or more passing tests have failed, or,
•one or more failing tests have passed

54. @BagmarAnand
Functional (e2e) Automation as Hard Gate!
https://github.com/znsio/teswiz/blob/main/docs/HardGate.md

55. Applitools Visual AI
for validations
#11
@BagmarAnand

56. AI-powered Validations
@BagmarAnand

57. 90% less code to write & maintain with infinite coverage.
WITH APPLITOOLS AI
Every Element Is Validated They Look & Work Perfect
Use Applitools Ultrafast Grid (UFG)
-Test is simpler – one call to Applitools
(eyes.checkWindow()) validates the full screen
-Run the test once
-Get results from all browsers automatically
-Less test data
-No additional load on the application environment
@BagmarAnand

58. •Works for all platforms
•Native & hybrid apps – android, iOS
•Web browsers
•Desktop applications
•Electron applications
•Seamless scaling using Applitools Ultrafast Grid
@BagmarAnand
Applitools Visual AI

59. Specify as many browsers with viewports and devices
as required for validation
You do not need to do cross-browser validation at the
end anymore!
@BagmarAnand
AI-powered Cross Browser Test Automation

60. reportportal as a
Central reporting server
#12
@BagmarAnand

61. @BagmarAnand

62. A central reporting server for your
organization
@BagmarAnand

63. Test Execution
Real-time status
•See progress of launches
currently in progress
•Can also see details of tests
that are currently running, till
the point of execution
@BagmarAnand

64. Test Execution Details – Device farm report link & Device logs
•The link to the device farm test execution
dashboard is available in the result
•teswiz attaches browser logs/device logs
automatically to the result in ReportPortal
@BagmarAnand

65. Test Execution Details – with screenshots
•The test result includes screenshots as
captured by the test
@BagmarAnand

66. Test Execution Details – Applitools Visual AI Validation Results
•The test result includes the status of
Applitools Visual AI validation
•Link to the Applitools dashboard is available in
the result
@BagmarAnand

67. Test Execution Trend Analysis
•Each test shows the trend of its execution –
giving an indication of (in)stability
@BagmarAnand

68. Test Results – Next Steps
•On investigation of the failed tests, mark the
failures with appropriate reasons (as
configured)
@BagmarAnand

69. Auto-analysis of failed tests
Analyse the failure reasons by Auto-Analyzer based on Machine Learning
@BagmarAnand

70. Auto Analysis of Test Failures
•Why waste time
marking the test
failed for the same
reason as last time?
•ReportPortal can do
this automatically
for you with the
Auto Analysis and
Pattern Analysis
feature
@BagmarAnand

71. Auto Analysis of Test Failures
@BagmarAnand

72. Test Result
Visualization
Configure simple and
understandable
reports
•Create as many dashboards as
relevant for the team
•Dashboards may be for different
persona/role, giving appropriate
information
@BagmarAnand

73. •Teswiz and karate test frameworks can automatically upload
test results to your reportportal server
•sendToReportPortal:
•https://github.com/znsio/sendToReportPortal/blob/main/importRes
ultsAndUpdateAttributes.sh
•Can upload junit test results generated by any type of tests to
reportportal with relevant test execution metadata
reportportal.io
@BagmarAnand

74. Challenges
•Ensuring Test
Environment Consistency
•Coordinated Test
Execution
•Test setup & execution on
CI Agents
Solutions
Consistent environment setup
Test Automation Framework support
Node setup
Script for downloading artifacts
Proxy Handling
Downloading dependencies - Uber Jar
Browsers in docker
Template for build pipelines
Task groups for release pipelines
Hard Gate
AI for validations
Central reporting server
Summary
@BagmarAnand

75. @BagmarAnand
Anand Bagmar@BagmarAnand
Thank you