AI Assisted Continuous Testing - Talk Track v2.pdf

 In my talk today I will explain how to transform an organization that is currently using "Continuous Testing" as part of their DevSecOps solution to "AI-Assisted Continuous Testing".
I will cover:
What AI-Assisted Continuous Testing is.
I will compare AI Assisted Testing to existing Continuous Testing.
I will highlight some of key challenges that I found during different types of transformation projects fusng A I enabled tools.
and
I will explain some tools and transformation practices that I find useful to ensure success with AI-Assisted Continuous Test transformations for DevSecOps.
Finally I will conclude with my view of benefits, and a radical prediction for the future of DevSecOps enabled by AI-Assisted Continuous Testing.

2. •AI-Assisted Continuous Testing
•Comparison to existing Continuous Testing
•Challenges for transformation
•Transformation practices and tools
•Benefits
•Future Prediction
DALL-E generated image
Navigating to AI-Assisted Continuous Testing

3. Continuous Testing Blueprint
A strategy designed to reduce
lead times and failure rates in
continuous delivery pipelines and
in continuous operations, through
automated and iterative testing
processes, aiming for decreased
time from code commit to
production deployment and
reduced failures in production.
CT is much more than simply
“automating tests”.
Continuous Testing
Predicted
“Autonomous Continuous
Improvement”.

4. Test
Management
Test
Tools
Test
Automation
Leadership
and Culture
Continuous Testing Blueprint
Test Strategies and
Plans
Test Results
Analysis
Test Environment
Management
Continuous Testing
Tenets
Continuous Testing Blueprint
Artifacts
Development (Dev)
Artifacts Artifacts Artifacts Artifacts
Integration (CI) Delivery (CD)
Deployment (Deploy)Production (Prod)Planning (Plan)
Test Data
Management
“AI can assist every aspect of Continuous Testing. “

5. AI/ML Game-Changer for Testing
✓Software analysis
✓Test plan and script generation
✓Test data generation
✓Defect & vulnerabilities
detection / correction
✓Test optimization
✓Test maintenance
✓Report generation &analysis
✓Test scheduling and
management
✓Testability
“Time from Test Strategy to Test Results reduced to hours instead of days.”
“AI-Assist”

6. “Effectiveness of AI depends on the wisdom and skills of domain experts”
Deep Domain Knowledge
•Contextual and subject matter expertise
•Create templates and select tools
•Innovation and strategic planning
•Knows how to leverage AI
•Training and guiding AI models
•Bridge gaps between technology and
applications
•Ensure compliance and ethical standards
•Continual learning and model improvement
•Risk assessment and management
•Training and development
•Facilitate collaboration
AI / ML Agents Capabilities
•Specialist AI Agent roles
•Ingest guidance and use cases
•Identify information from adjacent domains
•Accumulate and organize information
▪Analyze dat and pattern recognition
▪Predictive analytics
▪Language, visual and audio processing
▪Machine Learning for Cybersecurity
▪Automated problem solving
▪Customization and personalization
▪Process automation
▪Real-time monitoring and alerts
Roles with AI-Assisted Continuous Testing

7. Deployment
Tests
Delivery
Testing
Integration
Testing
Dev
Test
•Test Strategies
•Test Plans
•Test Cases
•Test Scripts
•Test Notation
Definitions
•Test Environment
Notation
•Test Results
•Test Metrics and
Analytics
Version-
Managed
Templates and
Artifacts
AI Results and Analysis
AI-
Assisted
Test
Workflow
AI Test Environment Orchestration
and
Test Automation
AI-Assisted Continuous Testing Blueprint

8. CodeThreat - AI SAST
Keploy – Data to Test Generator
Examples of AI-Enabled Testing Tools
*Testbytes, Feb,2024
* https://www.linkedin.com/pulse/10-ai-tools-software-testing-you-should-know-softwaretestingcompany-uaxvc/
Refraction – Coding and
Scanning
Synthetic Users – User testing
UFT One – Function test
automation
Aqua ALM – test management
Eggplant – visual element
testing
Katalon – Regression test
generation
ACCELQ – zero code test
automation
Perfecto Scriptless – no
code generation
Tricentis Tosca – test
generation
Functionize – test generation Sapient.ai – unit test generation
There is no universal AI-
tool solution for
DevSecOps.

9. Traditional
Hours –
Days
AI-Assist
Seconds -
Minutes
1.Requirements Analysis
2.Test Strategy
3.Test Plans
4.Test cases
5.Test scripts
6.Test environments
7.Test campaigns
8.Test Executions
9.Test Reports
10.Test Reviews
11.Test Remedies
12.Test Regress
Macro-Flow
Steps:
Continuous Test Workflows
Savings (Reduction)
Lead Time: 50 - 90%
Costs: 70 - 95%
…
Micro-Flow Steps

10. Technology Transformations
•AI Integration
•Data Infrastructure
•Tool Modernization
Procedural Transformations
•Test creation and execution
•Data-driven decisions
•Feedback loops
Organizational Transformations
•Skill development
•Team structures
•Culture shifts
Quality Assurance Transformations
•Enhanced coverage and accuracy
•Continuous improvement
•Compliance and security
Transformation Use Cases

11. What Can Go Wrong?
×Consistency
×Readability
×Understandability
×Relevance
×Environment
×Controllability
×Observability
×Ethical and Bias
×Intuition
×Repeatability
×Coverage
×Compatibility
×Acceptance
×Data Quality
×Continuous Learning
A strategic transformation approach is needed !

12. Strategic Goals
•Model
Applications
•Maturity Model
•Goals
•Topics
•Priorities
Strategic Assessment
•Discovery Survey
•Gap Assessment
•Current state VSM
•Baseline metrics
Strategic Plan
•Future state VSM
•Themes, Epics, Stories,
Tasks
•Tools selection
•Dependencies
•Timeline
•Governance
Implementations
•Pipeline
•Migrations
•Integration
•Dashboard
•Governance
•Checkpoint
Assessments
5,6,7
REALIZE,
OPERATIONALIZE
EXPAND
4
SOLUTION
ALIGNMENT
3
ASSESSMENT
1,2
VISION/GOAL
ALIGNMENT
•Discovery Surveys
•Value Stream tools
•Topics and Practices
•Metrics and
Analytics
•Deliverable
templates
AI-Assisted
Tools & Templates
AI-assisted Analysis and Results Alignment
Transformation
Consultant
Process Leadership and
Collaboration
AI-Assisted CT Strategic Transformation Blueprint

13. Example Strategic Transformation Roadmap
Theme 1
Model AI Platform MVP
--------------
•Test environment,
generation, and
automation
•Templates & Tools
•Selected test types
•Metrics
•Workflows
•Training
•Retrospective
Theme 2
Consolidation
--------------
•Test Results side
•Standard pipelines
•Migrate other application
•Center of Excellence
•Guilds and Training
•Re-assessment
Theme 3
Coverage Expansion
--------------
•Types of tests
•Advance Training
•More applications
•Re-assessment
•Release auomation
Theme 4
Optimization
--------------
•Deployment
automation
•Accelerate test
creation
•Accelerate test
execution testing
•Accelerate test
analysis
•Continuous
Experimentation
1 2 3 4
Every transformation needs it own strategic roadmap!

14. Transformation Implementation Patterns that Work
Dedicated Platform Team Embedded Team
Outsourced Team
Hybrid Dedicated/Outsource Team

15. Transformation Patterns to Avoid

16. Motivating Continuous Testing Transformations
•Educate and Align
•Demonstrate Value
•Develop Skills and Infrastructure
•Cultural Shift Toward Testing Automation
•Incorporate Feedback Mechanisms
•Adapt Project Management and Processes
•Encourage Innovation
•Highlight ROI and Performance Metrics
•Strengthen Vendor Partnerships
•Manage Change Effectively

17. Measuring Transformation Success
DORA Metrics:
•Deployment Frequency (DF)
•Lead Time for Changes (LT)
•Change Failure Rate (CFR)
•Time to Restore Service (TRS)
Additional Metrics:
•Test Coverage
•Defect Detection Efficiency
•Automated Test Pass Rate
•Flakiness Score
•Test Execution Time
•Cost per Defect Found
•Feedback Loop Duration
•ROI of Testing

18. Measuring Transformation ROI
Costs:
•Development/Integration Costs
•Software Costs
•Hardware Costs
•Training Costs
•Maintenance Costs
Benefits:
•Increased Efficiency
•Reduced Labor Costs
•Improved Quality
•Enhanced Test Coverage
Intangible Benefits:
•Improved Team Morale and Productivity
•Enhanced Market Reputation

19. Benefits of Continuous Testing
✓On-time delivery
✓Improved predictability
✓Reduced cost of quality
✓Improved customer experience
✓Improved productivity
✓Better quality
✓Faster releases
✓Fewer security events
✓Improved compliance scores
“up to 70% efficiency gains due to the reduction of wait-time and
toil” - Without AI !
What will it be with AI-Assist ?
!

20. The Future – “DevInOps”
No Shift, No Lead Time
Enablers
•Observability
•AI-Automation
•Feature Flag Rollouts
•“No-Shift”
Faster Lead time
DevSecOps
Continuous Delivery
Shift-Left
Production
Development
in Production
“At some point smart automation will make it so there will be
nothing Left to shift! ”

21. AI-Assisted:
DevSecOps and SRE Transformations
Continuous Testing Consulting
EngineeringDevOps.com
[email protected]