ovum-real-app-testing-presentation--133465.pdf

Ovum

1. © Copyright Ovum 2007 www.ovum.com
Does Your Business Need Oracle 11g's Real
Application Testing: Findings from Ovum research
David Mitchell
Senior Vice President, IT Research, Ovum
[email protected]
[phone]

2. © Copyright Ovum 20072
Who is Ovum
An independent research analyst and advisory
firm, headquartered in London but operating globally
Founded over 20 years ago
Operating across all aspects of software, IT services, and
telecoms
Providing advice to enterprises, vendors and the investment
community
Part of the Informa and Datamonitor group of companies, FTSE
listed

3. © Copyright Ovum 20073
Agenda
•Oracle Real
Application
Testing study
•Conclusions
and benefits
•Current
practices and
trends in
testing
•Business
context
12
34

4. © Copyright Ovum 20074
•Oracle Real
Application
Testing study
•Conclusions
and benefits
•Current
practices and
trends in
testing
•Business
context
12
34

5. © Copyright Ovum 20075
Stimulus of the macro environment
Globalization
New
competitors
New
partners
Volatility
Market
fluctuation
Rapid M&A
moves
Technology
Web 2.0
enterprises
Shorter IP
uniqueness
timescales

6. © Copyright Ovum 20076
Response to the macro environment
Customer Intimacy
Product Innovation
Operational Excellence
Customer Intimacy
Business Agility
Operational Efficiency
Product Innovation
After Treacy and Weirsema , 1997, “The Discipline of Market Leaders”

7. © Copyright Ovum 20077
Stimulus of the meso environment
Security alerts
Software defects
Compliance
Business Growth
Support
Rapid response
needed
Rapid response
needed
Rapid response
needed
Rapid response
needed
Rapid response
needed

8. © Copyright Ovum 20078
•Oracle Real
Application
Testing study
•Conclusions
and benefits
•Current
practices and
trends in
testing
•Business
context
12
34

9. © Copyright Ovum 20079
Current practices
Testing takes a long time and is risky
Examples
Major international bank –18-24 month change control cycle
Publishing firm –without critical system for 2-3 months, testing forgot
major sub-system and no regression planning
Utility –spent nearly $30m on testing but still had major customer
billing system problems
Major technical problems still occur e.g. Skype

10. © Copyright Ovum 200710
Current practices –always squeezed

11. © Copyright Ovum 200711
Current practices –V-model
The V-shape model: stage and layers
Requirement
Code
Specification
Design
Unit Test
Integration
SystemTest
Acceptance

12. © Copyright Ovum 200712
Current practices –V-model, modified for
two new threads
Userstories
Code and unit
test
Integrate
Acceptance
test
The V-shape model: for Agile development methods
Prepare acceptance
tests
Prepare unit
tests

13. © Copyright Ovum 200713
Current practices –V-model, modified for
two new threads
For requirements –driven
testing
Requirement
Code
Specification
Design
Unit Test
Integration
SystemTest
Acceptance
The V-shape model: requirements driven testing
Prepare
tests
Prepare
tests
Prepare
tests

14. © Copyright Ovum 200714
IT Business
Management
Application Delivery
Support Services
Current practices –moving to ALM
CollaborationWorkflow
Project
Management
Configuration
Management
Defect
Tracking
Reporting
Project
Portfolio
management
Requirements
capture and
management
Sourcing
Change
management
Architecture
management
Planning &
estimating
Release
management
Quality
management
Monitoring &
Reporting
Architecture &
Design
Code Build Process Test Deploy
Architecture & Design

15. © Copyright Ovum 200715
Current practices –moving to ALM
The implications
IT Business
Management
Application Delivery
Support Services
Collaboration Workflow
Project
Management
Configuration
Management
Defect
Tracking
Reporting
Project
Portfolio
management
Requirements
capture and
management
Sourcing
Change
management
Architecture
management
Planning &
estimating
Release
management
Quality
management
Monitoring &
Reporting
Architec
ture &
Design
Code
Build
Process
Test Deploy
Architecture & Design
Testing integrated
into more stages
of IT lifecycle
More focus and
spend on testing
ITIL entry -move
from art to
science/engineeri
ng of IT
All IT
professionals
need to focus on
testing –not just
testers

16. © Copyright Ovum 200716
Current practices
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
200620072008200920102011
6 7 8 8 9 9
9 10 11 11 12 12
37 33 29 27 24 23
26 28 29 30 30 30
22 22 23 24 25 26
Management
QA
Develop
Design
Definition
Proportion of IT spend on different phases of the ALM lifecycle

17. © Copyright Ovum 200717
Current practices –system testing tasks
Requirement
Code
Specification
Design
Unit Test
Integration
SystemTest
Acceptance
Installation
testing
Backup and
recovery
testing
Stress and
volume
testing
Operational
process
testing
Other
system
testing

18. © Copyright Ovum 200718
Current practices–load testing
Capture
samples of
observed
workload
Create larger
simulated
workload
Execute
workload in
changed
environment
with load
testing tool
Compare
performance

19. © Copyright Ovum 200719
Current practices–load testing critical
flaws
Workload based
on small
samples, 1-3% is
common
Workload doesn’t
represent
monthly
transactions
Scaling doesn’t
produce
representative
concurrency
Creation and
maintenance of
ad hoc tools

20. © Copyright Ovum 200720
Current practices–load testing critical
flaws
Addressed by
the capabilities in
Oracle Real Application Testing
Workload based
on small
samples, 1-3% is
common
Workload doesn’t
represent
monthly
transactions
Scaling doesn’t
produce
representative
concurrency
Creation and
maintenance of
ad hoc tools

21. © Copyright Ovum 200721
•Oracle Real
Application
Testing study
•Conclusions
and benefits
•Current
practices and
trends in
testing
•Business
context
12
34

22. © Copyright Ovum 200722
Oracle Real Application Testing
Database Replay
Use cases
•Database upgrades, patches, schema changes, etc..
•Configuration changes -e.g. RAC or ASM moves
•Environment changes e.g. OS, hardware, network

23. © Copyright Ovum 200723
Oracle Real Application Testing
Database Replay
Functionality
•Workload capture –all database calls, configurable start/end time
•Workload processing –transforms capture files and generates
metadata to enable reply
•Workload replay –via replay client, identical timing and
concurrency
•Analysis and reporting tools

24. © Copyright Ovum 200724
Oracle Real Application Testing
Database Replay
Comparisons with traditional methods
•All database calls not sample
•True timing and concurrency
•Replay is client agnostic and more flexible e.g. to use multiple
host machines

25. © Copyright Ovum 200725
Oracle Real Application Testing
SQL Performance Analyser
Use cases
•Database upgrades and patches
•Platform configuration changes e.g. OS or hardware
•Schema –indexing, partition, materialized views
•SQL tuning e.g. SQL Profile creation

26. © Copyright Ovum 200726
Oracle Real Application Testing
SQL Performance Analyser
Functionality
•Capture SQL workload –into SQL Tuning Set
with SQL text, parsing schema, execution plans, statistics, etc.
•Reporting and analysis of workload and individual SQL statements,
with reporting drill-down.
•Recommendations on fixing regressions –SQL Tuning Advisor and
SQL Plan Baseline

27. © Copyright Ovum 200727
Oracle Real Application Testing
•SQL Performance Analyser
Comparisons with traditional methods
•Complete SQL workload with production context, e.g., bind
variables, session settings, etc
•Much reduced application setup stages
•Improved flexibility for detailed SQL analysis without manual
overhead
•Guided advice on tuning
•Back population of execution plans from test to production

28. © Copyright Ovum 200728
Analysis of Oracle Real Application
Testing
Case study based
Interviews with key technical staff
Details of technical deployments through beta phase
Description of benefits and issues

29. © Copyright Ovum 200729
Analysis of Oracle Real Application
Testing
Three beta customers
International hospitality and leisure organisation
Technical consultancy –including major international clients
International telecom equipment manufacturer
Common characteristics
Deployment in multi-terabyte production and test environments
Significant transaction workload and variability
Exercising different elements of Oracle Real Application Testing –
Database Replay & SQL Performance Analyser

30. © Copyright Ovum 200730
•Oracle Real
Application
Testing study
•Conclusions
and benefits
•Current
practices and
trends in
testing
•Business
context
12
34

31. © Copyright Ovum 200731
Conclusion and benefits
Greater agility
Enhanced service quality
Enhanced efficiency
Lower costs

32. © Copyright Ovum 200732
Benefit quotes…
“[without Real Application Testing, we don’t have a tool in place to tell our
customers that a database change we plan to make will work,”
•Reduced risk
“Database Replay could cut testing costs by one or two orders of
magnitude”
•Cost reduction
“you can gauge the impact of a specific change on a set of SQL
statements…This process is very easy with SQL Performance Analyser.
I’d say it provides a 1,000 times improvement for DBAs compared with
doing the tasks manually”
•Productivity

33. © Copyright Ovum 200733
Benefit Examples
Reduction in test cycle length by 80-90%
Reduces need to learn applications to build representative synthetic
workload
Reduces effort in test planning and design
Reduces effort in maintaining custom test scripts and tools
Reduce preparation for scenario testing
Removes test scale-up development
DBA productivity
Reduces need to most senior (and expensive) DBA staff on all
testing and performance tuning
Productivity of junior technical staff, increased by orders of
magnitude -reported by case studies. “Weeks to days”.

34. © Copyright Ovum 200734
Benefit Examples
•TCO enabler
–move to lower cost platforms more quickly e.g. Linux/RAC
combination
•Reputation and risk mitigation
–harder to quantify but think of the major problems and their impact…..
•Cost savings
–reduction in testing budget by 50-95% in case studies
–up to 70% more generally anticipated

35. © Copyright Ovum 200735
•Oracle Real
Application
Testing study
•Conclusions
and benefits
•Current
practices and
trends in
testing
•Business
context
12
34