Non-functional Testing (NFT) Overview

Quick overview of Non-Functional Testing

1. Non Functional Testing “Measure what is measurable and what is not measurable – Make measurable” -Galileo Galilei Assaf Halperin Sr. Consultant

2. What is NFT ? Anything in the product behavior or surrounding environments, which is not a functional aspect The functional execution of the product in order to achieve “other” product requirements Focus on Performance, Dependability, Operational aspects, production readiness issues Some methodologies and standards exist in the market place (ISO 9126)

3. Dependability What makes a good Software? (According to ISO 29126) Functionality Interoperability , Security Reliability Availability , Recoverability Maintainability Usability Efficiency Performance aspects Portability Installability

4. NFT Objectives Complete and enhance Package testing as part of product delivery readiness Improve quality of product in areas of performance, stability, resilience, operability etc. Reduce production risks and costs associated with non-functional aspects of the product Optimize the way product is installed, setup, configured, executes, managed and monitored Collect and produce measurements and metrics for internal R&D use (Sizing, Capacity, targets etc.) Improve and enhance in-depth knowledge of the product behavior, technologies in use Support Ad-Hoc requests for testing

5. NFT Testing Categories Performance Volume test Load testing Stress (Load + Volume) AKA Negative testing Function under load Balancing/Parallelism/Scaling Installation & Upgrade Usability Security Intrusion , Hardening Availability & Resilience H/A, BC ,DRP Testing Operability/Inter-Operability Stability testing Configuration Architecture Maintainability Cleanup / Purge Conversion / Data Migration

6. Test environments Long term use Reuse environment from one release to another Reflection of production Customer data Transactional data + History All Server types App lication, Web, Communication Auxiliary, Admin, Monitoring, Load servers Large population of Customers, transactions and historical data

7. Test environments – Continued Server & Storage aspects Multi-CPU, Multi-Core servers or Blades Large amounts of Memory Enterprise grade storage arrays Large amount of storage Multiple O/S environments Virtualization Clustering O/S, Databases Communications and Security devices

8. Test environments – Tools In-house tools Scripts Test automation and envelopes Application tools & Simulators 3 rd party tools Debuggers Performance & Load testing tools Profiling / Instrumentation tools Memory management tools Monitoring tools Tuning tools Simulators Many others….

9. Direct Staffing PM & Team Lead Environment & SCM Application / SME’s Testers Infrastructure Additional Staff (Ad-hoc & Expert support) Middleware Performance R&D Development Application experts Database Experts System Unix/Linux/Microsoft Storage guys Etc… Administration

10. NFT and the S/W Lifecycle

11. NFT and the S/W Lifecycle Performance testing – Case study

12. NFT and the S/W Lifecycle Performance testing – Case study Dev QA UAT Prod Post Prod Method 1 : No Performance testing Moto: “Everything is going to be OK” Chances of failure: 100% Things to prepare: Your CV Mitigation plan: Perform NFT No Testing

13. NFT and the S/W Lifecycle Performance testing – Case stufy Dev QA UAT Prod Post Prod Method 2 : Performance testing in Prod Moto: “ Perf . Eng. Team will fix…” Chances of failure : 90 % Things to prepare: Good excuses Mitigation plan: Perform NFT earlier NFT

14. NFT and the S/W Lifecycle Performance testing – Case stufy Dev QA UAT Prod Post Prod Method 3 : Performance testing in UAT Moto : “Let the End users find the issues..” Chances of failure : 80 % Things to prepare: Tylenol ( and good excuses) Mitigation plan: Perform NFT earlier NFT

15. NFT and the S/W Lifecycle Performance testing – Case study Dev QA UAT Prod Post Prod Method 4 : Performance testing in QA Moto: " Let’s keep it In-House… " Chances of failure : 30% Things to prepare: A Speech about bad programmers Mitigation plan: Perform NFT earlier NFT

16. NFT and the S/W Lifecycle Performance testing – Case stufy Dev QA UAT Prod Post Prod Method 5 : Performance testing in DEV Moto: " Developers responsibility " Chances of failure : 10% Things to prepare: A presentation about NFT Mitigation plan: Get an expert consultant NFT

17. NFT and the S/W Lifecycle Performance testing – Case stufy Dev QA UAT Prod Post Prod Method 6 : End-2-End NFT Moto: “Let’s build Quality Software” Chances of failure : 5% Things to prepare: Cappuccino Mitigation plan: Not needed NFT NFT NFT NFT NFT

18. Quick Recap NFT is Critical for producing quality software NFT has many categories and aspects Choose the ones that are critical for your company Finding and fixing problems late – is costly NFT should be an integral part of the S/W Lifecycle Make your company “NFT Aware” Serious NFT is expensive to do People, Equipment, Time & Effort, Overall costs Method #6 seems to work best

19. Thank you !