Test Strategy For Future Cloud Architecture

Great

1. Test Performance using cloud Abhinav Maheshwari (Chief Scientist & Director, 3Pillar Labs) Sayantam Dey (Architect, 3Pillar Labs )

2. objectives What are the benefits of using the cloud for testing performance over traditional means? What are the various tools available for cloud based performance testing? How can we use open source load testing frameworks in the cloud? How to design a custom framework for rapid testing with large number of concurrent users?

3. traditional Performance Testing Load Generation Server (High end or multiple servers) Application Server Workstation Deploy replica of the production environment in the LAN Deploy a high-end server for generating load (on the same LAN) Run agents of a load test tool like JMeter, Grinder or WAPT Connect remotely to load generation server Load generation server collects metrics and generates reports

4. Cloud based Performance Testing Load Generation Server (High end or multiple servers) Application Server Workstation Cloud machines i n different geographies Application Server Workstation

5. Benefits of using the cloud Traditional Approach Cloud based Approach Provisioning physical machines is costly, specially for medium sized companies Large number of small, virtual machines running in bursts, ideal usage of cloud from a cost perspective Manual provisioning of load generation server requires time and effort Simpler provisioning, can even be automated using API Application needs to be deployed in the same LAN Application may be hosted at any public IP, also more realistic from latency point of view Not dynamic to address any significant increase in the performance requirement Distribute load generation over multiple instances which can be increased dynamically Extremely difficult to simulate geographically distributed load Cloud clusters in different geographic locations can simulate geographically distributed load     

6. Cloud approach – choices ? Commercial Products 1 Record or upload scripts for usage scenarios using the web based interface 2 Configure testing parameters like number of virtual users, time for test, etc. through interface 3 Start the test through web based interface 4 Download reports and analyze

7. Open source frameworks 1 Create the required number of VM instances on the cloud, deploy the chosen load testing framework 2 Configure all the parameters for the framework and upload scripts written for the chosen framework 3 Run agents on the system under test and then start the test 4 Download reports and analyze Cloud approach – choices ? Commercial Products 1 Record or upload scripts for usage scenarios using the web based interface provided by the vendor 2 Configure testing parameters like number of virtual users, time for test, % allocation to usage scenarios etc. through interface 3 Start the test through web based interface 4 Download reports and analyze

8. Open source frameworks Very popular, large user base Very little programming knowledge required Handles many protocols Plugin architecture for extension Requires programming Scripting in Jython or Clojure Distributed framework Cloud approach – choices ? Commercial Products JMeter compatible Real time reports 10K virtual users x 25 hours @ $1000 Web Performance Selenium IDE scripts Real browser or simple request playback (virtual user) 10K virtual users x 25 hours @ $3750 Custom scripts Real time analytics Custom pricing

9. Open source frameworks Only few frameworks are mature enough to work over the cloud Only the cost of cloud infrastructure, which is much less expensive Completely flexible as scripting is done through open APIs Can only be used through command line, user has to care about calculating VM requirement, creation and closure Reports are basic and sometimes not helpful A custom framework? Commercial Products Not many choices since there are only a few products in the market May prove to be expensive for repeated load testing May use recording or proprietary scripting, which may not be fully flexible Provide web-based interface to directly run tests without worrying about virtual machines Provide easy to use reports    X  Sizable investment - Suitable for organizations needing repeated performance tests

10. Case study - hailstorm Configuration JMeter Amazon SUT Provisioning Manager JMeter Scripts Script Manager JMeter Controller Data Collector Report Engine JFreeChart AMI JMeter JRE Agent System Under Test (SUT) Agent Agent HTTP

11. Case study - hailstorm Observations Sharp rise in response time for over 50 users “Detail Page” degrades significantly for 75 users and above System is stable but peaks out at 75 concurrent users and starts degrading at 100 users Maximum throughput of 3.2 transactions per second at 75 users is low © Copyright 2012 3PILLAR GLOBAL All rights reserved Sample Report Summary

12. Case study - hailstorm © Copyright 2012 3PILLAR GLOBAL All rights reserved Page Response time Avg. ( ms ) Response time 90tile ( ms ) % errors Home Page 1384 2293 0.0 Login Page 559 764 0.0 Login 1050 1740 0.0 Browse 1996 2898 0.25 Category Browsing 2553 3601 0.0 Detail Page 5368 7231 0.27 Search 2054 2948 0.0 Sample Report for one iteration

13. Case study - Hailstorm Application A leading provider of online classroom and instruction solutions Key Challenge Required simulation of 40,000 concurrent users Rapid and cost effective solution to capture load metrics for large number of concurrent users The Solution Different JMeter scripts/plans seamlessly integrated Engaging reports pointing out bottlenecks generated