JaSST'18 Hokkaido Improve Automation Testing

we introduced automation in 2016, and improved it every year.
We set goal "Improve automation testing" in 2018.
talk about this activity

1. Improve Automation Testing ~ speed up deployment ~ Nov. 30, 2018 Sadaaki Emura Rakuten, Inc.

2. Introduction

3. Who am I Name : Sadaaki Emura (nickname M) join in Rakuten: 2007 Group: Service Quality Assurance in LPD established in 2016 Role: Test Engineer Lead, automation Hobby: jog , climbing , horse racing

4. LPD organization Product Manager Engineer QA ( Leisure Product Department )

5. Our team (QA) Scope : 5 services Members : 9 + off site (if needed) Skill : QA specialist (almost not programmer) Mission : Quality Assurance by manual & automation test

6. Our Automation Testing activity

7. Automation testing activity 2016 Proof of concept 2017 Expansion 2018 Improve speed

8. Why is this year theme “improve speed”? Make impact = release many apps frequently photoAC

9. What was bottleneck in development speed? Specification Design Development Test (E2E) Development Test Design Test case / script Implement Test Execution scope

10. What was bottleneck in development speed? 1. Setup test environment takes time 2. Test feedback was late 3. Scripting for PC and SP takes time

11. 1. Setup test environment takes time photoAC What was bottleneck in development speed? Example Install apps (1 st time) 120 min Setup devices 5 min Run apps 2 min Closing 5 min :

12. 2. Test feedback was late (E2E) ■ One job has Mouse click : 3 sec Input form : 5 sec Page transition : 10 sec : 2-3 mins / page 10 mins / job What was bottleneck in development speed? ■ Single process worsens speed Test job Test job Test job 10 mins

13. For PC For SP What was bottleneck in development speed? 3. Scripting for PC and SP takes time Apps support PC and SP Need to create 2 script

14. How did we resolve?

15. Solution to resolve these bottleneck Every time, Everywhere, Everyone can use Many test run at same time One script covers multiple devices Features

16. Test script management Mobile Labo Architecture Chrome FF IE iPhone Android Edge Safari CI environment Appium API Selenium API Selenium API Client side Test Environment (Mobile Labo ) Clients Windows server MAC server MAC server Test script management Test script management https://jenkins.io/ Java

17. Mobile Labo image https://qiita.com/emurin Document ▼ Mac Windows Smart phone Appium /Selenium

18. 1. Every time, Everywhere, Everyone can use per person every time As Is Install apps (1 st time) 120 min Setup devices 5 min Run apps 2 min Closing 5 min : To Be Connect API 1 min

19. 2. Many test run at same time FireFox Chrome android iOS iOS One JENKINS runs a couple of test script

20. 3. One script covers multiple devices PC SP Able to run test in each device ※apps rule - same function - same class name (HTML) : example : responsive site

21. What’s happened?

22. Improvement effect 1 Example ( keiba ) Betting Lite ( 投票 Lite) in Horse Racing PC & SP are same function Step is Choose race, Bet , Confirm , Complete 9 betting method / 4 type Selecting order of placing(Max 3 rd ) ⇒ 736 test case x 2 devices(PC , SP) PC SP

23. Improvement effect 1 (initial phase) Manual testing Previous automation Mobile labo 120 hour 60 hour Method Speed 30 hour ※1 include scripting , setup task ※2 include scripting task ※1 ※2 75% CUT

24. Improvement effect 1 (CI phase) Manual testing Previous automation Mobile labo 120 hour 1.5 hour Method Speed 0.5 hour ※1 include setup task 99% CUT ※1

25. Improvement effect 2 Example (Wedding) Wedding Service is responsive site Reserve wedding hall Review wedding hall Search wedding hall : ⇒ 302 test case x 2 devices(PC , SP) PC SP

26. Improvement effect 2 (initial phase) Manual testing Previous automation Mobile labo 50 hour 20 hour Method Speed 11 hour ※1 include scripting , setup task ※2 include scripting task ※1 ※2 78% CUT

27. Improvement effect 2 (CI phase) Manual testing Previous automation Mobile labo 50 hour 0.5 hour Method Speed 0.2 hour ※1 include setup task 99% CUT ※1

28. Improvement effect Mobile Labo improves speed FREEIMAGES/Geo Okretic

29. Next challenge

30. photoAC Next challenge encounter issues

31. Issue 1 : clash with others iPhone 1 Android Chrome iPhone 2 iPhone 1 Android Chrome iPhone 2 clash Able to use devices Not able to use same device at same time

32. Issue 2 : need to know device location iPhone 1 Android 1 Appium API MAC server 1 iPhone 2 Android 2 iPhone X Android X Appium API MAC server X … Clients set configuration Appium IP address & port Capabilities (UDID etc ) Where ?

33. New problem and Next step Improve speed more Clash with others Need to know device location FREEIMAGES/Alexander

34. In this year ?

35. New problem and Next step (This was situation in Aug.) Improve speed more Clash with others Need to know device location FREEIMAGES/Alexander Improved

36. New problem was resolved in 2 months Android Selenium hub Appium node Android Appium node ・・・・ Handle node (client does not need to know where appium ) Block client if devices are busy

37. How did we choose automation testing tool? Free license License fee ◎ × Programming skill × (Ruby programmer) ◎ Maintenance cost × (Ruby programmer) △ (annual license support) Support browser ◎ ( IE,Edge,Chrome,FF,SP ) ○ (depend on tool ) Setup cost × ◎ Low cost Low skill