Coverage measurement in model-based testing of web applications: �Tool support and an industrial experience report

Presented in: The 20th Workshop on Advances in Model-Based Testing (A-MOST 2024), co-located with the 17th IEEE International Conference on Software Testing, Verification and Validation (ICST 2024),
May 27, 2024
Toronto, Canada

1. Coverage measurement in model-based testing of web applications: Tool support and an industrial experience report Presented in: The 20th Workshop on Advances in Model-Based Testing (A-MOST 2024), co-located with the 17th IEEE International Conference on Software Testing, Verification and Validation (ICST 2024), May 27, 2024 Toronto, Canada Previously in: Testinium

2. Context and industrial setting Model-based testing training, Istanbul, Turkiye, 2019 This work has been conducted in Testinium A.Ş. Corporation, based in İ stanbul, Türkiye ( testinium.com ) Testinium offers software testing products and services to a large number of clients i n Türkiye and across Europe at any given time As of 2024, more than 800 software test engineers work in the company This work is one recent output from our ongoing R&D activities, which started in 2019, to further improve state of test practices in the company and also test practices of its clients We have already published 15+ papers, given various industry talks, training sessions, and a number of open-source tools Our research approach: Action-Research Different than most classic / typical academic research Starts here… Gorschek et al., "A Model for Technology Transfer in Practice“, IEEE Software, vol. 23, pp. 88-95, 2006

3. Background: Doing MBT in practice since 2019 Since January 2019, we have pragmatically used Model-based Testing (MBT), to improve the company’s test-automation practices Our earlier experience and results were published in: Example of MBT models (and their formalism): Valid login From among 100+ MBT tools, we conducted a systematic tool-selection process and chose: the open-source popular MBT tool GraphWalker ( graphwalker.github.io ) System Under Test (SUT) : The company’s flagship product testinium.io ): a comprehensive test-automation management tool (design, develop, run, report test-scripts for Selenium, Appium, Gauge BDD, …) Partial MBT model 

4. Need for coverage analysis in our MBT projects We observed the need for coverage analysis …. As we were progressing with multiple large-scale MBT projects, in testing both our company’s internal systems and also our clients’ software (e.g., MBT of a bit-coin trading app) We saw the need for almost every type of coverage: Test coverage: the ratio of requirements, models or even user-interface (UI) artifacts, of the SUT, exercised (covered) by a given test suite Requirement coverage Model coverage Code coverage: the ratio of source code exercised by a given test suite Front-end code-coverage (browser- / client-side) Back-end code-coverage (server-side) We know that there are hundreds or even thousands of code and test coverage tools, which are used in different test automation contexts and for different programming languages. But we could not find an existing off-the-shelf coverage tool matching our exact need (for MBT of web and mobile apps, and the above four types of coverage) Thus  Decided to develop our own coverage tool

5. The MBTCover tool Usage context and architecture of the tool: (the diagram / model could look overwhelming, but let’s review it part by part)

6. MBTCover tool in action An example large MBT test suite: SUT: The company’s flagship product testinium.io ) “Testing our own test tool!” 18 test models were developed Important note: We have observed that design and development of executable MBT models is not highly effort-intensive (even for large systems), once engineers get the expertise. For this particular SUT, the effort was about 2 man-weeks. Using automated model generation approaches can be error-prone, may require even more time to inspect and fix, and thus not effective # of (sub-)models=18 Total # of nodes=170 Total # of edges=260 The login MBT model (one of the 18 models): … 18 models in total, connected by “shared’ nodes ( orange color )

7. MBTCover tool in action Zoomed next… SUT MBT test suite MBTCover MBTCover

8. MBTCover tool in action

9. Usefulness and benefits of MBTCover tool We have not yet been able to quantitatively measure the benefits of the tool Conducting quantitative benefit analysis in industry and/or experiments are costly (often seen as “burden”); and thus, not easy to convince engineers / managers to conduct them However, as an alternative: In our conversations with eight test engineers in the company, their opinion has been that MBTCover is a useful tool in exploring the ongoing progress, scale and extent of MBT test automation ,… via the four consolidated coverage data that it provides: front-end code, back-end code, requirements and model coverage By using MBTCover, our test engineers have reported that they are able to: (1) monitor progress and extent of MBT testing in a long project (2) improve MBT test suites to ensure full coverage w.r.t. any of the four coverage domains above (front-end, back-end, requirements and model coverage) (3) pinpoint unused parts of JS files, in large test runs, and thus optimize the size of JS code files (4) assess traceability of requirements to MBT models to code, which have been useful in various regression testing and change-impact analysis scenarios

10. We believe in open-source Most our non-commercially-sensitive tools have been made open-source The MBTCover is fully open-source in: www.github.com/vgarousi/MBTCover The online repository also includes the tool’s design document, several SUT examples, their UML use-case requirements, and demo videos MBT test model repositories (to apply MBTCover to, as “examples”) www.github.com/vgarousi/MBTofTestinium MBT test suites of the UK’s three COVID contact-tracing apps: www.github.com/vgarousi/MBTofCOVIDapps Various demo videos: A YouTube playlist: www.bit.ly/MBTVideosTestinium  Demo of MBTCover: youtu.be/jTw8u4JD9JE?si

11. Discussions - Q/A Calgary Baku Ankara Belfast Cities that this work has some forms of roots in … Ottawa İ stanbul