On the Rationale and Use of Assertion Messages in Test Code: Insights from Software Practitioners

This presentation explores the use and impact of assertion messages in software testing. The study, based on a survey of 138 software practitioners, investigates the frequency, rationale, construction, and maintenance of assertion messages in test code. Key findings reveal that while assertion messages are widely used, there are challenges in maintaining their clarity and relevance. The research provides insights into how assertion messages contribute to debugging processes and offers recommendations for improving test code practices and tools.

Presented at: The 40th International Conference on Software Maintenance and Evolution (ICSME '24)

Date of Conference: Oct 2024
Conference Location: Flagstaff, AZ, United States

Preprint: https://www.peruma.me/publication/2024-icsme-assert/2024-ICSME-ASSERT.pdf

1. International Conference on Software Maintenance and Evolution
On the Rationale and Use of Assertion Messages in
Test Code: Insights from Software Practitioners
Anthony Peruma, Taryn Takebayashi, Rocky Huang, Joseph Averion, Veronica
Hodapp, Christian Newman, Mohamed Mkaouer
October 2024 | Flagstaff, AZ, United States

2. Which version would you prefer to
debug the failed test?

3. Information to start your debugging process
Assertion Messages

4. Why study
assertion
messages?
Assertion methods:
Core to unit tests
Assertion messages:
Optional yet impactful
Natural language:
Express the intent of a test in human-
readable terms
Limited prior work:
Focused on repository mining
Practitioner perspectives:
Understand the decision-making process

5. OUR
GOAL
Insights on the rationale,
construction, maintenance, and
effectivenessof assertion messages
in software testing.

6. RESEARCH
QUESTIONS
02
03
Message Construction
Maintenance Practices
Usage Frequency & Rationale01
Integration In Debugging04

7. STUDY DESIGN
138 completed
responses
1,000 invitations
No compensation
Keyword search
Manual profile review
22 survey questions

8. QUESTION TYPES
Ranking Questions
•Rationale: Capture relative importance and preferences
•Analysis:BordaCount method
Multi-Choice Questions
•Rationale: Identify diverse, non-exclusive information
•Analysis:Frequency analysis
•Note: Includes “Other” free-text response
Single-Choice Questions
•Rationale: Identify mutually exclusive information
•Analysis:Frequency analysis
•Note: Includes “Other” free-text response
Open-Ended Questions
•Rationale: Collect qualitative insights
•Analysis:Thematic analysis

9. RESULTS

10. Participant Demographics
Global Representation: 30
countries
•40% from the United States
•7% from Brazil
•6% from Canada
Role Distribution:
•57% Software
Engineers/Developers
•22% QA/Test Engineers
Professional Experience:
•42% with 10+ years of programming
experience
•81% are working full-time
Testing Framework Expertise:
•All participants are familiar with unit
testing
•JUnit (54%), NUnit(10%), PyTest
(10%)
Unit Testing as Part of Job Duties:
•Only 6.5% of participants do not engage in unit
testing at work
A diverse and representative sample

11. Assertion messages are often included but not universally
RESULTS: RQ 1 –Frequency and Rationale
Usage Frequency
•61.59% use assertion messages always or
very often
•19.56% rarely or never use assertion
messages
Reasons for Including
•Help troubleshoot test case failures
(32.51%)
•Improve understandability (26.63%)
•Serve as documentation (18.89%)
Reasons for Excluding
•Default execution log is sufficient
(24.44%)
•Keep assert methods simple
(23.11%)
•Reduce development time (16.44%)

12. RESULTS: RQ 2 –Message Construction
Messages typically describe the failure and are influenced by the assert method
Elements Considered
•Variables/functions in assert arguments
(20%)
•Type of assert method (19.42%)
•Exception/Error handling code (16.23%)
Message Content (Ranked)
•Description of the failure
•Expected outcome
•Actual outcome
Composition Style(Ranked)
•String Literal Only
➢assertTrue("List size too small", list.size() > 0)
•Variable/Method Calls Only
➢assertEquals(r.getValue(), a, b)
•Combination
➢assertFalse("too big: " + a, a>5))

13. RESULTS: RQ 3 –Maintenance and Challenges
Maintaining assertion messages is crucial and should be proactive
Poor Quality Indicators
(Ranked)
•Lack of clarity on failure
•Missing diagnostic data
•Excessive technical jargon
Maintenance Challenges
•Accurately reflecting failure (20%)
•Balancing conciseness and
descriptiveness (19%)
•Keeping messages up-to-date (17%)
Message Update Frequency (Ranked)
•When encountering test failures
•During code reviews
•After production code updates

14. RESULTS: RQ 4 –Integration in Debugging
Assertion messages are valuable in the debugging process
Starting pointfor
debugging test
failures
(32.97%)
Combining with
debugging tools/IDE
features
(21.15%)
Combine with
custom logging
statements
(14.34%)
Collaborationwith
team members
(11.47%)

15. Key Implications & Recommendations
Potential to significantly improve software testing practices
Improve Test Generation
Enhance automated test generation to include meaningful
assertion messages
Develop Appraisal Tools
Evaluate and suggest improvements for assertion messages
Enhance Debugging Integration
Better integrate assertion messages into debugging workflows
and IDE features
Further Research
Investigate the relationship between assertion messages and test
code identifiers

16. CONCLUSION
Assertionmessagesarevaluablebutnotuniversallyused
Challengesinmaintainingclear,up-to-dateassertionmessages
Serveasacrucialstartingpointindebuggingtestfailures
Potentialforimprovedtoolingformessagecreationandmanagement
01
THANK YOU!
Anthony Peruma, Taryn Takebayashi, Roc ky Huang, Joseph Averion,
Veronic a Hodapp, Christian Newman, Mohamed Mkaouer