10 ways to shoot yourself in the foot with tests - Shai Geva, PyCon IL, 2024

This document discusses 10 ways that tests can "shoot yourself in the foot" by being poorly designed or implemented. It covers issues like having no tests, unclear tests, tests that are not isolated, improper test scope, overuse of test doubles, and tests that are too slow. The key lessons are that tests need to be fast, isolated, and focused on validating core behaviors to form an effective feedback loop for developers.

1. TESTING
FOOTGUNS;
10 WAYS TO SHOOT YOURSELF IN THE FOOT
WITH TESTS
Shai Geva | Tech Lead @Sayata | @shai_ge

2. Shai Geva
~20 years making software
Love testing
Tech Lead, Sayata

3. Return
On
Investment

4. Tests
PROPERTIES

5. STRENGTH
Tests
PROPERTIES

6. STRENGTH
MAINTAINABILITY
Tests
PROPERTIES

7. PERFORMANCE
STRENGTH
MAINTAINABILITY
Tests
PROPERTIES

9. THERE ARE
NO TESTS
1

10. UNTESTED TESTS
2

11. 3
THE TESTS ARE
NOT ISOLATED

13. 4NO
LOCALITY
BEHAVIOR
OF

14. def test_something():
data = Path(PATH_TO_DATA_FILE).read_text()
assert calc_something(data) == 4.5

15. def test_something():
data = “””
{
<JSON data>
}
“””
assert calc_something(data) == 4.5

16. 5 UNCLEAR LANGUAGE
test_ (): ...
test_ (): ...

17. GUIDELINES

18. GUIDELINES
DECISIVE
LANGUAGE

19. GUIDELINES
DECISIVE
LANGUAGE
SPECIFIC
EXPLICIT

20. BookStore

21. test_edit_book():

22. test_edit_book_works_correctly():

23. test_edit_book_works_correctly():

24. test_user_should_be_able_to_edit_their_own_book():

25. test_user_should_be_able_to_edit_their_own_book():

26. test_user_can_edit_their_own_book():

27. 6TESTING
TOO
MANY
THINGS

28. SINGLE
FACT
BEHAVIOR

29. BookStore

30. test_user_can_edit_their_own_book

31. test_user_can_edit_their_own_book
test_edit_book

32. 7
IMPROPER
TEST SCOPE

33. COMPLETE STORY

34. COMPLETE STORY
COHESIVE WHOLE

35. COMPLETE STORY
COHESIVE WHOLE
BEHAVIOR / IMPLEMENTATION

36. Book Store MySQL

37. BEHAVIOR TEST IMPLEMENTATION TEST

38. BEHAVIOR TEST IMPLEMENTATION TEST
WITH BUG
WITHOUT BUG

39. BEHAVIOR TEST IMPLEMENTATION TEST
WITH BUG
WITHOUT BUG

40. def test_editing_description_sets_correct_value():
BEHAVIOR TEST IMPLEMENTATION TEST

41. BEHAVIOR TEST IMPLEMENTATION TEST
# Create book
# Edit book
# Get updated description
# Assert correctness
# Create book
# Edit book
# Get updated description
# Assert correctness
def test_editing_description_sets_correct_value():

42. # Create book - API
requests.post(...
# Edit book - API
requests.post(...
# Get updated description - API
new_desc = requests.get(…
# Assert correctness
assert new_desc == ...
# Create book - DB
# Edit book - API
# Get updated description - DB
# Assert correctness
BEHAVIOR TEST IMPLEMENTATION TEST
def test_editing_description_sets_correct_value():

43. # Create book - DB
DbBook.creat_new(...
# Edit book - API
requests.post(...
# Get updated description - DB
new_desc = DbBook.query_one(…
# Assert correctness
assert new_desc == ...
BEHAVIOR TEST IMPLEMENTATION TEST
# Create book - API
requests.post(...
# Edit book - API
requests.post(...
# Get updated description - API
new_desc = requests.get(…
# Assert correctness
assert new_desc == ...
def test_editing_description_sets_correct_value():

44. BEHAVIOR TEST IMPLEMENTATION TEST
# Create book - DB
DbBook.creat_new(...
# Edit book - API
requests.post(...
# Get updated description - DB
new_desc = DbBook.query_one(…
# Assert correctness
assert new_desc == ...
# Create book - API
requests.post(...
# Edit book - API
requests.post(...
# Get updated description - API
new_desc = requests.get(…
# Assert correctness
assert new_desc == ...
def test_editing_description_sets_correct_value():WHAT
COHESIVE HOW
INCOHESIVE

45. OUR CODE CHANGE

46. BOOK TABLE
DESCRIPTION DESC_ID ID
DESCRIPTION TABLE
VALUE ID
BOOK TABLE
DESCRIPTION ID …

47. BOOK TABLE
DESCRIPTION ID …
BOOK TABLE
DESCRIPTION DESC_ID ID
DESCRIPTION TABLE
VALUE ID

48. BOOK TABLE
DESCRIPTION DESC_ID ID
DESCRIPTION TABLE
VALUE ID
Book Store
/edit-book
/new-book
/get-book

49. BOOK TABLE
DESCRIPTION DESC_ID ID
DESCRIPTION TABLE
VALUE ID
Book Store
/edit-book
/new-book
/get-book
BEHAVIOR
TEST

50. BOOK TABLE
DESCRIPTION DESC_ID ID
DESCRIPTION TABLE
VALUE ID
Book Store
/edit-book
/new-book
/get-book
BEHAVIOR
TEST

51. BOOK TABLE
DESCRIPTION DESC_ID ID
DESCRIPTION TABLE
VALUE ID
Book Store
/edit-book
/new-book
/get-book
BEHAVIOR
TEST

52. BOOK TABLE
DESCRIPTION DESC_ID ID
DESCRIPTION TABLE
VALUE ID
Book Store
/edit-book
/new-book
/get-book
IMPLEMENTATION
TEST

53. BOOK TABLE
DESCRIPTION DESC_ID ID
DESCRIPTION TABLE
VALUE ID
Book Store
/edit-book
/new-book
/get-book
IMPLEMENTATION
TEST

54. BOOK TABLE
DESCRIPTION DESC_ID ID
DESCRIPTION TABLE
VALUE ID
Book Store
/edit-book
/new-book
/get-book
IMPLEMENTATION
TEST

55. BOOK TABLE
DESCRIPTION DESC_ID ID
DESCRIPTION TABLE
VALUE ID
Book Store
/edit-book
/new-book
/get-book

56. BOOK TABLE
DESCRIPTION DESC_ID ID
DESCRIPTION TABLE
VALUE ID
Book Store
/edit-book
/new-book
/get-book
BEHAVIOR
TEST

57. BOOK TABLE
DESCRIPTION DESC_ID ID
DESCRIPTION TABLE
VALUE ID
Book Store
/edit-book
/new-book
/get-book
IMPLEMENTATION
TEST

58. BOOK TABLE
DESCRIPTION DESC_ID ID
DESCRIPTION TABLE
VALUE ID
Book Store
/edit-book
/new-book
/get-book
IMPLEMENTATION
TEST

59. COHESIVE
BEHAVIOR
INCOHESIVE
IMPLEMENTATION

60. COHESIVE
BEHAVIOR
INCOHESIVE
IMPLEMENTATION
PASSING == WORKS?

61. COHESIVE
BEHAVIOR
INCOHESIVE
IMPLEMENTATION
PASSING == WORKS?
CATCHES BUGS
AS EXPECTED?

62. COHESIVE
BEHAVIOR
INCOHESIVE
IMPLEMENTATION
PASSING == WORKS?
CATCHES BUGS
AS EXPECTED?
ONLY NECESSARY WORK?

63. COHESIVE
BEHAVIOR
INCOHESIVE
IMPLEMENTATION
PASSING == WORKS?
CATCHES BUGS
AS EXPECTED?
ONLY NECESSARY WORK?
HIGH CONFIDENCE?

64. SCARY
CHANGES
DRAMATIC
DIFFERENCE

65. BEHAVIOR
COHESIVE

66. 8TEST
DOUBLES
EVERW
E
ERYH

67. TEST DOUBLES == IMPLEMENTATION

68. REAL
TEST
DOUBLE

69. REAL
TEST
DOUBLE

70. REAL
TEST
DOUBLE

71. REAL
TEST
DOUBLE

72. REAL
TEST
DOUBLE

73. USE WITHCAUTION

74. USE WITH
BUT
CAUTION
HOW?

75. DESIGN

76. MySQL db_fake = []

77. TEST
THE FAKE

79. WHY NOT BOTH?

80. MySQL
service db_fake
service

81. service
MySQL
db_fake
service

82. service
MySQL
db_fake
service

83. USE, AND VERIFY

84. Footgun #: the tests are slow
SLOW
TESTS
9

85. SLOW TESTS #1:
THE BOTTLENECK
AND THE TIME BOMB

86. SLOW TESTS: THE BOTTLENECK AND THE TIME BOMB

87. >>> workday_hours = 10

88. >>> workday_hours = 10
>>> test_run_hours = 2

89. >>> workday_hours = 10
>>> test_run_hours = 2
>>> test_run_per_day = 10 / 2
5

90. >>> workday_hours = 10
>>> test_run_hours = 2
>>> test_run_per_day = 10 / 2
5
>>> if num_finished_tasks_today > 5:

91. >>> workday_hours = 10
>>> test_run_hours = 2
>>> test_run_per_day = 10 / 2
5
>>> if num_finished_tasks_today > 5:

92. >>> workday_hours = 10
>>> test_run_hours = 2
>>> test_run_per_day = 10 / 2
5

93. >>> max_reasonable_test_run_min
10
>>> max_worst_case_test_run_min
15

94. DEFUSE THE BOMBHOW TO

95. DEFUSE THE BOMB
PREPARED TO OPTIMIZE
HOW TO
BE

96. DEFUSE THE BOMB
PREPARED TO OPTIMIZE
CAN RUN IN PARALLEL
HOW TO
BE
TESTS

97. DEFUSE THE BOMB
PREPARED TO OPTIMIZE
CAN RUN IN PARALLEL
ISOLATED TESTS
HOW TO
BE
TESTS
WRITE

98. SLOW TESTS 2#:
THE FEEDBACK LOOP AND
THE BUG FUNNEL

99. FAST = 3 seconds, watch

100. FAST = 3 seconds, watch
SLOW = 10 minutes, CI

101. BE FAST
IN PRACTICE

102. HOW LONG FOR THE TESTS TO RUN?
HOW LONG TO CATCH A BUG?

103. ALL BUGS
FASTEST: e.g. < 10 sec
EVEN SLOWER: CI, 2 min
REALLY SLOW: CI, 15 min
LESS FAST: < 1 min

104. ALL BUGS
CI END-TO-END

105. ALL BUGS (10)
10
CI END-TO-END

106. ALL BUGS (10)
2
2
4UTs FOR THIS MODULE
CI UTs
CI END-TO-END
ALL LOCAL UTs 2

107. 10
WRONG
PRIORITIES

108. Tests
PROPERTIES
PERFORMANCE
STRENGTH
MAINTAINABILITY

109. STRENGTH

110. PERFORMANCE
STRENGTH
MAINTAINABILITY

111. WHY?

112. SLOW

113. SLOW
TOO
LONG

114. SLOW
TOO
LONG
FEWER
TESTS

115. SLOW
TOO
LONG
FEWER
TESTS
WEAK

116. HARD TO
MAINTAIN
EXPENSIVE
FEWER
TESTS
WEAK

117. PERFORMANCE
STRENGTH
MAINTAINABILITY

118. SUMMING UP

119. TOOLS:
>>> pytest-watch
Re-run tests on file changes
>>> pytest-testmon
Only run tests that can be impacted
by the code that changed
>>> pytest-xdist
Run tests in parallel
>>> pytest-env, pytest-dotenv
Deal with env vars
>>> unittest subTest Organizing test output
when you have large tests.
For pytest: pytest-subtests
>>> factory-boy, faker
Generate random test data
>>> hypothesis !
Property-based testing - for getting strong
tests.
>>> coverage.py
Python test-coverage
>>> vcrpy
Record HTTP requests
Slides:
bit.ly/testing_footguns_pycon_il_2024

120. 1.There are no tests
2.Untested tests
3.No Locality of Behavior
4.Unclear language
5.Testing too many things
6.The tests are not isolated
7.Improper scope
8.Test doubles
9.Slow tests
10.Wrong priorities

121. Shai Geva
Slides:
bit.ly/testing_footguns_pycon_il_2024
QUESTIONS?
@shai_ge