PyCon JP 2024 Streamlining Testing in a Large Python Codebase .pdf

Maintaining code quality in a growing codebase is challenging. We faced issues like increased test suite execution time, slow test startups, and coverage reporting overhead. By leveraging open-source tools, we significantly enhanced testing efficiency. We utilized pytest-xdist for parallel test execution, reducing test times and accelerating development. Optimizing test startup with Docker and Kubernetes for CI, and pytest-hot-reloading for local development, improved productivity. Customizing coverage tools to target updated files minimized overhead. This resulted in an 8000-case increase in test volume, 85% test coverage, and CI tests completing in under 15 minutes.

1. Streamlining Testing in a Large
Python Codebase
Jimmy Lai, Staff Software Engineer, Zip

Sept. 28, 2024
https://www.slideshare.net/jimmy_lai/presentations

2. Python Testing: pytest, coverage, and continuous integration01
02
03
04
05
Outline
The Slow Test Challenges
Optimization Strategies
Results
Recap

3. Zip is the world’s leading
Intake & Procurement Orchestration Platform

4. A Large Python Codebase
100 developers
We’re hiring fast

1

5. A Large Python Codebase
100 developers
We’re hiring fast
2.5 million lines of
Python code
Doubling every year
1 2

6. Scaling Challenges
100 developers
We’re hiring
2.5 million lines of
Python code
Doubling every year
1 2
Number of tests and
tech debt increase
fast
3

7. Why Tests?
Quality Assurance
1

8. Why Tests?
Quality Assurance Refactoring Confidence
1 2

9. Why Tests?
Quality Assurance Refactoring Confidence Documentation
1 2 3

10. Test Execution Time01
02
03
Useful Test Metrics
Test Reliability
Test Coverage

11. Simple Testing using pytest
https://pypi.org/project/pytest/
# in helper.py
def is_even(number: int) -> bool:
if number % 2 == 0:
return True
else:
return False

12. Simple Testing using pytest
https://pypi.org/project/pytest/
# in helper.py
def is_even(number: int) -> bool:
if number % 2 == 0:
return True
else:
return False

# in test_helper.py
from helper import is_even

def test_is_even_with_even_number():
assert is_even(4) == True


def test_is_even_with_zero():
assert is_even(0) == True

13. Simple Testing using pytest
https://pypi.org/project/pytest/
# in helper.py
def is_even(number: int) -> bool:
if number % 2 == 0:
return True
else:
return False

# in test_helper.py
from helper import is_even

def test_is_even_with_even_number():
assert is_even(4) == True


def test_is_even_with_zero():
assert is_even(0) == True

> pytest . -vv

======= test session starts =======
collected 2 items

test_helper.py::test_is_even_with_even_number PASSED
test_helper.py::test_is_even_with_zero PASSED

======= 2 passed in 0.03s =======

14. Simple Testing using pytest
https://pypi.org/project/pytest/
# in helper.py
def is_even(number: int) -> bool:
if number % 2 == 0:
return True
else:
return False

# in test_helper.py
from helper import is_even

def test_is_even_with_even_number():
assert is_even(4) == True


def test_is_even_with_zero():
assert is_even(0) == True

> pytest . -vv

======= test session starts =======
collected 2 items

test_helper.py::test_is_even_with_even_number PASSED
test_helper.py::test_is_even_with_zero PASSED

======= 2 passed in 0.03s =======

Test Execution TimeTest Reliability

15. Measure Test Coverage
> pytest --cov . -vv

======= test session starts =======
collected 2 items

test_helper.py::test_is_even_with_even_number PASSED
test_helper.py::test_is_even_with_zero PASSED

------------- coverage -------------
Name Stmts Miss Cover
------------------------------------
helper.py 5 1 80%
test_helper.py 6 0 100%
------------------------------------
TOTAL 11 1 91%

======= 2 passed in 0.03s =======

https://pypi.org/project/pytest-cov/
Test Coverage

16. Measure Test Coverage
> pytest --cov . -vv

======= test session starts =======
collected 2 items

test_helper.py::test_is_even_with_even_number PASSED
test_helper.py::test_is_even_with_zero PASSED

------------- coverage -------------
Name Stmts Miss Cover
------------------------------------
helper.py 5 1 80%
test_helper.py 6 0 100%
------------------------------------
TOTAL 11 1 91%

======= 2 passed in 0.03s =======

To increase the test coverage: add a new test case
for odd numbers
https://pypi.org/project/pytest-cov/
Test Coverage

17. Continuous Integration
Practice: continuously merge changes into the shared codebase
while ensuring the quality through automated testing

18. Continuous Integration
Practice: continuously merge changes into the shared codebase
while ensuring the quality through automated testing

●Developers submit a pull request (PR) for code review

19. Continuous Integration
Practice: continuously merge changes into the shared codebase
while ensuring the quality through automated testing

●Developers submit a pull request (PR) for code review
●Run tests to verify the code changes

20. Continuous Integration
Practice: continuously merge changes into the shared codebase
while ensuring the quality through automated testing

●Developers submit a pull request (PR) for code review
●Run tests to verify the code changes
●Merge a PR after all tests passed and approved

21. Continuous Integration
Practice: continuously merge changes into the shared codebase
while ensuring the quality through automated testing

●Developers submit a pull request (PR) for code review
●Run tests to verify the code changes
●Merge a PR after all tests passed and approved
Ensure that test reliability and test coverage meet the required
thresholds

22. Continuous Integration using Github Workflows
# File: .github/workflows/ci.yml
name: CI

on:
pull_request: # on updating a pull request
branches:
- main
push: # on merging to the main branch
branches:
- main
https://docs.github.com/en/actions/using-workflows

23. Continuous Integration using Github Workflows
jobs:
run-pytest:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.13'
- run: pip install -r requirements.txt
- run: pytest
# File: .github/workflows/ci.yml
name: CI

on:
pull_request: # on updating a pull request
branches:
- main
push: # on merging to the main branch
branches:
- main
https://docs.github.com/en/actions/using-workflows

24. Challenge: Test Execution Time Increases Over Time
Number of tests
increases
1
Pain Point:
Long Test Execution Time

25. Challenge: Test Execution Time Increases Over Time
Number of tests
increases
Codebase size
increases
1 2
Pain Point:
Test Coverage OverheadPain Point:
Long Test Execution Time

26. Challenge: Test Execution Time Increases Over Time
Number of tests
increases
Codebase size
increases
Number of
dependencies increases


1 2 3
requirements.txt
Pain Point:
Test Coverage Overhead Pain Point: Slow Test StartupPain Point:
Long Test Execution Time

27. ?????? Strategy #1: Parallel Execution

28. Run Tests in Parallel on multiple CPUs
https://pypi.org/project/pytest-xdist/
pytest -n 8 # use 8 worker processes


# use all available CPU cores
pytest -n auto

29. Run Tests in Parallel on multiple CPUs
https://pypi.org/project/pytest-xdist/
pytest -n 8 # use 8 worker processes


# use all available CPU cores
pytest -n auto
N: number of CPUs (e.g. 8 cores)
Test Execution Time ÷ N


10,000 tests ÷ N is still slow

30. Run Tests in Parallel on multiple Runners
https://pypi.org/project/pytest-split/
# Split tests into 10 parts and run the 1st part
pytest --splits 10 --group 1

31. Run Tests in Parallel on multiple Runners
https://pypi.org/project/pytest-split/
# Split tests into 10 parts and run the 1st part
pytest --splits 10 --group 1


N: number of CPUs
Test Execution Time ÷ N
M: number of runners

10,000 tests ÷ N ÷ M

32. Run Tests in Parallel on multiple Runners
https://pypi.org/project/pytest-split/
# Split tests into 10 parts and run the 1st part
pytest --splits 10 --group 1

# Assumption: All tests have the same
# test execution time.
# Unbalanced test execution time can lead to
# unbalanced Runner durations



N: number of CPUs
Test Execution Time ÷ N
M: number of runners

10,000 tests ÷ N ÷ M

33. Run Tests in Parallel on multiple Runners
https://pypi.org/project/pytest-split/
# Split tests into 10 parts and run the 1st part
pytest --splits 10 --group 1

# Assumption: All tests have the same
# test execution time.
# Unbalanced test execution time can lead to
# unbalanced Runner durations


# To collect test execution time
pytest --store-durations

# To use the collected time
pytest --splits 10 --group 1 --durations-path
.test_durations
N: number of CPUs
Test Execution Time ÷ N
M: number of runners

10,000 tests ÷ N ÷ M

34. Use Multi-Runners and Multi-CPUs in a Github Workflow
python-test-matrix:
runs-on: ubuntu-latest-8-cores # needs larger runner configuration
strategy:
fail-fast: false # to collect all failed tests
matrix:
group: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
steps:
- run: pytest -n auto -split 10 --group ${{ matrix.group }} ...

https://docs.github.com/en/actions/using-workflows

35. python-test-matrix:
runs-on: ubuntu-latest-8-cores # needs larger runner configuration
strategy:
fail-fast: false # to collect all failed tests
matrix:
group: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
steps:
- run: pytest -n auto -split 10 --group ${{ matrix.group }} ...

Use Multi-Runners and Multi-CPUs in a Github Workflow
https://docs.github.com/en/actions/using-workflows
10 x 8 = 80 concurrent test worker processes

36. ?????? Strategy #2: Cache

37. Cache Python Dependency Installation
pip install -r requirements.txt
# resolve dependency versions
# download and install dependencies

38. Cache Python Dependency Installation
pip install -r requirements.txt
# resolve dependency versions
# download and install dependencies

# In Github Workflow
steps:
- uses: actions/cache@v3
id: dependency-cache
with:
key: ${{ hashFiles('requirements.txt') }}

- if: steps.dependency-cache.outputs.cache-hit != 'true'
run: pip install -r requirements.txt

39. Cache Python Dependency Installation
pip install -r requirements.txt
# resolve dependency versions
# download and install dependencies

# In Github Workflow
steps:
- uses: actions/cache@v3
id: dependency-cache
with:
key: ${{ hashFiles('requirements.txt') }}

- if: steps.dependency-cache.outputs.cache-hit != 'true'
run: pip install -r requirements.txt


Save 5-10 minutes on each CI run in a large
codebase

40. Cache Python Dependency Installation
pip install -r requirements.txt
# resolve dependency versions
# download and install dependencies

# In Github Workflow
steps:
- uses: actions/cache@v3
id: dependency-cache
with:
key: ${{ hashFiles('requirements.txt') }}

- if: steps.dependency-cache.outputs.cache-hit != 'true'
run: uv pip install -r requirements.txt --system

Save 5-10 minutes on each CI run in a large
codebase


Use uv to install faster
https://pypi.org/project/uv/

41. Cache Non-Python Dependency Installation
Common non-Python dependencies:
●Python and Node interpreters
●Database: PostgresSQL
●System packages: protobuf-compiler, graphviz, etc.
●Browsers for end-to-end tests: Playwright

42. Cache Non-Python Dependency Installation
Common non-Python dependencies:
●Python and Node interpreters
●Database: Postgres
●System packages: protobuf-compiler, graphviz, etc.
●Browsers for end-to-end tests: Playwright


# Dockerfile
FROM … # a base image
RUN sudo apt-get install -y postgresql-16 protobuf-compiler

43. Cache Non-Python Dependency Installation
Common non-Python dependencies:
●Python and Node interpreters
●Database: Postgres
●System packages: protobuf-compiler, graphviz, etc.
●Browsers for end-to-end tests: Playwright

# Dockerfile
FROM … # a base image
RUN sudo apt-get install -y postgresql-16 protobuf-compiler
# After publishing the image
# to a registry

# Github Workflow
Jobs:
run-in-container:
runs-on:ubuntu-latest
container:
image: …

44. Cache Non-Python Dependency Installation
Common non-Python dependencies:
●Python and Node interpreters
●Database: Postgres
●System packages: protobuf-compiler, graphviz, etc.
●Browsers for end-to-end tests: Playwright

# Dockerfile
FROM … # a base image
RUN sudo apt-get install -y postgresql-16 protobuf-compiler
Save 10 minutes or more on each CI run
in a large codebase
https://docs.github.com/en/actions/using-jobs/running-jobs-in-a-container
# After publishing the image
# to a registry

# Github Workflow
Jobs:
run-in-container:
runs-on:ubuntu-latest
container:
image: …

45. ?????? Strategy #3: Skip Unnecessary Computations

46. Skip Unnecessary Tests and Linters
Only run specific tests when only specific code are changed

https://github.com/marketplace/actions/changed-files

47. Skip Unnecessary Tests and Linters
Only run specific tests when only specific code are changed
# Github workflow
jobs:
changed-files:
outputs:
has-py-changes: ${{ steps. find-py-changes.outputs .any_changed }}
runs-on: ubuntu-latest
steps: actions/checkout@v4
- uses: tj-actions/changed-files @44
id: find-py-changes
with:
files: **/*.py

https://github.com/marketplace/actions/changed-files

48. Skip Unnecessary Tests and Linters
Only run specific tests when only specific code are changed
# Github workflow
jobs:
changed-files:
outputs:
has-py-changes: ${{ steps. find-py-changes.outputs .any_changed }}
runs-on: ubuntu-latest
steps: actions/checkout@v4
- uses: tj-actions/changed-files@44
id: find-py-changes
with:
files: **/*.py
run-pytest:
needs: changed-files
if: needs.changed-files.outputs.has-py-changes == 'True'
steps:
- run: pytest

https://github.com/marketplace/actions/changed-files

49. Skip Unnecessary Tests and Linters
Only run specific tests when only specific code are changed
# Github workflow
jobs:
changed-files:
outputs:
has-py-changes: ${{ steps. find-py-changes.outputs .any_changed }}
runs-on: ubuntu-latest
steps: actions/checkout@v4
- uses: tj-actions/changed-files@44
id: find-py-changes
with:
files: **/*.py
run-pytest:
needs: changed-files
if: needs.changed-files.outputs.has-py-changes == 'True'
steps:
- run: pytest

https://github.com/marketplace/actions/changed-files

50. Only run specific tests when only specific code are changed
# Github workflow
jobs:
changed-files:
outputs:
has-py-changes: ${{ steps. find-py-changes.outputs .any_changed }}
runs-on: ubuntu-latest
steps: actions/checkout@v4
- uses: tj-actions/changed-files@44
id: find-py-changes
with:
files: **/*.py
run-pytest:
needs: changed-files
if: needs.changed-files.outputs.has-py-changes == 'True'
steps:
- run: pytest

Skip Unnecessary Tests and Linters
?????? Can also only runs on updated files in linters
✨ Modularize code and use build systems to run even fewer tests
https://github.com/marketplace/actions/changed-files

51. Skip Coverage Analysis for Unchanged Files
# pytest --cov by default measures coverage for all files
and it’s slow in a large codebase

# Add --cov=UPDATED_PATH1 --cov=UPDATED_PATH2 … to only
measure the updated files

52. Skip Coverage Analysis for Unchanged Files
# pytest --cov by default measures coverage for all files
and it’s slow in a large codebase

# Add --cov=UPDATED_PATH1 --cov=UPDATED_PATH2 … to only
measure the updated files
Save 1 minute or more on each CI run in a
large codebase

53. ?????? Strategy #4: Modernize Runners

54. Use Faster and Cheaper Runners
Utilize next-generation instances with optimized CPU and memory
configurations to achieve faster and more cost-effective
performance

55. Use Faster and Cheaper Runners
Utilize next-generation instances with optimized CPU and memory
configurations to achieve faster and more cost-effective
performance




Or use 3rd-party-hosted runner providers:
●Namespace
●BuildJet

56. Use self-hosted runners with auto-scaling
https://github.com/actions/actions-runner-controller/
Use Actions Runner Controller to deploy auto-scaling runners on Kubernetes
with custom hardware specifications (e.g. AWS EC2)

57. Use self-hosted runners with auto-scaling
https://github.com/actions/actions-runner-controller/
Use Actions Runner Controller to deploy auto-scaling runners on Kubernetes
with custom hardware specifications (e.g. AWS EC2)

Achieved 5x+ cost savings and 2x+ faster test speeds compared to using
GitHub-hosted runners

58. Rujul Zaparde
Co-Founder and CEO
Continuously optimizing CI test execution time to improve
developer experiences
Results

59. Rujul Zaparde
Co-Founder and CEO
Continuously optimizing CI test execution time to improve
developer experiences
Results
Increasing test coverage with
beer quality assurance

60. Recap: ?????? Strategies for Scaling Slow Tests
in a Large Codebase
Parallel Execution01
02
03
04
Cache
Skip Unnecessary Computations
Modernize Runners

61. Rujul Zaparde
Co-Founder and CEO
Lu Cheng
Co-Founder and CTO
Engineering Blog
hps://engineering.ziphq.com

Job Opportunities
hps://ziphq.com/careers

Thank You!