Modern binary build systems - PyCon 2024

Modern binary build systems have made shipping binary packages for Python much easier than ever before. This talk discusses three of the most popular build systems for Python packages using the new standards developed for packaging.

1. Henry Schreiner • May 17, 2024 • PyCon US
Modern binary build systems

2. About me
@henryiii
2
Became involved in packaging via Scikit-HEP
Wanted to ship binary extensions
(boost-histogram, awkward, iminuit)
Joined forces with cibuildwheel
Currently maintain 34+ packages
21 in top 8,000 on PyPI
pybind11 (python_example, cmake_example, scikit_build_example) • 

cibuildwheel • build • pipx • pyproject-metadata • 

scikit-build (core, cmake, ninja, moderncmakedomain, example-projects) • 

nox • validate-pyproject(-schema-store) • pytest GHA annotate-failures • 

Plumbum • flake8-errmsg • check-sdist •

boost-histogram • Hist • UHI • Vector • GooFit • Particle • DecayLanguage •
uproot-browser • Conda-Forge ROOT •

Scientific-Python/cookie • repo-review • meson-python • 

POVM • hypernewsviewer •

CLI11 • beautifulhugo • Jekyll-Indico
https://iscinumpy.dev
(Slides will be posted here later)

3. Benefits?
3
Fast
Wrap existing libraries
Cross-language
Ship existing CLI tools on PyPI
Can release GIL
Of the top 8,000 packages on PyPI:
6,433 ship a single wheel
896 ship multiple wheels (probably binaries!)
671 don't ship a wheel (unknown)
Over 10%!
charset-normalizer367 M
pyyaml279 M
numpy246 M
cryptography238 M
cf207 M
pandas195 M
protobuf174 M
markupsafe151 M
wrapt120 M
pyarrow110 M
sqlalchemy104 M
aiohttp102 M
scipy101 M
multidict97 M
psutil96 M
yarl95 M
frozenlist90 M
pillow89 M
grpcio88 M
greenlet88 M
pydantic-core83 M

4. Pure Python packages
4
Build configuration One wheel, one sdist
Binding API Build configuration Many wheels
pybind11 (C++11)
nanobind (C++17)
Cython (C, C++)
SWIG (C, C++)
PyO3 (Rust)
Native (C)
cibuildwheel
maturin-action (Rust)
Scikit-build-core
meson-python
maturin
Binary packages
hatchling
setuptools
poetry-core
build
hynek/build-and-inspect-python-package
flit-core
pdm-backend
...

5. Making the wheels
cibuildwheel on GitHub Actions
5
on: [push, pull_request]
jobs:
build_wheels:
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-latest, windows-latest, macos-13, macos-14]
steps:
- uses: actions/checkout@v4
- uses: pypa/[email protected]
- uses: actions/upload-artifact@v4
with:
name: wheels-${{ strategy.job-index }}
path: ./wheelhouse/*.whl
[tool.cibuildwheel]
test-extras = "test"
test-command = "pytest {project}/tests"

6. cibuildwheel
Local runs
6
pipx run cibuildwheel --only pp310-manylinux_x86_64

7. cibuildwheel
New features
•CPython 3.13b1 (opt-in)
•Inherit for overrides (except config-settings for now)
•Musllinux 1.2 default
•Support Apple Silicon runners on GHA
•Additional flags to build frontend
•Better reproducible build support
•Easier local runs
7
WIP:
Fix config-settings inherit override bug
Free-threaded support (in manylinux now!)
Investigating uv support

8. Before PEP 517 (2017)
Setuptools / distutils
8
from setuptools import Extension, setup
setup(
name = "mylib",
version = "1.0.0",
ext_modules = [
Extension(
name = "mylib.foo",
sources = [ "foo.c"],
),
]
)
Simple! Unless you need…
Third party dependencies
Compiler specific flags, like C++ version
Parallel (file) compiles
Caching / smart recompiles
bdist_wheel customizations (no public API)
Fortran
Other compilers
IDE support
Tooling integration (debuggers, etc)
Cross-compilation
Setuptools isn’t trying to add features to support this
So everyone has to reinvent the wheel
NumPy had 13,000 LoC for building!
(MANIFEST.in omitted)

9. Before PEP 517 (2017)
Extending setuptools / distutils
9
Examples of setuptools-based builders
Scikit-build (classic)
setuptools-rust
pybind11’s setup_helper
cython (integrated)
Distutils -> setuptools has layered complexity
hard to debug
easy to break on update
hard to extend

10. Modern build system
Hooks for build backends
10
Simple standardized API
Supports installing build requirements (dynamically too)
Backend is responsible for basically everything
Editable installs added later

11. Build backend hooks
11
Dynamically request
dependencies per stage
get_requires_for_build_sdist(…)
get_requires_for_build_wheel(…)
get_requires_for_build_editable(…)
build_sdist(…)
build_wheel(…)
build_editable(…)
prepare_metadata_for_build_wheel()
Given source and config,
make *.tar.gz file
And .whl file
And “editable” via .whl file
Optional get metadata
without build

12. Rise of PEP 517 build backends
The new era of Python packaging
12
Now we have binary ones too!
enscons (2017)
maturin (2019)
meson-python (2021)
scikit-build-core (2022)
Pure-Python backends were first:
flit-core (2017*)
poetry-core (2018*)
pdm-backend (2020*)
hatchling (2022)
setuptools (2022)
Special mention: enscons was the first PEP 517 build backend for binaries!
(We’ll focus on the other three today)

13. History of meson-python
13
SciPy started the project in preparation of Python 3.12
Other major Scientific Python libraries joined
NumPy went from 13K to 2K LoC for building
(Not counting forking meson (100K) and vendoring it)

14. Meson-Python
Meson
14
example-project
├── example.cpp
├── pyproject.toml
└── meson.build
[build-system]
requires = ["meson-python", "pybind11"]
build-backend = "mesonpy"
[project]
name = "example"
version = "0.0.1"
#include <pybind11/pybind11.h>
namespace py = pybind11;
float square(float x) { return x * x; }
PYBIND11_MODULE(example, m) {
m.def("square", &square);
}
project(
'example',
'cpp',
default_options: [
'cpp_std=c++11',
],
)
py = import('python').find_installation(pure: false)
pybind11_dep = dependency( 'pybind11')
py.extension_module(
'_core',
'example.cpp',
install: true,
dependencies : [pybind11_dep],
)
pipx run build --installer=uv
(Also requires git set up)

15. Meson-Python
Features
15
Can skip PEP 621 and use Meson config for quick projects
Several options can be specified in pyproject.toml or via config-settings
Developed rebuild support for editable installs
Ninja package added only if missing

16. Scikit-build
A family of tools
•Scikit-build-core
•Scikit-build (classic)
•Ninja Python Distribution (pip install cmake)
•CMake Python Distribution (pip install ninja with jobserver support)
•ModernCMakeDomain (Sphinx plugin)
•Dynamic-Metadata (WIP project for general metadata plugins)
16
Scikit-build-core funded via NSF grant OAC 2209877

17. Scikit-build-core
CMake
17
example-project
├── example.cpp
├── pyproject.toml
└── CMakeLists.txt
[build-system]
requires = ["scikit-build-core", "pybind11"]
build-backend = "scikit_build_core.build"
[project]
name = "example"
version = "0.0.1"
cmake_minimum_required (VERSION 3.15…3.29)
project(example LANGUAGES CXX)
set(PYBIND11_NEWPYTHON ON )
find_package(pybind11 CONFIG REQUIRED)
pybind11_add_module(example example.cpp)
install(TARGETS example LIBRARY DESTINATION .)
pipx run build --installer=uv
#include <pybind11/pybind11.h>
namespace py = pybind11;
float square(float x) { return x * x; }
PYBIND11_MODULE(example, m) {
m.def("square", &square);
}

18. Scikit-build-core
Features
•Automatic inclusion of cmake/ninja as needed
•Limited API / Stable ABI and pythonless tags supported via config option
•Support for writing out to extra wheel folders (scripts, headers, data, metadata)
•Dedicated entrypoints for packages to host CMake code
•Experimental editable mode support, with optional experimental auto rebuilds on import
•Free-threaded Python support
18

19. Scikit-build-core config system
A peek into the internals
19
Dataclass-based config system
Generates JSONSchema
Generates section in README using cog
Supports pyproject.toml, config-settings,
and environment variables
@dataclasses.dataclass
class WheelSettings:
packages: Optional[List[ str]] = None
"""
A list of packages to auto-copy into the wheel. If this is not set, it will
default to the first of ``src/<package>``, ``python/<package>``, or
``<package>`` if they exist. The prefix(s) will be stripped from the
package name inside the wheel.
"""
py_api: str = ""
"""
The Python tags. The default (empty string) will use the default Python
version. You can also set this to "cp37" to enable the CPython 3.7+ Stable
ABI / Limited API (only on CPython and if the version is sufficient,
otherwise this has no effect). Or you can set it to "py3" or "py2.py3" to
ignore Python ABI compatibility. The ABI tag is inferred from this tag.
"""
expand_macos_universal_tags: bool = False
"""
Fill out extra tags that are not required. This adds "x86_64" and "arm64"
to the list of platforms when "universal2" is used, which helps older
Pip's (before 21.0.1) find the correct wheel.
"""

20. Scikit-build-core advanced config
Over 40 options
20
[tool.scikit-build]
minimum-version = "0.2"

cmake.verbose = true
logging.level = "INFO"

build-dir = "build/{wheel_tag}"

sdist.include = ["src/some_generated_file.txt" ]
sdist.reproducible = false

wheel.expand-macos-universal-tags = true

install.components = [ "python"]

[tool.scikit-build.cmake.define]
SOME_DEFINE = {env="SOME_DEFINE", default="EMPTY"}

21. Scikit-build-core
Dynamic metadata
21
[tool.scikit-build.metadata.version]
provider = "scikit_build_core.metadata.regex"
input = "src/mypackage/__init__.py"
[[tool.scikit-build.generate]]
location = "install"
path = "mypackage/_version.py"
template = '''
version = "${version}"
'''
Join us on scikit-build/dynamic-metadata if you are
interested in generalizing for multiple backends!

22. Scikit-build-core
Overrides
22
[[tool.scikit-build.overrides]]
if.platform-system = "darwin"
cmake.version = ">=3.18"
Designed after cibuildwheel's overrides, inspired by mypy
Match top to bottom
Can use if.any
Can inherit with "append", "prepend", or "none"
Takes regex, specifier set, or bool
[[tool.scikit-build.overrides]]
if.state = "editable"
build-dir = "build"
[[tool.scikit-build.overrides]]
if.any.env.SET_FOO = true
if.any.env.FOO_SET = true
inherit.cmake.define = "append"
cmake.define.FOO = "1"

23. Nanobind example
Using scikit-build-core
23
# CMakeLists.txt
cmake_minimum_required (VERSION 3.15...3.26)
project(nanobind_example LANGUAGES CXX)
find_package(Python 3.8
  REQUIRED COMPONENTS Interpreter Development.Module
  OPTIONAL_COMPONENTS Development.SABIModule)
find_package(nanobind CONFIG REQUIRED)
nanobind_add_module(
  nanobind_example_ext
  STABLE_ABI
  NB_STATIC
  src/nanobind_example_ext.cpp
)
install(TARGETS nanobind_example_ext LIBRARY DESTINATION nanobind_example)
# pyproject.toml
[build-system]
requires = [
  "scikit-build-core",
  "nanobind",
]
build-backend = "scikit_build_core.build"
[tool.scikit-build]
wheel.py-api = "cp312"
[tool.cibuildwheel.macos.environment]
MACOSX_DEPLOYMENT_TARGET = "10.14"
[project]
name = "some-package"
version = "1.0.0"

24. Scikit-build-core
Some projects
24
Rapids.ai
Added to file generator and
deployed across all projects
[tool.scikit-build]
wheel.packages = ["zmq"]
wheel.license-files = [ "licenses/LICENSE*"]
cmake.version = ">=3.15"
# only build/install the pyzmq component
cmake.targets = ["pyzmq"]
install.components = [ "pyzmq"]
PyZMQ
Added to file generator and
deployed across all projects
[tool.scikit-build]
minimum-version = "0.8"
build-dir = "build/{wheel_tag}"
cmake.version = "" # We are cmake, so don't request cmake
ninja.make-fallback = false
wheel.py-api = "py3"
wheel.expand-macos-universal-tags = true
wheel.install-dir = "cmake/data"
[[tool.scikit-build.generate]]
path = "cmake/_version.py"
template = '''
version = "${version}"
'''
cmake (python distributions)
Fun meta problem: we are cmake!

25. Bonus: Scikit-build-core plugins
Scikit-build-core is also useful for creating plugins!
25
[build-system]
requires = ["hatchling", "scikit-build-core~=0.9.0" ]
build-backend = "hatchling.build"
[project]
name = "hatchling_example"
version = "0.1.0"
[tool.hatch.build.targets.wheel.hooks.scikit-build]
experimental = true
This will add a CMake extension to a hatchling project!

26. PyPI stats
Just download every pyproject.toml on PyPI
26
https://github.com/henryiii/pystats & https://hugovk.github.io/top-pypi-packages
maturin: 1,744
scikit_build_core.build: 222
mesonpy: 123
maturin
#81 pydantic-core 82,784,026
#108 rpds-py 61,448,636
#250 pendulum 26,584,509
#298 tokenizers 22,436,866
#330 orjson 19,225,277
#402 safetensors 15,136,797
#511 ruff 9,718,295
#663 cramjam 6,773,950
#664 watchfiles 6,741,022
#732 nh3 6,193,025
#790 polars 5,731,171
#810 uv 5,443,201
#875 dbt-extractor 4,830,288
#1124 jellyfish 2,835,548
#1319 deltalake 2,017,365
#1375 py-spy 1,827,491
#1766 tlparse 1,126,435
#2001 pywinpty 871,147
#2158 y-py 750,010
#2269 clarabel 686,786
#2338 kornia-rs 648,550
#2901 regress 403,177
#3165 rtoml 327,378
#3309 tsdownsample 302,822
#3348 grimp 294,794
#3386 chia-rs 287,786
#3525 openlineage-sql 269,605
#3694 h3ronpy 253,144
#3737 cmsis-pack-manager 247,677
#3989 blake3 218,936
#4168 pyxirr 200,494
#4449 sqlglotrs 175,146
#4481 akinator-py 173,092
#4565 hf-transfer 165,738
#5479 clvm-tools-rs 125,016
#5699 css-inline 114,355
#5947 mitmproxy-rs 104,179
#6269 py-sr25519-bindings 91,533
#6558 python-calamine 82,461
#6626 qh3 80,379
#6674 mitmproxy-wireguard 79,065
#6712 deptry 78,113
#6907 polars-lts-cpu 73,655
#7019 minify-html 70,966
#7039 solders 70,434
#7042 topgrade 70,373
#7240 typos 66,022
#7313 tzfpy 64,947
#7512 netifaces2 61,198
#7516 rjieba 61,141
#7566 lintrunner 60,240
#7647 py-bip39-bindings 58,624
#7698 py-ed25519-zebra-bindings 57,611
#7752 uuid-utils 56,790
scikit-build-core
#229 pyzmq 30,543,660
#645 lightgbm 7,046,707
#709 cmake 6,369,474
#1771 phik 1,118,403
#2090 clang-format 797,795
#3005 awkward-cpp 369,071
#3544 llama-cpp-python 267,061
#4047 openexr 213,581
#4636 coreforecast 160,430
#4711 sparse-dot-topn 154,874
#5685 laszip 114,988
#6299 iminuit 90,796
#6594 spglib 81,422
meson-python
#18 numpy 245,757,478
#26 pandas 195,143,812
#61 scipy 100,730,066
#107 scikit-learn 62,470,305
#112 matplotlib 59,187,475
#208 contourpy 33,391,835
#437 scikit-image 13,129,903
#475 pywavelets 11,126,522
#1781 pygobject 1,104,329
#1875 meson-python 988,201
#1924 scs 934,563
#4505 dftd3 170,689
#5410 dbus-python 129,002
(Doesn't include RAPIDS.ai packages like cudf)

27. PyPI Stats
Looking at every pyproject.toml
27
tool.*:
poetry: 52183
black: 24730
pytest: 18229
isort: 18111
setuptools: 15210
mypy: 13030
ruff: 12213
coverage: 10749
setuptools_scm: 7006
hatch: 6900
pylint: 4261
pyright: 3635
flit: 3039
flake8: 2216
pdm: 1995
tox: 1577
semantic_release: 1553
cibuildwheel: 1474
codespell: 1431
maturin: 1207
poetry-dynamic-versioning: 1150
Can look at the most
popular tool sections:
This helped black
check the impact
of a config bug!
tool.black.line-length contents:
120: 5755
88: 3578
79: 2876
100: 2714
80: 834
...
Can check to see what
our users are using and
setting!
*:
build-system: 136738
tool: 100937
project: 51268
options: 305
tools: 279
metadata: 202
coverage: 148
mypy: 127
requires: 119
flake8: 119
virtualenvs: 90
build-backend: 81
pytest: 76
dependencies: 67
bdist_wheel: 40
build: 36
pypi: 34
build_system: 32
dev-dependencies: 30
package: 30
bumpver: 28
manageprojects: 25
too: 25
...
Or typos:
Nothing past
here is valid!
(Use validate-pyproject)!

28. PyPI Stats
Scikit-build-core usage
28
tool.scikit-build.*.*:
metadata.version: 83
build-dir: 78
wheel.packages: 76
sdist.include: 69
minimum-version: 68
sdist.exclude: 60
cmake.define: 57
wheel.expand-macos-universal-tags: 54
wheel.py-api: 53
cmake.verbose: 47
cmake.minimum-version: 44
cmake.build-type: 43
cmake.args: 32
wheel.install-dir: 32
cmake.version: 30
logging.level: 29
wheel.license-files: 20
install.components: 20
cmake.targets: 17
experimental: 15
ninja.version: 14
ninja.make-fallback: 14
ninja.minimum-version: 13
cmake.source-dir: 12
sdist.cmake: 12
sdist.reproducible: 9
install.strip: 9
generate: 9
wheel.exclude: 6
editable.mode: 5
strict-config: 5
metadata.readme: 5
editable.verbose: 4
editable.rebuild: 4
wheel.platlib: 3
backport.find-python: 3
wheel.cmake: 2
wheel.build-tag: 2
metadata.scripts: 2
overrides: 2
metadata.optional-dependencies: 1
tool.scikit-build.wheel.py-api contents:
{}: 146661
'cp312': 30
'py3': 10
'py2.py3': 7
'cp310': 1
'py37': 1
'cp37': 1
tool.scikit-build.minimum-version contents:
{}: 146646
'0.4': 18
'0.8': 11
'0.5': 9
'0.5.1': 6
'0.8.1': 5
'0.9': 4
'0.3': 2
'0.5.0': 2
'0.6.1': 2
'0.7': 2
'0.2.1': 2
'0.2': 1
'0.6': 1
'0.8.2': 1
'0.4.4': 1
'0.3.0': 1

29. Maturin
Cargo (Rust)
29
A Rust-based build system for Cargo
example-project
├── src/lib.rs
├── Cargo.toml
└── pyproject.toml
[package]
name = "package"
version = "0.1.0"
edition = "2021"
[lib]
name = "example"
crate-type = ["cdylib"]
[dependencies.pyo3]
version = "0.21.1"
features = [
"extension-module",
"abi3-py38",
"experimental-declarative-modules" ,
]
use pyo3::prelude::*;
#[pymodule]
mod example {
use super::*;
#[pyfunction]
fn add(x: i64, y: i64) -> i64 {
x + y
}
}
[build-system]
requires = ["maturin~=1.5"]
build-backend = "maturin"
[project]
name = "example"
dynamic = ["version"]
pipx run build —installer=uv
maturin build --release

30. Maturin
Features
30
Written in Rust
(can build a package without running Python!)
Limited API doesn't even require Python at all!
Cargo's build-from-source packaging is perfect
for cross-compilation and making wheels!
CLI can create a new project.
Why Rust?
Rust/Cargo makes using
libraries as easy as Python!
Cross-compiles are great
(can avoid most manylinux issues)
Tooling is fantastic
(linter, formatter, IDEs, etc)
PyO3 bindings are great
Single binary approach
works well for wheels
(And maybe some memory safety stuff)

31. Maturin-Action
GitHub Action for Building
31
Great at cross-compiling with extensive support
(fast)
Builds non-Python Rust parts once for all versions
Every wheel supported except musl s390x wheels
(some platforms like 32-bit musl can't be done with
cibuildwheel since rust doesn't build cargo for them)
...
Compiling lexical-sort v0.3.1
Compiling taplo v0.13.0
Compiling pep508_rs v0.6.0
Compiling pyproject-fmt-rust v1.1.1 (/home/runner/...
Finished release [optimized] target(s) in 13.53s
! Including files matching "rust-toolchain.toml"
! Built wheel for abi3 Python ≥ 3.8 to dist/...
⚠ Warning: PyPy does not yet support abi3 so the build...
Compiling pyo3-build-config v0.21.2
Compiling pyo3-macros-backend v0.21.2
Compiling pyo3-ffi v0.21.2
Compiling pyo3 v0.21.2
Compiling pyo3-macros v0.21.2
Compiling pyproject-fmt-rust v1.1.1 (/home/runner/...
Finished release [optimized] target(s) in 4.25s
! Including files matching "rust-toolchain.toml"
! Built wheel for PyPy 3.8 to dist/pyproject_fmt_rust-...
Compiling pyo3-build-config v0.21.2
Compiling pyo3-macros-backend v0.21.2
Compiling pyo3-ffi v0.21.2
Compiling pyo3 v0.21.2
Compiling pyo3-macros v0.21.2
Compiling pyproject-fmt-rust v1.1.1 (/home/runner/...
Finished release [optimized] target(s) in 4.24s
...

32. Resouces
Scientific-Python
Development Guide
32
Useful for all projects
(not just scientific ones!)
Up-to-date guides
Integrated template rendered
in some places via cog
Versions updated by CI
Integrated WASM repo-review

33. Resources
Scientific-python/cookie
33
11 backends
Cookiecutter & copier supported
In sync with the dev guide
Generation tested by nox
# The name of your project
myprog
# The name of your (GitHub?) org
henryiii
# The url to your GitHub or GitLab repository
https://github.com/henryiii/myprog
# Your name
Henry Schreiner
# Your email
[email protected]
# A short description of your project
A great package.
# Select a license
BSD
# Choose a build backend
Scikit-build-core - Compiled C++ (recommended)
# Use version control for versioning
No
Copying from template version 2024.4.23
create .
create CMakeLists.txt
create .pre-commit-config.yaml
create pyproject.toml
create tests
create tests/test_package.py
create tests/test_compiled.py
create .git_archival.txt
create LICENSE
create docs
create docs/conf.py
create docs/index.md
create README.md
create .gitignore
create .github
create .github/workflows
create .github/workflows/cd.yml
create .github/workflows/ci.yml
create .github/CONTRIBUTING.md
create .github/dependabot.yml
create .gitattributes
create .copier-answers.yml
create noxfile.py
create .readthedocs.yaml
create src
create src/main.cpp
create src/myprog
create src/myprog/__init__.py
create src/myprog/_core.pyi
create src/myprog/py.typed

34. Resources
Repo-review
34
Python 3.10+ framework
sp-repo-review: checks from guide
Supports WebAssembly and CLI
(and pre-commit, etc)
A checker for your checker config!
Checks linked to integrated tags in guide
Validate-pyproject also integrated

35. Summary
So much has happened in the last two years!
•Three fantastic build-backends are now available for binary projects
•Great resources available for developing packages
•You can ship C/C++/Rust code for all major platforms even for a small project
35

36. Credits and thanks
36
scikit-build team
Jean-Christophe Fillion-Robin (Kitware)
Matt McCormick (Kitware)
Cristian Le
meson-python team
Ralf Gommers
Daniele Nicolodi
Henry Schreiner
Thomas Li
Filipe Laíns (Emeritus)
cibuildwheel team
Joe Rickerby
Yannick Jadoul
Matthieu Darbois
Grzegorz Bokota
And thanks to the maturin developers for that fantastic tool!
Thanks to my other team members!