Ingesting in Rust by Armin Ronacher from Sentry

At Sentry we handle hundreds of thousands of events a second — from tiny metric to huge memory dump. What started as a small Django application had to be scaled up to support global points of presence, lower latency and significantly higher throughput. This talk walks through our experience building a Rust based ingestion service to replace our Python one, the lessons we have learned and where we made changes from the original design.

1. Ingesting in Rust Armin Ronacher Principal Architect at Sentry

2. Armin Ronacher ( he/him) Principal Architect at Sentry I created Flask I love Queues, Pipelines and everything related Away from work I juggle three kids (not literally)

3. Setting the Stage

4. Errors

5. Replays

6. Tracing

7. Profiling

8. Rust at Sentry

9. Why is there Rust to Begin With? Initially personal interest, unrelated to Sentry Sentry was built on Python Good FFI between Rust and Python meant extension modules were an option Allowed efficient code sharing of performance critical code Source map library in Rust, exposed to Python

10. Rust in the Pipeline Pieces we will cover here Relay (Ingestion / Aggregation Service) Symbolicator (Stackwalk, Symbolize, Source Maps) Python Modules (Shared Code) * * a lot of it is forward looking

11. Did it work?

12. Rust Again? Short: w ould pick it again Benefits are obvious Downsides are obvious too It might work for you, it might not

13. The Choice is Obvious Today

14. Ingestion Flow SDK Relay Sentry Envelope Event Envelope Project Config Rate Limits

15. Building Relay

16. Relay Goals What it has to do: Efficiently Enforce Quotas Geo-distributed deployments Perform Sampling Data Format Conversion and Forwarding Perform Metrics Extraction PII Stripping

17. Picking Rust Sentry’s language stack was Python originally We acquired some Rust experience via out CLI Built a Python extension module reusing CLI code successfully Liked the idea of reusing code across our stack

18. How to Relay Sentry’s /store/ endpoint was written in Python “Anything goes” (Bad for Rust) No schema, inconsistent behavior Customers sent custom data Various quirks (submit via transparent pixel) Unreasonably lenient

19. Relay as Library Fix up existing protocol where possible Write new protocol normalizer in Rust, expose to Python Side-by-side run old and new normalization, compare results Learnings: serde has a lot of flaws

20. Serde Limits Rather low recursion limits Old Python JSON code accepted Infinity/NaN Python <-> Python event submission contained lots of NaNs Workaround: pass over byte stream to replace “NaN” with “0 “ etc. Limited Data Model Serde relies on problematic in-band signalling for data model foreign types Recursion without Limits Complex recovery from errors, not enough state

21. In Band Signalling Large integer “42”: {"$serde_json::private::Number": "42"} https://github.com/serde-rs/serde/issues/1183 https://github.com/serde-rs/serde/issues/1463

22. Footgun: memmove (!?) Rust makes some expensive patterns look cheap Performance Issue #1: Huge Return Types (Large Error Types) Performance Issue #2: Accidental, Unexpected Clones

23. Large Error Types Boxing errors internally creates a significant performance improvement

24. 10000% Speedup Apply offsets to N tokens, accidentally perform memmove over M bytes N times

25. Picking Frameworks

26. Relay first Steps old tokio + actix + actix-web (pre async/await) actix looked appealing: actor framework, resonates actix has no reasonable concept of backpressure management over time became a mess

27. Relay Today Axum Custom service layer async/await (mostly) More explicit back-pressure management

28. PyO3

29. Python Modules Export via PyO3 Rust code to Python Use maturin to create and distribute wheels Very good solution for GIL and borrow management

30. Thank you! Let’s connect. Armin Ronacher Principal Architect at Sentry