Why We’re Rewriting SQLite in Rust by Glauber Costa

As we were adding Vector Search to SQLite, we had a crazy idea. What could we achieve if we were to completely rewrite SQLite in Rust? This talk explains what drove us down this path, how we’re using deterministic simulation testing to ensure the reliability of the Rust rewrite, and the lessons learned (so far). I will show how a reimagining of this iconic database can lead to performance improvements of over 500x in some cases by looking at what powers it under the hood.

1. A ScyllaDB Community
Why We’re Rewriting
SQLite in Rust
Glauber Costa
CEO

2. Glauber Costa

Founder/CEO - Turso
■Linux Kernel contributor
■VP Field Engineering at ScyllaDB
■Staff Engineer at Datadog
■Founder of Turso

3. Why Rust?

4. Why Rust specifically?
It could have worked in Zig.
Ultimately, whatever works.
Team was very familiar with Rust.
But ultimately, the borrow checker!

5. Why rewrite the most deployed
database in the solar system?

6. Can the best be better?
■SQLite is great for reads, but writes are painful: single writer.
■SQLite is hard to extend. Extensions exist, but:
■Not memory-safe,
■Can’t add new index types,
■Can’t add new data types and affinities,
■Can’t influence the language syntax.
■Lacks modern features like CDC, Encryption, Vector Search.

7. Can the best be better?
■SQLite is great for reads, but writes are painful: single writer.
■SQLite is hard to extend. Extensions exist, but:
■Not memory-safe,
■Can’t add new index types,
■Can’t add new data types and affinities,
■Can’t influence the language syntax.
■Lacks modern features like CDC, Encryption, Vector Search.

8. SQLite - Open Source, Not Open Contribution
Small and restricted team of contributors.
Contributions not accepted, or rarely accepted.
Test suite is legendary, but also proprietary.

9. Every rewrite is an opportunity
■New async architecture.
■Concurrent writes, solving the notorious “database is locked” problem.
■Deterministic Simulation Testing from day1 to match SQLite’s reliability.

10. This is a performance conf!

11. Async
Multithreading
Incrementalism

12. Asynchronous interface
SQLite’s C API is synchronous
Meaning I/O is synchronous.

Turso on Linux: io_uring: asynchronous fsync.

13. MVCC

14. MVCC (Multi-Version Concurrency Control)
●MVCC: Optimistic concurrency control.
●Transactions that do not conflict can proceed in parallel.
●Conflict resolution at COMMIT time.
●Async fsync makes sure the next transactions can start already.

15. MVCC in Turso
●Lifting the single-writer limitation.
●Concurrent transactions can progress independently until they commit.

16. SQLite parallel write behavior
●Writing batches of 10 rows, with a 1ms think time in the transaction:

BEGIN;
Insert <row1>
…
call_an_api(1ms);
…
INSERT <row10>
COMMIT;

17. SQLite vs Turso write throughput
10 rows per transaction
1 ms simulated think time

18. SQLite write throughput

19. Multi-threading

20. Threads are… fine ?

21. The hierarchy of evil
Def.Evil
Maybe a bit evil
Not Evil

Threads

22. Opening a new connection: O(N) on tables.

23. Opening a new connection: O(N) on tables.

24. Opening a new connection: O(1) on tables.

25. Opening a new connection: O(1) on tables.

26. Incrementalism

27. SQLite schema changes
sqlite> create table foo(bar);
sqlite> create index bar_i on foo(bar);
sqlite> select * from sqlite_master;
table|foo|foo|2|CREATE TABLE foo(bar)
index|bar_i|foo|3|CREATE INDEX bar_i on foo(bar)
-- a new change now will reparse the entire schema!

28. Turso schema changes are incremental

29. Live Materialized Views

30. How Materialized Views work

31. Materialized Views on an in-process database?

32. DBSP

33. DBSP

34. Conclusion:
●We believe SQLite needs a facelift to power modern applications.
●Turso is an Open Contribution rewrite of SQLite.
●Even on performance, there is a lot of room for improvement.
●Perf improvements are architectural, not language-derived.
●Foundation is: Async I/O, multithreading, and incrementalism.

35. Thank you! Let’s connect.
Glauber Costa
[email protected]
@glcst
blog.turso.tech
github.com/tursodatabase/turso