MySQL NDB Cluster 101

Introduction into MySQL NDB Cluster 8.0 at MySQL Day before FOSDEM 2020

2. MySQL Cluster
Proven in serving billions of people
every day when making phone calls,
playing online games or handling
financial transactions.

3. Massively linear scale
Always-On 99.9999% Availability
Sharded In-Memory Datasets
Always Consistent
Parallel Real-Time Performance.
Auto-partitioning, data distribution
and replication built-in.
Read- and Write Scale-Out
to many TB on commodity hardware.
Designed for mission critical
systems. Masterless, shared-nothing
with no single point of failure.
Transactional consistency across
distributed and partitioned dataset.
Out of the box straightforward
application programming.
Ease of use
Open Source
Written in C++. Can be used standalone
or with MySQL as a SQL front-end.

4. NDB is the highest throughput
distributed, in-memory, transactional
database in the world.
It’s open source!

5. Horizontal and vertical scale with MySQL Cluster
15TB in-memory per node

100+TB with on-disk
100+TB to 5 PB

in a single cluster
1 - 2GB/sec read/write 

transactions per shard
400k TPM DBT2

per 1 CPU socket
100+ data nodes
1-4 replicas
10k threads for parallel
query processing

6. Cluster drives systems with massive scale and real-time answering times
100M trades
analyzed 

every year
20B $ volume
every day
2+B mobile phone
users 

world wide


100+M in a
single installation
30+M active
gamers

every month
12+M in one
multi-player
> > >

7. MySQL Cluster linear scale
NoSQL Performance
Confidential – Oracle Internal/Restricted/Highly Restricted
Memory optimized tables
Durable
Mix with disk-based tables
Parallel table scans for non-indexed
searches
MySQL Cluster FlexAsych
200M NoSQL Reads/Second
!-!!!!
!50,000,000!!
!100,000,000!!
!150,000,000!!
!200,000,000!!
!250,000,000!!
2!4!6!8!10!12!14!16!18!20!22!24!26!28!30!32!
Reads!per!second!
Data!Nodes!
FlexAsync!Reads!

8. MySQL Cluster Industries
Telecom
Gaming & Massive Parallel Online Games
Financials

9. MySQL Cluster Use Cases
•IOT - applications that write massive amounts of data in short periods of
time
•Shopping carts - durable, fast and frequent updates
•User tracking and monitoring
•Authentication and session management systems - real-time user
verification
•User profile and directories - reliable real-time information store
•Messaging systems

10. Use cluster if you need
Performance
Read and write
scalability
Real-time in-memory
Super low guaranteed
latency
Auto-Sharding
Elastic
Consistency
Always on
Always consistent
ACID
Global consistent view
of the data data
Cross-shard, cross-
replica foreign keys
SQL and NoSQL
Distributed key value
Advanced joins
Parallel distributed
query
Highly Available
Simple HA
Simple programing

11. MySQL Cluster Usage
•Key Value store

+ High Availability, Scale-Out, Durability
•Transactional object store

+ Multi row transactions and consistency
•Relational database

+ SQL joins, foreign keys, triggers, stored procedures,
generated columns, JSON, GIS, …
•

12. Usages

13. Confidential – Oracle Internal/Restricted/Highly Restricted
Steel Belted Radius Carrier Server
Provides centralized Authentication, Authorization, and
Accounting (AAA or Triple A) management for users who
connect and use a network service.


https://www.juniper.net/documentation/en_US/sbr-carrier8.3.0
Juniper
Radius Server
Switch
Scalable Session State Register
MySQL Cluster

14. Bredbandsbolaget (B2)
B2 application Nexus to authenticate and grant
customers access to B2 services including
broadband, VoIP and TV broadcasting.
MySQL Cluster for mission critical
authentication and authorisation services.
Up and serving through on-line versions
upgrades and maintenance since 2003!
Real Time Network Operation Platform
Internal Backoffice Systems
Subscriber Profiles
Authentication Services
B2 Core Network

15. Stream processing in the financial industry
Live trading data
Kafka
Messages
Message
Formatting
Message Event Bus
NDB native
Client processors

16. Confidential – Oracle Internal/Restricted/Highly Restricted
Hadoop (HopsFS) 

with MySQL Cluster
NameNodes
Leader
HDFS Client
DataNodes
hops.io
ClusterJ
Small
Files

17. Architecture
Clients
Cluster front-
ends and
connectors
Data Nodes
Partitioning- and distribution engine

18. Deployment examples
Dev setup
App or MySQL Server
connecting directly
locally to single data
node.
High available setup
Co-located app/MySQL
Server and data nodes
using shared memory
connections.
Classical 3-tier setup
Redundant
SQL and NoSQL mixed
access
Multiple SQL nodes and
native node.js accessing
same datasets

19. Sharding and redundancy with
no single point of failure
Multiple copies of data are maintained for availability
A group of data nodes shares the same data
1 - 4 replicas

20. High availability
•Up to 4 replicas per shard
•All replicas active
•Always consistent
•Microsecond failover

21. High availability
•Majority for 3 and 4 nodes
•Arbitration for 2 replicas
•Proudly using 2-phase

22. Lets crash an Availability Zone (AD)
•No service 

interruption
source: LogicalClocks, MySQL Innovation Day, January 23, 2019

23. Synchronous replication locally and asynchronous between physical locations
•With awareness of data locality and availability domains for cloud
Cluster
Data Nodes
Replication
Btw, conflict detection 

and automated resolution.

24. Data distribution
•Auto-sharding and distribution
No name-node or central master
•Each dataset is split into fragments and
distributed across data nodes
•Within a cluster data is always transactionally
consistent
MySQL Cluster Data Nodes
PK Service Data
253 Tiktok xxx
892 Snapchat xxx
253 Discord xxx
739 Instagram xxx

25. Data distribution
•Partition of all data to thousands of
virtual partitions
•Virtual partitions distributed to data
nodes
…
PK Service Data
253 Tiktok xxx
892 Snapchat xxx
253 Discord xxx
739 Instagram xxx

26. Data distribution
•Deterministic, random distribution by default
•Data in up to 32 data node local partitions
•each controlled by their own threads
•key to parallel query processing
PK Service Data
253 Tiktok xxx
892 Snapchat xxx
253 Discord xxx
739 Instagram xxx

27. Data distribution and HA
•Each partition is synchronously replicated to
peer data nodes
•ReplicaSets are called Node Group
•All partitions active for read and write
PK Service Data
253 Tiktok xxx
892 Snapchat xxx
253 Discord xxx
739 Instagram xxx
Node Group Node Group

28. On-line Scaling and Elasticity - Repartitioning
•Virtual partitions re-distributed on-line when adding more data nodes
•Designed to be a slow background process not impacting real-time
performance.

29. On-line Scaling and Elasticity - Repartitioning
•Minimal amount of data moved
•No re-hashing necessary
•Similar to consistent hashing

30. Writing data
writes Data Memory
(RAM)
•Memory is locked so it won’t swap

31. Writing data
•Writes log to disk asynchronously in
background
- key to real-time in-memory transactions
•Partial checkpoints of data memory
- fast node starts and system recovery
Flush writes to disk in
background checkpoints
writes
…
Data Memory
(RAM)

32. Writing data
•Writes go to data memory and commit log
•In-memory data reads never involve disks
Flush writes to disk in
background checkpoints
Commit Log (REDO)
writes
…
time
…
Data Memory
(RAM)

33. Data Node
Multiple Threads per data node
Parallel execution of
… multiple queries from …
… multiple users on …
… multiple MySQL Servers
Communication with signals
Goal: minimize context switching
NIC
Main thread
Local
Data
Managers
Receive
Send
Disk/SSD IO
Data Memory
(RAM)

34. Lock free multi core VM
•Data is partitioned inside the data nodes
•Communication asynchronously, event driven
on cluster’s VM
•No group communication - instead using
Distributed row locks
Non-blocking 2-phase commit
NIC
Main thread
Local
Data
Managers
Receive
Send
Disk/SSD IO
Data Memory
(RAM)

35. Queries in multithreaded NDB Virtual Machine
•Even a single query from MySQL Server executed in parallel
Data Memory
(RAM)
QUERY …

36. Massive parallel system executing parallel queries
Receive Send
Transaction Data Manager
Data Node Data Node
Receive Send
Transaction Data Manager

37. Data distribution awareness
•Key-value with hash on
primary key
•Complemented by
ordered in-memory-
optimised T-Tree
indexes for fast
searches
For PK operations
NDB data partition
is simply calculated
PK Service Data
739 Instagram xxx

38. Consolidated view of distributed data
•Clients and MySQL
Servers see a
consolidated view of
the distributed data
• Joins are pushed down
to data nodes
• Parallel cross-shard
execution in the data
nodes
• Result consolidation in
MySQL Server
Consolidated view of distributed data

39. Parallel cross-partition queries
•Parallel execution on the
data nodes and within
data nodes
•64 cpus per node
leveraged
•parallelizes single queries
•144 data nodes 

x 32 partitions 

= 4608! CPUs
+ 32 other processing
threads per node
•automatic batching, event
driven and asynchronous
PK Service Data
253 Tiktok xxx
892 Snapchat xxx
253 Discord xxx
739 Instagram xxx

40. Parallel cross-partition queries
$ SELECT * FROM services
LEFT JOIN data USING(service)
Data Nodes
Service Data
Snapchat xxx
PK Service
892 Snapchat
PK Service Data
892 Snapchat xxx
… … …
Parallel execution of
single queries
on all data nodes and
within data nodes

41. MySQL Cluster - matured with advanced features
•Disk tables for even bigger data
•IO control for efficient disk usage
•Local domain / availability domain
•Shared memory connections
•Sub-second failover and self-healing
•Online scaling, maintenance and version upgrades
•Heartbeat mechanisms to support nontrivial network failures
•many more …

42. Thank You
Bernd Ocklin
Snr Director
MySQL Cluster Development