Hyperdex - A closer look

2. HyperDex
A Closer Look

3. Dr. Stefan Schadwinkel
Co-Founder / Analytics Engineer
[email protected]
– DECK36 is a young spin-off from
ICANS
– Small team of 7 engineers
– Longstanding expertise in
designing, implementing and
operating complex web systems
– Developing own data intelligence-
focused tools and web services
– Offering our expert knowledge in:
– Automation & Operations
– Architecture & Engineering
– Analytics & Data Logistics

4. BACKGROUND
*log: Storm-based Analytics RT

5. BACKGROUND
*log
Our *log provides stream-based real-time analytics. We need a serious DB.
We need to focus on servicing each request, scale easily & fast, throughput must be
consistent, we need secondary indices, and the possibility to compute aggregations.

MongoDB, Cassandra, Riak, MariaDB

HyperDex: A Distributed, Searchable Key-Value Store.
Robert Escriva, Bernard Wong and Emin Gün Sirer.
In Proceedings of the SIGCOMM Conference, Helsinki, Finland, August 2012.
http://hyperdex.org/papers/hyperdex.pdf

6. WHY HYPERDEX?
Next Generation K/V

7. WHY HYPERDEX?
Features.
CAP - Common Buzz: Consistent, Available, Partition-tolerant – Pick any two.
From http://hyperdex.org/FAQ/: HyperDex is designed to withstand a threshold of
failures desired by the application. The level of fault-tolerance is tunable by the system
administrator. HyperDex guarantees consistency, availability in the presence of less
than f faults, and partition tolerance for partitions that affect less than f nodes, where f
is a user-tunable parameter.

- Fully linearizable. Every ‘get’ always returns the latest ‘put’.
- Tolerates up to f failures.
- Query secondary attributes almost as fast as the primary key.
- Rich data types: Strings, Floats, Ints, Lists, Maps, Sets
- Atomic, multi-key transactions. (Commercial)

8. HOW?
Hyperspace Hashing

9. HYPERSPACE HASHING
Mapping Data into Euclidean Space
Each object is mapped into space. Space is mapped onto servers.
One hyperspace relates to one table. HyperDex can manage multiple independent
hyperspaces.

10. HYPERSPACE HASHING
So far, so good. Aww, wait!
The curse of dimensionality.
The volume of the resulting hyperspace grows
exponentially in the number of dimensions/
attributes.

For instance, a table with 9 dimensions requires 2
9
regions. That’s a minimum of 512 servers.

11. HYPERSPACE HASHING
Logarithms to the rescue!
Subspaces.
HyperDex splits the hyperspace into multiple lower dimensional subspaces. Thus, the
volume of the space only grows linearly. Not only does this reduce the number of
machines required to store the data, search becomes more efficient, because less
machines need to be contacted. A key subspace is added to distinguish key lookup
from single attribute searches. Each subspace stores a full copy of the object.

12. HOW?
Value-dependent Chaining

13. VALUE-DEPENDENT CHAINING
Consistency and Replication.
We have copies of each object in each subspace.
Value-dependent chaining keeps all copies consistent and provides strong consistency
(linearizability) and fault tolerance in the presence of concurrent updates.

14. VALUE-DEPENDENT CHAINING
Consistency.
HyperDex propagates each update deterministically to all relevant spaces.
Update u1: PUT (insert key)
-  h1, h2, h3

Chains are executed from the
end.

Head = Point leader. The same
for each key.

The point leader knows all
updates. Dependencies are
embedded in the chain.

15. VALUE-DEPENDENT CHAINING
Replication.
HyperDex inserts replicas for each region into the chain.
Consider Update u1:
-  h1, h2, h3
-  h1, h1‘, h2, h2‘, h3, h3‘
-  h1, h1‘, h2‘, h2‘‘, h3, h3‘
Replicas are always updated
first.

Failures do not compromise
strong consistency.

Clients are only acknowledged
after full replication is achieved.

16. THE PARTS OF THE MACHINE
HyperDex - Nuts and Bolts

17. THE PARTS OF THE MACHINE
The Slave Node.
Everything is C++.
The slave nodes are not particularly interesting.

18. THE PARTS OF THE MACHINE
The Coordinator & the Configuration.
A logically centralized coordinator maintains global state.
-  Own replicated state machine for the coordinator called “replicant”. This is what
Zookeeper does for Hadoop et al.
-  Global state is maintained as Configuration.
-  The coordinator has no state of the stored objects, only mappings and servers.
-  Instance: IP, Port, Instance ID.
-  The coordinator creates new configurations based on changes and failures and
distributes it to the client.

19. THE PARTS OF THE MACHINE
The Client.
The client is part of the whole system, not just a customer.
-  Client receives new configurations from the coordinator.
-  Switching to a new configuration is atomic.
-  Client only contacts relevant nodes. This is significant for performance.
-  Clients must be “intelligent”. No REST.
-  A load-balancing proxy layer could help. But isn’t there.
-  Full support for C++, Python.
-  Partial support for Java (uses the C++ driver through JNI), Node.JS, Ruby
-  Using layers skips features. Java driver doesn’t support “count”.

20. THE REAL WORLD™
HyperDex Tutorial

21. THE REAL WORLD ™
Install.
Pre-build packages.
Supports CentOS, Debian, Fedora, Ubuntu.
But not all versions.
And not everything. Read: “No package for the Java driver”.

Build from source.
Good luck.

Be super conscious of package versions.
More on that in a minute.

22. THE REAL WORLD ™
Start the Daemons.
Coordinator.
# hyperdex coordinator -f -l 127.0.0.1 -p 1982

Data Nodes.
# hyperdex daemon -f --listen=127.0.0.1 --listen-port=2022 \
--coordinator=127.0.0.1 --coordinator-port=1982 \
--data=./data0/

# hyperdex daemon -f --listen=127.0.0.1 --listen-port=2032 \
--coordinator=127.0.0.1 --coordinator-port=1982 \
--data=./data1/

23. THE REAL WORLD ™
Client Demo
The Python client is the HyperDex shell. Create Hyperspace.
# python

24. THE REAL WORLD ™
Client Demo
Create a client. Basic PUT/GET. Uses Key subspace.
# python

25. THE REAL WORLD ™
Client Demo
Search. Uses further subspaces.

26. THE REAL WORLD ™
Client Demo
Updates and Range Query/Search.

27. THE REAL WORLD™
Performance

28. THE REAL WORLD ™
Bashing the Prophetess & the Giant.
Performance Benchmarks use the YCSB against Cassandra and MongoDB.
Dedicated cluster of 14 Nodes in the VICCI cloud.
Take it with a grain of salt.
I’m missing Riak.

29. THE REAL WORLD ™
Throughput.

30. THE REAL WORLD ™
Latency.

31. THE REAL WORLD ™
Latency.

32. THE REAL WORLD ™
Scaling.

33. THE REAL WORLD™
Experiences & Findings

34. THE REAL WORLD ™
Experiences. Findings.
Minor versions are incompatible.
-  hyperdex-1.0.rc4 vs. hyperdex-1.0.rc5
-  import hyperclient vs. hyperdex.admin, hyperdex.client
-  (hyperdisk) vs. leveldb vs. hyperleveldb

-  There goes my dream of using the PHP driver on github.
-  Migration? No idea.
-  Compile? Use VM to go.

35. THE REAL WORLD ™
Experiences. Findings.
It’s just a K/V store.
-  No methods to do distributed computations. Python map/reduce is on the agenda.
No Dynamo Ring. But a chain to rule them all.
-  Fault-tolerance with f dedicated nodes is fine, but what about multiple datacenters?
It’s a quite young project with few committers.
Important internals change between minor versions.
Not much sleep for them. How about your DevOps?

36. REMEMBER?
*log: Storm-based Analytics RT

37. WHAT ABOUT?
*log
We chose Riak.
-  Excellent Java driver.
-  We don’t need transactions.
-  During development, our schema will change often.
-  Operational ease, easy to scale, excellent feedback.
-  Map/reduce in Erlang and JS. Can use the result of a secondary index query.
-  Solr Integration with Riak Search. Not at the moment, but we deal with content.
We like HyperDex.
-  Really interesting concepts and advancements, but atm not the perfect fit.
-  Implemented a storage backend abstraction layer. Easy to switch to HyperDex once
its more mature.

38. Thank You.