AgilData - How I Learned to Stop Worrying and Evolve with On-Demand Schemas

My talk from SYSCON Cloud Expo 2015 NYC

1. How$I$Learned$to$Stop$Worrying$and$Evolve$$
with$On8Demand$Schemas
Dan$Lynn,$CEO$
www.agildata.com,
[email protected],
@danklynn

3. ...I#was#struck#by#lightning...and#bit#by#a#cobra.

6. Water$is$quite$Agile

7. 2001:$The$Agile$
Manifesto$is$
published
Today
Massive Progress

8. 1.Customer#satisfaction#by#rapid#delivery#of#useful#software#
2.Welcome#changing#requirements,#even#late#in#development#
3.Working#software#is#delivered#frequently#(weeks#rather#than#months)#
4.Close,#daily#cooperation#between#business#people#and#developers#
5.Projects#are#built#around#motivated#individuals,#who#should#be#trusted#
6.FaceItoIface#conversation#is#the#best#form#of#communication#(coIlocation)#
7.Working#software#is#the#principal#measure#of#progress#
8.Sustainable#development,#able#to#maintain#a#constant#pace#
9.Continuous#attention#to#technical#excellence#and#good#design#
10.Simplicity—the#art#of#maximizing#the#amount#of#work#not#done—is#essential#
11.SelfIorganizing#teams#
12.Regular#adaptation#to#changing#circumstance
12$Principles$
of$
Agile$$
Development$

10. http://www.gartner.com/newsroom/id/2823619
Mobile market is putting pressure on development to adopt
rapid development
and deployment practices that constantly iterate as
expectations change.
Traditional Development Practices Will Fail….
Agile Development is Essential.
August, 2014

11. But…

12. Questions?

13. Changing,the,database,
schema,late,in,the,
development,tends,to,cause,
wide9spread,breakages,in,
application,software.,
99,Martin,Fowler
http://martinfowler.com/articles/evodb.html

14. ...changing,a,schema,after,
deployment,results,in,
painful,data,migration,
problems,
99,also$Martin,Fowler
http://martinfowler.com/articles/evodb.html

15. …which$can$be$frustrating

17. The$Database$
Must$be$Agile

18. What$makes$a$database$Agile?

19. It’s More Than Schema Migration

20. Ok,$so$what$makes$a$database$Agile?

21. Simple

22. Scalable

23. Streaming

25. CREATE STREAM s_customer_orders AS
SELECT o.order_id, o.order_amount,
c.first_name, c.last_name
FROM orders_stream o
JOIN customers c ON o.customer_id = c.id ;
SQL

26. How?

27. Processing,must,happen,
next$to$the$data

28. AgilData Broker
v_people

partition 0
SELECT a, b
AgilData Broker
v_people

partition 1
SELECT a, b
USE SCHEMA example;


CREATE STREAM s_one AS SELECT a, b FROM v_people;
Local execution Local execution
Processing$Must$Happen$Next$To$The$Data

29. CREATE SCHEMA example WITH 2 PARTITIONS;


CREATE VIEW v_people (

a Person,
b INTEGER
) PARTITION BY Modulus( a.id);
AgilData Broker
v_people

partition 0
AgilData Broker
v_people

partition 1
User-Defined Partitioner
Your$data$views$are$Just$Snapshots$of$Streams

30. http://wearcam.org/dundas-square/2004sep23/cement.jpg

31. Master Slave
Replication
Binary log
files
Be$log8oriented

32. Master Slave
Binary log
files
Binary log
files
Binary log
files
Be$log8oriented

33. Master Slave
Binary log
files
Binary log
files
Binary log
files
Be$log8oriented

34. Binary log
files
Binary log
files
Binary log
files
0 1 2 3 4 5 6 7 8 9 . .
INSERT INTO employees ...
UPDATE employees
SET salary = 1 MILLION DOLLARS
Be$log8oriented

35. AgilData Broker
Stream Input (local WAL or network)
Stream Output (Partition: 0..n)
Stream Output (Partition: 0..n)
Stream Output (WAL or in-memory)
Stream Runtime

User-Defined Functions, Stream Functions, SELECTs

(Optimized by Execution Planner)
Network Out
Logs$Allow$Arbitrary$Stream$Computation

36. Zookeeper
Zookeeper
Zookeeper
AgilData Broker
LEADER - Topic: example Partition: 0
FOLLOWER - Topic: example Partition: 1
AgilData Broker
FOLLOWER - Topic: example Partition: 0
LEADER - Topic: example Partition: 1
Replication
Replication
Logs$get$you$reliability

37. AgilData Cluster
Distributed
Stream Processing
High-Performance
Embedded Storage
Database Versioning
User-Defined Functions

& Extensible Data Types
Intelligent Query Planning Job Scheduling
3rd-party
data stores
Message QueuesEvent Streams
Clients
Applications
(web apps, APIs, mobile)
Analytics
(e.g. Tableau, Spark)
Monitoring & Management

38. The simplest way for enterprises to leverage
big data without having to become experts
in a complex and changing toolset.
Dan Lynn
www.agildata.com
[email protected]
@danklynn
Come see us at
Booth #340!