Introduction to Spark (Intern Event Presentation)

An introduction to Apache Spark from its creator, Matei Zaharia, for the intern event hosted by Databricks.

1. Introduction to Spark
Matei Zaharia
Databricks Intern Event, August 2015

2. What is Apache Spark?
Fast and general computing engine for clusters
Makes it easy and fast to process large
datasets
•APIs in Java, Scala, Python, R
•Libraries for SQL, streaming, machine learning, …
•100x faster than HadoopMapReducefor some
apps

3. About Databricks
Founded by creators of Spark in 2013
Offers a hosted cloud service built on Spark
•Interactive workspace with notebooks, dashboards, jobs

4. 0
20
40
60
80
100
120
140
160
2010 2011 2012 2013 2014 2015
Contributors
Contributors / Month to Spark
Community Growth
Most active open source project
in big data

5. Spark Programming Model
Write programs in terms of transformations on
distributed datasets
Resilient Distributed Datasets (RDDs)
•Collections of objects stored in memory or disk across a
cluster
•Built via parallel transformations (map, filter, …)
•Automatically rebuilt on failure

6. Example: Log Mining
Load error messages from a log into memory, then
interactively search for various patterns
lines = spark.textFile(“hdfs://...”)
errors = lines.filter(lambda s: s.startswith(“ERROR”))
messages = errors.map(lambda s: s.split(‘\t’)[2])
messages.cache()
Block 1
Block 2
Block 3
Worker
Worker
Worker
Driver
messages.filter(lambda s: “MySQL” in s).count()
messages.filter(lambda s: “Redis” in s).count()
. . .
tasks
results
Cache 1
Cache 2
Cache 3
Base RDDTransformed RDD
Action
Result: full-text search of Wikipedia in
0.5 sec (vs20s for on-disk data)

7. Example: Logistic Regression
0
500
1000
1500
2000
2500
3000
3500
4000
1 5 10 20 30
Running Time (s)
Number of Iterations
Hadoop
Spark
110 s / iteration
first iteration80 s
further iterations1
s
Iterative algorithm used in machine learning

8. Source: Daytona GraySortbenchmark, sortbenchmark.org
2100 machines2013 Record:
Hadoop
72 minutes
2014
Record:
Spark
207
machines
23 minutes
On-Disk Performance
Time to sort 100TB

9. Higher-Level Libraries
Spark
Spark
Streaming
real-time
Spark SQL
structured data
MLlib
machine
learning
GraphX
graph

10. Higher-Level Libraries
// Load data using SQL
points = ctx.sql(“select latitude, longitude from tweets” )
// Train a machine learning model
model = KMeans.train(points, 10)
// Apply it to a stream
sc.twitterStream(...)
.map(lambda t: (model.predict(t.location), 1))
.reduceByWindow(“5s”, lambda a, b: a + b)

11. Demo

12. Over 1000 production users, clusters up to 8000
nodes
Many talks online at spark-summit.org
Spark Community

14. Ongoing Work
Speeding up Spark through code generation
and binary processing (Project Tungsten)
R interface to Spark (SparkR)
Real-time machine learning library
Frontend and backend work in Databricks
(visualization, collaboration, auto-scaling, …)

15. Thank you.
We’re hiring!