Live Demo: Introducing the Spark Connector for MongoDB

In this session we will guide you on the principles of MongoDB and Spark and provide examples using the new MongoDB-Spark connector.

1. MongoDB Connector For Spark

2. HDFS Distributed Data

3. Spark Stand Alone YARN Mesos HDFS Distributed Resources

4. YARN Spark Mesos HDFS Spark Stand Alone Hadoop Distributed Processing

5. YARN Spark Mesos Hive Pig HDFS Hadoop Spark Stand Alone D omain Specific Languages

6. YARN Spark Mesos Hive Pig Spark SQL Spark Shell Spark Streaming HDFS Spark Stand Alone Hadoop

7. YARN Spark Mesos Hive Pig Spark SQL Spark Shell Spark Streaming Spark Stand Alone Hadoop

8. Stand Alone YARN Spark Mesos Spark SQL Spark Shell Spark Streaming

9. Stand Alone YARN Spark Mesos Spark SQL Spark Shell Spark Streaming

10. executor Worker Node executor Worker Node Master Spark Connector Driver Application

11. Parellelize Parellelize Parellelize Parellelize

12. Parellelize Parellelize Parellelize Parellelize Transform Transform Transform Transform

13. Transformations filter( func ) union( func ) intersection( set ) distinct( n ) map( function )

14. Parellelize Parellelize Parellelize Parellelize Transform Transform Transform Transform Transform Transform Transform Transform

15. Parellelize Parellelize Parellelize Parellelize Transform Transform Transform Transform Transform Transform Transform Transform Action Action Action Action

16. Actions collect() count() first() take( n ) reduce( function )

17. Parellelize Parellelize Parellelize Parellelize Transform Transform Transform Transform Transform Transform Transform Transform Action Action Action Action Result Result Result Result

18. Parellelize Parellelize Parellelize Parellelize Transform Transform Transform Transform Transform Transform Transform Transform Action Action Action Action Result Result Result Result Lineage

19. Parellelize Parellelize Parellelize Parellelize Transform Transform Transform Transform Transform Transform Transform Transform Action Action Action Action

20. Parellelize Parellelize Parellelize Parellelize Transform Transform Transform Transform Transform Transform Transform Transform Action Action Action Action

21. Parellelize Parellelize Parellelize Parellelize Transform Transform Transform Transform Transform Transform Transform Transform Action Action Action Action Result Result Result Result

22. Using the Connector

23. https://github.com/mongodb/mongo- spark

24. http://spark.apache.org/docs/latest/

25. { "_id" : ObjectId("578be1fe1fe699f2deb80807"), "user_id" : 196, "movie_id" : 242, "rating" : 3, "timestamp" : 881250949 }

26. ./b in/spark-shell \ --conf \ "spark.mongodb.input.uri=mongodb://127.0.0.1/movies.movie_ratings" \ --conf \ "spark.mongodb.output.uri=mongodb://127.0.0.1/movies.user_recommendations" \ --packages org.mongodb.spark:mongo-spark-connector_2.10:1.0.0

27. ./bin/spark-shell \ --conf \ "spark.mongodb.input.uri=mongodb://127.0.0.1/movies.movie_ratings" \ --conf \ "spark.mongodb.output.uri=mongodb://127.0.0.1/movies.user_recommendations" \ --packages org.mongodb.spark:mongo-spark-connector_2.10:1.0.0

28. ./bin/spark-shell \ --conf \ "spark.mongodb.input.uri=mongodb://127.0.0.1/movies.movie_ratings" \ --conf \ "spark.mongodb.output.uri=mongodb://127.0.0.1/movies.user_recomm endations" \ --packages org.mongodb.spark:mongo-spark-connector_2.10:1.0.0

29. ./bin/spark-shell \ --conf \ "spark.mongodb.input.uri=mongodb://127.0.0.1/movies.movie_ratings" \ --conf \ "spark.mongodb.output.uri=mongodb://127.0.0.1/movies.user_recommendations" \ --packages org.mongodb.spark:mongo-spark-connector_2.10:1.0.0

30. import com.mongodb.spark._ import com.mongodb.spark.rdd.MongoRDD import org.bson.Document val rdd = sc.loadFromMongoDB() for( doc <- rdd.take( 10 ) ) println( doc )

31. Read Config Write Config

32. Aggregation Filters $match | $project | $group

33. JSON JSON JSON JSON JSON JSON JSON JSON JSON JSON JSON

34. JSON JSON JSON JSON JSON JSON JSON JSON JSON JSON JSON

35. val aggRdd = rdd.withPipeline( Seq( Document.parse( "{ $match: { Country: \"USA\" } }" ) ) )

36. Spark SQL + Dataframes

37. RDD + Schema = Dataframe

38. JSON JSON JSON JSON JSON JSON JSON JSON JSON JSON JSON $sample

39. Data Locality mongos

40. Courses and Resources

41. https://university.mongodb.com/courses/M233/about

42. THANKS! @blimpyacht