A Hands-on Introduction to MapReduce (in Python)

This presentation explains how to build a simple map reduce algorithm step by step.

1. A Hands-on Introduction to MapReduce in Python David Massart, PhD

2. Who Am I ?

3. Outline Set-up and requirements Counting words Limitations Map / Reduce Mapping Shuffling Reducing Hadoop

4. Environment Set-up Required Unix-like shell Linux Mac OS X Windows + Cygwin Python (e.g., anaconda) Good to have Java 8 Hadoop 2.6

5. Moby Dick by Herman Melville Download Moby Dick: wget https://www.gutenberg.org/cache/epub/2701/pg2701. txt Rename it input.txt : mv pg2701.txt input.txt

6. c at input.txt

7. Counting Words

8. ./ counter.py < input.txt

9. Limitations Processing time is, at best, proportional to the size of the text Actually, p erformance decreases with the size of the dictionary Very large texts can require more than one disk

10. MapReduce , Part 1: Mapping

11. ./ mapper.py < input.txt

12. MapReduce , Part 2: Shuffling Redistribute data based on the output keys produced by the " mapper” So that all data belonging to one are grouped together

13. ./ mapper.py < input.txt | sort

14. MapReduce , Part 3: Reducing

15. ./ mapper.py < input.txt | sort | ./ reducer.py

16. Hadoop