HADOOP AND MAPREDUCE ARCHITECTURE-Unit-5.ppt

fgfgf

1. CSA15-CLOUDCOMPUTINGANDBIGDATAANALYTICS
Dr.J.Praveenchandar
AssociateProfessor/CSE
HADOOPANDMAPREDUCE
ARCHITECTURE
Dr.J.Praveenchandar/CSE

2. HADOOPANDMAPREDUCEARCHITECTURE
Dr.J.Praveenchandar/CSE
Bigdata–ApacheHadoop&HadoopEcoSystem–Analyzingdata
withHadoopstreaming–HDFSconcept–InterfacetoHDFS-
MovingDatainandoutofHadoop–IntroductiontoMapReduce–
MapReduceAlgorithmandArchitecture–Understandinginputs
andoutputsofMapReduce-AnatomyofMapReduceJobrun–
FailuresinclassicalMapReduceandYARN–Jobscheduling-Data
Serialization.

3. Big data
•BigDataisacollectionofdatathatishugeinvolume,yetgrowing
exponentiallywithtime.
•Itisadatawithsolargesizeandcomplexitythatnoneoftraditional
datamanagementtoolscanstoreitorprocessitefficiently.
•Bigdataisalsoadatabutwithhugesize.
ExampleofBigData
•NewYorkStockExchange SocialMedia
Dr.J.Praveenchandar/CSE

4. ApacheHadoop&HadoopEcosystem
•ApacheHadoopisanopensourceframework
intendedtomakeinteractionwithbigdataeasier,
•Hadoophasmadeitsplaceintheindustriesand
companiesthatneedtoworkonlargedatasets
whicharesensitiveandneedsefficienthandling.
•Hadoopisaframeworkthatenablesprocessingof
largedatasetswhichresideintheformofclusters.
•Beingaframework,Hadoopismadeupofseveral
modulesthataresupportedbyalargeecosystemof
technologies.
Dr.J.Praveenchandar/CSE

5. Amulti-node Hadoopcluster
Dr.J.Praveenchandar/CSE

6. ApacheHadoop&HadoopEcosystem
Dr.J.Praveenchandar/CSE
•HadoopEcosystemisaplatformorasuitewhichprovidesvarious
servicestosolvethebigdataproblems.
TherearefourmajorelementsofHadoopi.e.
•HDFS,
•MapReduce,
•YARN,and
•HadoopCommon.

7. Dr.J.Praveenchandar/CSE

8. ComponentsthatcollectivelyformaHadoopecosystem
Dr.J.Praveenchandar/CSE
•HDFS:HadoopDistributedFileSystem
•YARN:YetAnotherResourceNegotiator
•MapReduce:ProgrammingbasedDataProcessing
•Spark:In-Memorydataprocessing
•PIG,HIVE:Querybasedprocessingofdataservices
•HBase:NoSQLDatabase
•Mahout,SparkMLLib:MachineLearningalgorithmlibraries
•Solar,Lucene:SearchingandIndexing
•Zookeeper:Managingcluster
•Oozie:JobScheduling

9. HDFS:
Dr.J.Praveenchandar/CSE
•HDFSistheprimaryormajorcomponentofHadoopecosystemand
isresponsibleforstoringlargedatasetsofstructuredor
unstructureddataacrossvariousnodesandtherebymaintainingthe
metadataintheformoflogfiles.
•HDFSconsistsoftwocorecomponentsi.e.
Namenode
DataNode

10. YARN:
Dr.J.Praveenchandar/CSE
•YetAnotherResourceNegotiator,asthenameimplies,YARNisthe
onewhohelpstomanagetheresourcesacrosstheclusters.Inshort,
itperformsschedulingandresourceallocationfortheHadoop
System.
•Consistsofthreemajorcomponentsi.e.
ResourceManager
NodesManager
ApplicationManager

11. MapReduce:
Dr.J.Praveenchandar/CSE
•Bymakingtheuseofdistributedandparallelalgorithms,MapReduce
makesitpossibletocarryovertheprocessing’slogicandhelpsto
writeapplicationswhichtransformbigdatasetsintoamanageable
one.
•MapReducemakestheuseoftwofunctionsi.e.Map()andReduce()
whosetaskis:
Map()performssortingandfilteringofdataandtherebyorganizingthemin
theformofgroup.Mapgeneratesakey-valuepairbasedresultwhichislater
onprocessedbytheReduce()method.
Reduce(),asthenamesuggestsdoesthesummarizationbyaggregatingthe
mappeddata.Insimple,Reduce()takestheoutputgeneratedbyMap()as
inputandcombinesthosetuplesintosmallersetoftuples.

12. PIG:
Dr.J.Praveenchandar/CSE
•PigwasbasicallydevelopedbyYahoowhichworksonapigLatin
language,whichisQuerybasedlanguagesimilartoSQL.
•Itisaplatformforstructuringthedataflow,processingand
analyzinghugedatasets.
•Pigdoestheworkofexecutingcommandsandinthebackground,all
theactivitiesofMapReducearetakencareof.
•Aftertheprocessing,pigstorestheresultinHDFS.

13. HIVE:
Dr.J.Praveenchandar/CSE
•WiththehelpofSQLmethodologyandinterface,HIVEperforms
readingandwritingoflargedatasets.
•However, itsquerylanguageiscalledasHQL(HiveQueryLanguage).
•Itishighlyscalableasitallowsreal-timeprocessingandbatch
processingboth.
•Also,alltheSQLdatatypesaresupportedbyHivethus,makingthe
queryprocessingeasier.
•SimilartotheQueryProcessingframeworks,HIVEtoocomeswith
twocomponents:JDBCDriversandHIVECommandLine.

14. ApacheSpark:
Dr.J.Praveenchandar/CSE
•It’saplatformthathandlesalltheprocessconsumptivetaskslike
batchprocessing,interactiveoriterativereal-timeprocessing,graph
conversions,andvisualization,etc.
•Itconsumesinmemoryresourceshence,thusbeingfasterthanthe
priorintermsofoptimization.
•Sparkisbestsuitedforreal-timedatawhereasHadoopisbestsuited
forstructureddataorbatchprocessing,hencebothareusedinmost
ofthecompaniesinterchangeably.

15. Dr.J.Praveenchandar/CSE

16. HadoopStreaming
Dr.J.Praveenchandar/CSE
•ItisautilityorfeaturethatcomeswithaHadoopdistributionthat
allowsdevelopersorprogrammerstowritetheMap-Reduce
programusingdifferentprogramminglanguageslikeRuby,Perl,
Python,C++
•Ifwearereadinganimagedatathenwecangeneratekey-valuepair
foreachpixelwherethekeywillbethelocationofthepixelandthe
valuewillbeitscolorvaluefrom(0-255)foracoloredimage.
•Nowthislistofkey-valuepairsisfedtotheMapphaseandMapper
willworkoneachofthesekey-valuepairofeachpixelandgenerate
someintermediatekey-valuepairswhicharethenfedtothe
Reducerafterdoingshufflingandsortingthenthefinaloutput
producedbythereducerwillbewrittentotheHDFS.

17. HDFSconcept
Dr.J.Praveenchandar/CSE
•TheHadoopDistributedFileSystem(HDFS)istheprimarydata
storagesystemusedbyHadoopapplications.
•HDFSemploysaNameNodeandDataNodearchitectureto
implementadistributedfilesystemthatprovideshigh-performance
accesstodataacrosshighlyscalableHadoopclusters.
•HDFSenablestherapidtransferofdatabetweencomputenodes.
•Atitsoutset,itwascloselycoupledwithMapReduce,aframework
fordataprocessingthatfiltersanddividesupworkamongthenodes
inacluster,anditorganizesandcondensestheresultsintoa
cohesiveanswertoaquery.

18. Dr.J.Praveenchandar/CSE

19. HDFS
Dr.J.Praveenchandar/CSE
•HDFSusesaprimary/secondaryarchitecture.
•TheHDFScluster'sNameNodeistheprimaryserverthatmanages
thefilesystemnamespaceandcontrolsclientaccesstofiles.
•AsthecentralcomponentoftheHadoopDistributedFileSystem,the
NameNodemaintainsandmanagesthefilesystemnamespaceand
providesclientswiththerightaccesspermissions.
•Thesystem'sDataNodesmanagethestoragethat'sattachedtothe
nodestheyrun

20. HDFS
Dr.J.Praveenchandar/CSE
•HDFSexposesafilesystemnamespaceandenablesuserdatatobe
storedinfiles.
•Afileissplitintooneormoreoftheblocksthatarestoredinasetof
DataNodes.
•TheNameNodeperformsfilesystemnamespaceoperations,
includingopening,closingandrenamingfilesanddirectories.
•TheNameNodealsogovernsthemappingofblockstothe
DataNodes.
•TheDataNodesservereadandwriterequestsfromtheclientsofthe
filesystem.Inaddition,theyperformblockcreation,deletionand
replicationwhentheNameNodeinstructsthemtodoso.

21. HDFS
Dr.J.Praveenchandar/CSE
•HDFSsupportsatraditionalhierarchicalfileorganization.
•Anapplicationorusercancreatedirectoriesandthenstorefiles
insidethesedirectories.
•Thefilesystemnamespacehierarchyislikemostotherfilesystems--
ausercancreate,remove,renameormovefilesfromonedirectory
toanother.

22. FeaturesofHDFS
ThereareseveralfeaturesthatmakeHDFSparticularlyuseful,
including
Datareplication.Thisisusedtoensurethatthedataisalways
availableandpreventsdataloss.
Forexample,whenanodecrashesorthereisahardwarefailure,
replicateddatacanbepulledfromelsewherewithinacluster,so
processingcontinueswhiledataisrecovered.
Faulttoleranceandreliability.HDFS'abilitytoreplicatefileblocks
andstorethemacrossnodesinalargeclusterensuresfaulttolerance
andreliability.
Highavailability.Asmentionedearlier,becauseofreplicationacross
notes,dataisavailableeveniftheNameNodeoraDataNodefails.
Dr.J.Praveenchandar/CSE

23. FeaturesofHDFS
Dr.J.Praveenchandar/CSE
•Scalability.BecauseHDFSstoresdataonvariousnodesinthe
cluster,asrequirementsincrease,aclustercanscaletohundredsof
nodes.
•Highthroughput.BecauseHDFSstoresdatainadistributed
manner,thedatacanbeprocessedinparallelonaclusterofnodes.
This,plusdatalocality(seenextbullet),cuttheprocessingtimeand
enablehighthroughput.
•Datalocality.WithHDFS,computationhappensontheDataNodes
wherethedataresides,ratherthanhavingthedatamovetowhere
thecomputationalunitis.

24. BenefitsofusingHDFS
Dr.J.Praveenchandar/CSE
•Costeffectiveness.
•Largedatasetstorage.
•Fastrecoveryfromhardwarefailure.
•Portability.
•Streamingdataaccess.

25. HDFSusecasesandexamples
Dr.J.Praveenchandar/CSE
•TheHadoopDistributedFileSystememergedatYahooasapartof
thatcompany'sonlineadplacementandsearchengine
requirements.
•Likeotherweb-basedcompanies,Yahoojuggledavarietyof
applicationsthatwereaccessedbyanincreasingnumberofusers,
whowerecreatingmoreandmoredata.
•EBay,Facebook,LinkedInandTwitterareamongthecompanies
thatusedHDFStounderpinbigdataanalyticstoaddress
requirementssimilartoYahoo's.

26. MovingdataintoandoutofHadoop
Dr.J.Praveenchandar/CSE
•Datamovementisoneofthosethingsthatyouaren’tlikelytothink
toomuchaboutuntilyou’refullycommittedtousingHadoopona
project,atwhichpointitbecomesthisbigscaryunknownthathasto
betackled.
•Ingressandegressrefertodatamovementintoandoutofasystem,
respectively.
Keyelementsofdatamovement
•MovinglargequantitiesofdatainandoutofHadoopofferslogistical
challengesthatincludeconsistencyguaranteesandresourceimpacts
ondatasourcesanddestinations.

27. Dr.J.Praveenchandar/CSE

28. Keyelementsofdatamovement
Dr.J.Praveenchandar/CSE
•Idempotence:Anidempotentoperationproducesthesameresultno
matterhowmanytimesit’sexecuted.
•Aggregation:ItisperformedtoacquirethefinalresultoftheMapReduce
job,thatiscombiningtheoutputoftheMapperanddisplayingthe
result.
•Dataformattransformation
Thedataformattransformationprocessconvertsonedataformat
intoanother.
•Monitoring
ensuresthatfunctionsareperformingasexpectedinMonitoring
automated
systems.
•Dataformattransformation
Thedataformattransformationprocessconvertsonedataformatinto
another.

29. MapReduce
Dr.J.Praveenchandar/CSE
•MapReduceisaprogrammingmodelforwritingapplicationsthat
canprocessBigDatainparallelonmultiplenodes.
•MapReduceprovidesanalyticalcapabilitiesforanalyzinghuge
volumesofcomplexdata.
•TraditionalEnterpriseSystemsnormallyhaveacentralizedserverto
storeandprocessdata.
•Traditionalmodeliscertainlynotsuitabletoprocesshugevolumes
ofscalabledataandcannotbeaccommodatedbystandarddatabase
servers.
•Moreover,thecentralizedsystemcreatestoomuchofabottleneck
whileprocessingmultiplefilessimultaneously.

30. MapReduceWorks
Dr.J.Praveenchandar/CSE
•TheMapReducealgorithmcontainstwoimportanttasks,namely
MapandReduce.
•TheMaptasktakesasetofdataandconvertsitintoanothersetof
data,whereindividualelementsarebrokendownintotuples(key-
valuepairs).
•TheReducetasktakestheoutputfromtheMapasaninputand
combinesthosedatatuples(key-valuepairs)intoasmallersetof
tuples.
•Thereducetaskisalwaysperformedafterthemapjob.

31. MapReduce
Dr.J.Praveenchandar/CSE

32. MapReduce
Dr.J.Praveenchandar/CSE
•InputPhase−HerewehaveaRecordReaderthattranslateseach
recordinaninputfileandsendstheparseddatatothemapperin
theformofkey-valuepairs.
•Map−Mapisauser-definedfunction,whichtakesaseriesofkey-
valuepairsandprocesseseachoneofthemtogeneratezeroormore
key-valuepairs.
•IntermediateKeys−Theykey-valuepairsgeneratedbythemapper
areknownasintermediatekeys.
•Combiner−AcombinerisatypeoflocalReducerthatgroups
similardatafromthemapphaseintoidentifiablesets.
•ShuffleandSort−TheReducertaskstartswiththeShuffleandSort
step.

33. MapReduce
Dr.J.Praveenchandar/CSE
•Reducer−TheReducertakesthegroupedkey-valuepaireddataas
inputandrunsaReducerfunctiononeachoneofthem.
•Here,thedatacanbeaggregated,filtered,andcombinedinanumber
ofways,anditrequiresawiderangeofprocessing.
•Oncetheexecutionisover,itgiveszeroormorekey-valuepairsto
thefinalstep.
•OutputPhase−Intheoutputphase,wehaveanoutputformatter
thattranslatesthefinalkey-valuepairsfromtheReducerfunction
andwritesthemontoafileusingarecordwriter.

34. Map&fReduce
Dr.J.Praveenchandar/CSE

35. MapReduce-Example
•Letustakeareal-worldexampletocomprehendthepowerof
MapReduce.
•Twitterreceivesaround500milliontweetsperday,whichisnearly
3000tweetspersecond.
•ThefollowingillustrationshowshowTweetermanagesitstweets
withthehelpofMapReduce.
Dr.J.Praveenchandar/CSE

36. MapReduce
Dr.J.Praveenchandar/CSE
•MapReducealgorithmperformsthefollowingactions−
•Tokenize−Tokenizesthetweetsintomapsoftokensandwrites
themaskey-valuepairs.
•Filter−Filtersunwantedwordsfromthemapsoftokensandwrites
thefilteredmapsaskey-valuepairs.
•Count−Generatesatokencounterperword.
•AggregateCounters−Preparesanaggregateofsimilarcounter
valuesintosmallmanageableunits.

37. MapReduce-Algorithm
•TheMapReducealgorithmcontainstwoimportanttasks,namely
MapandReduce.
•ThemaptaskisdonebymeansofMapperClass
•ThereducetaskisdonebymeansofReducerClass.
•Mapperclasstakestheinput,tokenizesit,mapsandsortsit.The
outputofMapperclassisusedasinputbyReducerclass,whichin
turnsearchesmatchingpairsandreducesthem.
Dr.J.Praveenchandar/CSE

38. MapReduce-Algorithm
Dr.J.Praveenchandar/CSE
•MapReduceimplementsvariousmathematicalalgorithmstodividea
taskintosmallpartsandassignthemtomultiplesystems.In
technicalterms,MapReducealgorithmhelpsinsendingtheMap&
Reducetaskstoappropriateserversinacluster.
Thesemathematicalalgorithmsmayincludethefollowing−
•Sorting
•Searching
•Indexing
•TF-IDF

39. MapReduce-Algorithm
Dr.J.Praveenchandar/CSE
Sorting
•SortingisoneofthebasicMapReducealgorithmstoprocessand
analyzedata.
•MapReduceimplementssortingalgorithmtoautomaticallysortthe
outputkey-valuepairsfromthemapperbytheirkeys.
Searching
•SearchingplaysanimportantroleinMapReducealgorithm.Ithelps
inthecombinerphase(optional)andintheReducerphase.
•LetustrytounderstandhowSearchingworkswiththehelpofan
example.

40. MapReduce-Algorithm
Dr.J.Praveenchandar/CSE
Indexing
•Normallyindexingisusedtopointtoaparticulardataanditsaddress.
•ItperformsbatchindexingontheinputfilesforaparticularMapper.
TF-IDF
•TF-IDFisatextprocessingalgorithmwhichisshortforTermFrequency−
InverseDocumentFrequency.
•Itisoneofthecommonwebanalysisalgorithms.Here,theterm'frequency'
referstothenumberoftimesatermappearsinadocument.
TermFrequency(TF)
•Itmeasureshowfrequentlyaparticulartermoccursinadocument.
•Itiscalculatedbythenumberoftimesawordappearsinadocument
dividedbythetotalnumberofwordsinthatdocument.

41. MapReduceArchitecture
Dr.J.Praveenchandar/CSE
•MapReduceandHDFSarethetwomajorcomponents
ofHadoopwhichmakesitsopowerfulandefficienttouse.
•MapReduceisaprogrammingmodelusedforefficientprocessingin
paralleloverlargedata-setsinadistributedmanner.
•Thedataisfirstsplitandthencombinedtoproducethefinal
result.
•ThepurposeofMapReduceinHadoopistoMapeachofthejobsand
thenitwillreduceittoequivalenttasksforprovidingless
overheadovertheclusternetworkandtoreducetheprocessing
power.
•TheMapReducetaskismainlydividedintotwophasesMap
PhaseandReducePhase.

42. Dr.J.Praveenchandar/CSE

43. ComponentsofMapReduceArchitecture:
Dr.J.Praveenchandar/CSE
•Client:TheMapReduceclientistheonewhobringstheJobtothe
MapReduceforprocessing.
•Job:TheMapReduceJobistheactualworkthattheclientwantedto
dowhichiscomprisedofsomanysmallertasksthattheclientwants
toprocessorexecute.
•HadoopMapReduceMaster:Itdividestheparticularjobinto
subsequentjob-parts.
•Job-Parts:Thetaskorsub-jobsthatareobtainedafterdividingthe
mainjob
•InputData:ThedatasetthatisfedtotheMapReduceforprocessing.
•OutputData:Thefinalresultisobtainedaftertheprocessing.

44. Jobtrackerandthetasktracker
Dr.J.Praveenchandar/CSE
•JobTracker:TheworkofJobtrackeristomanagealltheresources
andallthejobsacrosstheclusterandalsotoscheduleeachmapon
theTaskTrackerrunningonthesamedatanodesincetherecanbe
hundredsofdatanodesavailableinthecluster.
•TaskTracker:TheTaskTrackercanbeconsideredastheactual
slavesthatareworkingontheinstructiongivenbytheJobTracker.
ThisTaskTrackerisdeployedoneachofthenodesavailableinthe
clusterthatexecutestheMapandReducetaskasinstructedbyJob
Tracker.

45. UNDERSTANDINGINPUTSANDOUTPUTSINMAPREDUCE
Dr.J.Praveenchandar/CSE

46. INPUTSANDOUTPUTSINMAPREDUCE
•Datainput:-
•ThetwoclassesthatsupportdatainputinMapReduceare
InputFormatandRecord-Reader.
•TheInputFormatclassisconsultedtodeterminehowtheinputdata
shouldbepartitionedforthemaptasks,andtheRecordReader
performsthereadingofdatafromtheinputs.
Dr.J.Praveenchandar/CSE

47. INPUTSANDOUTPUTSINMAPREDUCE
Dr.J.Praveenchandar/CSE
Dataoutput:-
•MapReduceusesasimilarprocessforsupportingoutputdataasit
doesforinputdata.
•Twoclassesmustexist,anOutputFormatandaRecordWriter.
•TheOutputFormatperformssomebasicvalidationofthedatasink
properties,andtheRecordWriterwriteseachreduceroutputtothe
datasink.

48. AnatomyofaMapReduceJobRun
Dr.J.Praveenchandar/CSE
•YoucanrunaMapReducejobwithasinglemethodcall:submit()on
aJobobject(youcanalsocallwaitForCompletion(),
•whichsubmitsthejobifithasn’tbeensubmittedalready,thenwaits
forittofinish
•Thismethodcallconcealsagreatdealofprocessingbehindthe
scenes.ThissectionuncoversthestepsHadooptakestorunajob.

49. AnatomyofaMapReduceJobRun
Dr.J.Praveenchandar/CSE
Herearefiveindependententities
•Theclient,whichsubmitstheMapReducejob.
•TheYARNresourcemanager,whichcoordinatestheallocationofcompute
resourcesonthecluster.
•TheYARNnodemanagers,whichlaunchandmonitorthecompute
containersonmachinesinthecluster.
•TheMapReduceapplicationmaster,whichcoordinatesthetasksrunning
theMapReducejob.TheapplicationmasterandtheMapReducetasksrun
incontainersthatarescheduledbytheresourcemanagerandmanaged
bythenodemanagers.
•Thedistributedfilesystem(normallyHDFS)whichisusedforsharingjob
filesbetweentheotherentities.

50. Dr.J.Praveenchandar/CSE

51. FailuresinClassicMapReduce
Dr.J.Praveenchandar/CSE
•IntheMapReduce1runtimetherearethreefailuremodesto
consider:failureoftherunningtask,failureofthetastracker,and
failureofthejobtracker.Let’slookateachinturn.
TaskFailure
•Considerfirstthecaseofthechildtaskfailing.
•Themostcommonwaythatthishappensiswhenusercodeinthe
maporreducetaskthrowsaruntimeexception.
•Ifthishappens,thechildJVMreportstheerrorbacktoitsparent
tasktracker,beforeitexits.Theerrorultimatelymakesitintothe
userlogs.
•Thetasktrackermarksthetaskattemptasfailed,freeingupaslotto
runanothertask.

52. FailuresinClassicMapReduce
Dr.J.Praveenchandar/CSE
TasktrackerFailure
•Failureofatasktrackerisanotherfailuremode.
•Ifatasktrackerfailsbycrashing,orrunningveryslowly,itwillstop
sendingheartbeatstothejobtracker(orsendthemvery
infrequently).
•Thejobtrackerwillnoticeatasktrackerthathasstoppedsending
heartbeats.
•Atasktrackercanalsobeblacklistedbythejobtracker,evenifthe
tasktrackerhasnotfailed.

53. FailuresinClassicMapReduce
Dr.J.Praveenchandar/CSE
JobtrackerFailure
•Failureofthejobtrackeristhemostseriousfailuremode.
•Hadoophasnomechanismfordealingwithfailureofthe
jobtracker—itisasinglepointoffailure—sointhiscasethejobfails.
•However,thisfailuremodehasalowchanceofoccurring,sincethe
chanceofaparticularmachinefailingislow.
•Afterrestartingajobtracker,anyjobsthatwererunningatthetimeit
wasstoppedwillneedtobere-submitted.