CS8091_BDA_Unit_IV_Stream_Computing

Stream Computing Platforms

1. CS8091 / Big Data Analytics
III Year / VI Semester

2. UNIT IV -STREAM MEMORY
IntroductiontoStreamsConcepts–StreamDataModeland
Architecture-StreamComputing,SamplingDatainaStream–
FilteringStreams–CountingDistinctElementsinaStream–
Estimatingmoments–CountingonenessinaWindow–Decaying
Window–RealtimeAnalyticsPlatform(RTAP)applications-Case
Studies-RealTimeSentimentAnalysis,StockMarketPredictions.
UsingGraphAnalyticsforBigData:GraphAnalytics.

3. Stream Computing
Ahighperformancecomputersystemthatanalyzes
multipledatastreamsfrommanysources.
Streamcomputingisusedtomeanpullinginstreamsof
data,processingthedataandstreamingitbackoutasa
singleflow.
Itusessoftwarealgorithmsthatanalyzesthedatainreal
timeasitstreamsintoincreaseandaccuracywhendealing
withdatahandlingandanalysis.

4. Stream Computing

5. Stream Computing
Streamcomputingdeliversreal-timeanalytic
processingonconstantlychangingdatain
motion.
Itallowstocaptureandanalyzealldatainall
thetime,justintime.

6. Stream Computing
Streamanalyzesdatabeforeyoustoreit.
Analyzedatathatisinmotion(Velocity)
Processanytypeofdata(Variety)
Streamsisdesignedtoscaletoprocessanysizeof
datafromTerabytestoZetabytesperday.

7. Stream Computing
Storeless
Analyzemore
Makebetterdecisions,faster

8. Stream Computing
DataStreamprocessingplatforms:
Manyoftheseareopensourcesolutions.
Theseplatformsfacilitatetheconstructionofreal-time
applications,inparticularmessage-orientedorevent-
drivenapplicationswhichsupportingressofmessages
oreventsataveryhighrate,transfertosubsequent
processing,andgenerationofalerts.

9. Stream Computing
DataStreamprocessingplatforms:
Theseplatformsaremostlyfocusedonsupporting
event-drivendataflowthroughnodesinadistributed
systemorwithinacloudinfrastructureplatform.
TheHadoopecosystemcoversafamilyofprojects
thatfallundertheumbrellaofinfrastructurefor
distributedcomputingandlargedataprocessing.

10. Stream Computing
DataStreamprocessingplatforms:
Hadoopincludesanumberofcomponents,andbelow
isthelistofcomponents:
MapReduce,adistributeddataprocessingmodeland
executionenvironmentthatrunsonlargeclustersof
commoditymachines.
HadoopDistributedFileSystem(HDFS),adistributedfile
systemthatrunsonlargeclustersofcommoditymachines

11. Stream Computing
DataStreamprocessingplatforms:
Hadoopincludesanumberofcomponents,andbelowis
thelistofcomponents:
ZooKeeper,adistributed,highlyavailablecoordinationservice,
providingprimitivessuchasdistributedlocksthatcanbeusedfor
buildingdistributedapplications.
Pig,adataflowlanguageandexecutionenvironmentforexploring
verylargedatasets.PigsrunsonHDFSandMapReduceclusters.
Hive,adistributeddatawarehouse.

12. Stream Computing
DataStreamprocessingplatforms:
Itisdevelopedtosupportprocessinglargesetsof
structured,unstructured,andsemi-structureddata,
butitwasdesignedasabatchprocessingsystem.

13. Stream Computing
DataStreamprocessingplatforms–SPARK:
ApacheSparkismorerecentframeworkthatcombinesan
enginefordistributingprogramsacrossclustersof
machineswithamodelforwritingprogramsontopofit.
Itisaimedataddressingtheneedsofthedatascientist
community,inparticularinsupportofRead-Evaluate-Print
Loop(REPL)approachforplayingwithdatainteractively.

14. Stream Computing
DataStreamprocessingplatforms–SPARK:
SparkmaintainsMapReduce’slinearscalabilityand
faulttolerance,butextendsitinthreeimportantways:
First,ratherthanrelyingonarigidmap-then-reduceformat,
itsenginecanexecuteamoregeneraldirectedacyclicgraph
(DAG)ofoperators.Thismeansthatinsituationswhere
MapReducemustwriteoutintermediateresultstothe
distributedfilesystem,Sparkcanpassthemdirectlytothe
nextstepinthepipeline.

15. Stream Computing
DataStreamprocessingplatforms–SPARK:
SparkmaintainsMapReduce’slinearscalabilityandfault
tolerance,butextendsitinthreeimportantways:
Second,itcomplementsthiscapabilitywitharichsetof
transformationsthatenableuserstoexpresscomputationmore
naturally.
Third,Sparksupportsin-memoryprocessingacrossaclusterof
machines,thusnotrelyingontheuseofstorageforrecording
intermediatedata,asinMapReduce.

16. Stream Computing
DataStreamprocessingplatforms–SPARK:
Sparksupportsintegrationwiththevarietyoftoolsinthe
Hadoopecosystem.
Itcanreadandwritedatainallofthedataformatssupportedby
MapReduce.
ItcanreadfromandwritetoNoSQLdatabaseslikeHBaseand
Cassandra.
Itiswellsuitedforreal-timeprocessingandanalysis,supporting
scalable,highthroughput,andfault-tolerantprocessingoflivedata
streams.

17. Stream Computing
DataStreamprocessingplatforms–SPARK:
SparkStreaminggeneratesadiscretizedstream
(DStream)asacontinuousstreamofdata.
Regardinginputstream,SparkStreamingreceiveslive
inputdatastreamsthroughareceiveranddividesdata
intomicrobatches,whicharethenprocessedbythe
Sparkenginetogeneratethefinalstreamofresultsin
batches.

18. Stream Computing
DataStreamprocessingplatforms–SPARK:
SparkStreamingutilizesasmall-interval(inseconds)
deterministicbatchtoseparatestreamintoprocessable
units.
Thesizeoftheintervaldictatesthroughputand
latency,sothelargertheinterval,thehigherthe
throughputandthelatency.

19. Stream Computing
DataStreamprocessingplatforms–SPARK:
SinceSparkcoreframeworkexploitsmain
memory(asopposedtoStorm,whichisusing
Zookeeper)itsminibatchprocessingcanappearas
fastas“oneatatimeprocessing”adoptedin
Storm,despiteofthefactthattheRDDunitsare
largerthanStormtuples.

20. Stream Computing
DataStreamprocessingplatforms–SPARK:
Thebenefitfromtheminibatchistoenhancethe
throughputininternalenginebyreducingdata
shippingoverhead,suchasloweroverheadforthe
ISO/OSItransportlayerheader,whichwillallowthe
threadstoconcentrateoncomputation.
SparkwaswritteninScala,butitcomeswithlibraries
andwrappersthatallowtheuseofRorPython.

21. Stream Computing
DataStreamprocessingplatforms–Storm:
Stormisadistributedreal-timecomputationsystem
forprocessinglargevolumesofhigh-velocitydata.
Itmakesiteasytoreliablyprocessunboundedstreams
ofdataandhasarelativelysimpleprocessingmodel
owingtotheuseofpowerfulabstractions.

22. Stream Computing
DataStreamprocessingplatforms–Storm:
Aspoutisasourceofstreamsinacomputation.
Typically,aspoutreadsfromaqueuingbroker,suchas
RabbitMQ,orKafka,butaspoutcanalsogenerateitsown
streamorreadfromsomewhereliketheTwitterstreaming
API.
Spoutimplementationsalreadyexistformostqueuing
systems.

23. Stream Computing
DataStreamprocessingplatforms–Storm:
Aboltprocessesanynumberofinputstreamsand
producesanynumberofnewoutputstreams.
Theyareevent-drivencomponents,andcannotbeusedto
readdata.Thisiswhatspoutsaredesignedfor.
Mostofthelogicofacomputationgoesintobolts,suchas
functions,filters,streamingjoins,streamingaggregations,
talkingtodatabases,andsoon.

24. Stream Computing
DataStreamprocessingplatforms–Storm:
AtopologyisaDAGofspoutsandbolts,witheach
edgeintheDAGrepresentingaboltsubscribingtothe
outputstreamofsomeotherspoutorbolt.
Atopologyisanarbitrarilycomplexmultistagestream
computation;topologiesrunindefinitelywhendeployed.

25. Stream Computing
DataStreamprocessingplatforms–Storm:
Tridentprovidesasetofhigh-levelabstractionsinStorm
thatweredevelopedtofacilitateprogrammingofreal-time
applicationsontopofStorminfrastructure.
Itsupportsjoins,aggregations,grouping,functions,and
filters.Inadditiontothese,Tridentaddsprimitivesfor
doingstatefulincrementalprocessingontopofany
databaseorpersistencestore

26. Stream Computing
DataStreamprocessingplatforms–KAFKA:
Kafkaisanopensourcemessagebrokerproject
developedbytheApacheSoftwareFoundationand
writteninScala.
Theprojectaimstoprovideaunified,high-
throughput,low-latencyplatformforhandlingreal-
timedatafeeds.

27. Stream Computing
DataStreamprocessingplatforms–KAFKA:
AsingleKafkabrokercanhandlehundredsof
megabytesofreadsandwritespersecondfrom
thousandsofclients.
Inordertosupporthighavailabilityandhorizontal
scalability,datastreamsarepartitionedandspread
overaclusterofmachines.

28. Stream Computing
DataStreamprocessingplatforms–KAFKA:
KafkadependsonZookeeperfromtheHadoop
ecosystemforcoordinationofprocessingnodes.
ThemainusesofKafkaareinsituationswhen
applicationsneedaveryhighthroughputformessage
processing,whilemeetinglowlatency,high
availability,andhighscalabilityrequirements.

29. Stream Computing
DataStreamprocessingplatforms–Flume:
Flumeisadistributed,reliable,andavailableservice
forefficientlycollecting,aggregating,andmoving
largeamountsoflogdata.
Itisrobustandfaulttolerantwithtunablereliability
mechanismsandmanyfailoverandrecovery
mechanisms.Itusesasimpleextensibledatamodel
thatallowsforonlineanalyticapplication.

30. Stream Computing
DataStreamprocessingplatforms–Flume:
WhileFlumeandKafkabothcanactastheevent
backboneforreal-timeeventprocessing,theyhave
differentcharacteristics.
Flumeisbettersuitedincaseswhenoneneedsto
supportdataingestionandsimpleeventprocessing.

31. Stream Computing
DataStreamprocessingplatforms–AmazonKinesis:
AmazonKinesisisacloud-basedserviceforreal-timedata
processingoverlarge,distributeddatastreams.
AmazonKinesiscancontinuouslycaptureandstore
terabytesofdataperhourfromhundredsofthousandsof
sourcessuchaswebsiteclickstreams,financial
transactions,socialmediafeeds,ITlogs,andlocation-
trackingevents.

32. Stream Computing
DataStreamprocessingplatforms–AmazonKinesis:
KinesisallowsintegrationwithStorm,asitprovidesa
KinesisStormSpoutthatfetchesdatafromaKinesis
streamandemitsitastuples.
TheinclusionofthisKinesiscomponentintoaStorm
topologyprovidesareliableandscalablestreamcapture,
storage,andreplayservice.