Big data Analytics

Types of Big data analytics

1. CHAPTER 02: Big Data Analytics

2. BDA
•Bigdataanalyticsistheprocessofexamininglargeand
varieddatasets--i.e.,bigdata--touncover
–hiddenpatterns,
–unknowncorrelations,
–markettrends,
–customerpreferences
–Meaningful/actionableinformation/insightstomake
fasterandbetterdecisions(informeddecisions).

3. BDA
•Dataanalyticshelpstosliceanddicethedatatoextract
insightsthatallowtoleveragethisdatatogivean
organizationacompetitiveadvantage.
•BDAsupports360degreeviewofthecustomer
(clickstreamdatawhichisunstructured).

7. BDA
•Businessescanuseadvancedanalyticstechniquessuch
as
–textanalytics,
–machinelearning,
–predictiveanalytics,
–datamining,statistics
–andnaturallanguageprocessingtogainnewinsightsfrom
previouslyuntappeddatasourceswithexistingenterprisedata.

8. BDA
•Itistechnologyenabledanalyticstoprocessand
analyzebigdata.
•BDAisaboutgainingameaningful,deeper,and
richerinsightintothebusinesstosteeritintheright
direction,understandingthecustomer’s
demographicstocross-sellandup-selltothem,by
betterleveragingtheservicesofvendorsand
suppliers,etc.

10. BDA
•BDAisatighthandshakebetweenthree
communities:IT,businessusers,anddatascientists.
•BDAisworkingwithdatasetswhosevolumeand
varietyexceedthecurrentstorage,processing
capabilitiesandinfrastructureofanenterprise.
•BDAisaboutmovingcodetodata,becauseprograms
fordistributedprocessingistiny(fewKBs)compared
tothedata(TB,PB,EB,ZBandYB).

11. Types of USD available for analysis
BDA
RFIDCRMERPPOSWebsites
Social
Media

12. What big data analytics is not?

13. Why the sudden hype around BDA?
1.Dataisgrowingata40%compoundannualrate,
reachingnearly74ZBby2021.Thevolumeof
businessdataworldwideisexpectedtodouble
every1.2years.
–Examples:
–Wal-Mart:Processonemillioncustomer
transactionsperhour.
–Twitter:500milliontweetsperday.
–2.7billion“Likes”andcommentsarepostedby
Facebookusersinaday.

14. Why the sudden hype around BDA?
2.Costpergigabyteofstoragehashugelydropped.
3.Anoverwhelmingnumberofuserfriendly
analyticstoolsareavailableinthemarkettoday.
4.ThreeDigitalAccelerators:bandwidth,digital
storage,andprocessingpower.

15. Classification of Analytics
1.FirstSchoolofThought.
2.SecondSchoolofThought

16. First school of thought
•Classified analytics into:
–Basic analytics
–Operationalized analytics
–Advanced analytics
–Monetized analytics.
•convert into or express in the form of currency.

18. First school of thought
•Basic analytics:
–Deals with slicing and dicing of data to help with basic
business insights.
–Reporting on historical data, basic visualization, etc..
•Operationalized analytics:
–Focusingontheintegrationofanalyticsintobusiness
unitsinordertotakespecificactiononinsights.
–Analyticsareintegratedintobothproduction
technologysystemsandbusinessprocesses.

19. First school of thought
•Advanced analytics:
–Forecastingforthefuturebypredictiveand
prescriptivemodeling.
•Monetized analytics:
–Usedtoderivedirectbusinessrevenue.
–Theactofgeneratingmeasurableeconomicbenefits
fromavailabledatasources

20. Six ways to indirectly monetize your
data
1.Reduce costs
2.Enhance your product or service
3.Enter new business sectors or tap into new
types of customers
4.Develop new products, services, or markets.
5.Drive sales and marketing
6.Improve productivity and efficiencies.

21. Second school of thought

22. Classification of analytics

24. Analytics 1.0, 2.0 and 3.0

25. Analytics 1.0 (1950-2009)
•Descriptive statistics.
•Report on events, occurrences, etc. of the past.
•Key questions:
–What happened?
–Why did it happen?
•Data:
–CRM, ERP, or 3
rd
party applications
–Small and structured data sources, DW and Data mart
–Internally sourced.
–Relational databases.

26. Analytics 2.0 (2005-2012)
•Descriptive + predictive statistics
•Uses data from the past to make predictions for the
future.
•Key questions:
–What will happen?
–Why will it happen?
•Data:
–Big data (SD, USD and SSD)
–Massive parallel servers running Hadoop
–Externally sourced.
–DB applications, Hadoop clusters, Hadoop environment.

27. Analytics 3.0 (2012 to present)
•Descriptive +predictive + Prescriptive statistics.
•Uses data from the past to make predictions for the future and at
the same time make recommendations to leverage the situations
to one’s advantage.
•Key questions:
–What will happen?
–When will it happen?
–Why will it happen?
–What action should be taken?
•Data:
–Big data (SD, USD and SSD) in real-time.
–Internally + Externally sourced.
–In memory analytics, M/C learning, agile analytical methods.

28. Challenges that prevent business from
capitalizing on big data
1.Gettingthebusinessunitstoshareinformationacross
organizationalsilos.
2.Findingtherightskills(businessanalystsanddatascientists)
thatcanmanagelargeamountsofSD,SSD,andUSDandcreate
insightfromit.
3.Theneedtoaddressthestorageandprocessingoflarge
volume,velocityandvarietyofbigdata.
4.DecidingwhethertouseSDorUSD,internalorexternaldatato
makebusinessdecisions.
5.Choosingtheoptimalwaytoreportfindingandanalysisofbig
dataforthepresentationtomakethemoresense.
6.Determiningwhattodowithinsightscreatedfrombigdata.

29. Top challenges facing big data
1.Thepracticalissuesofstoringallthedata(Scale)
2.Security:lackofauthenticationandauthorization
3.Dataschema(Nofixedandrigidschema)leadsto
dynamicschema.
4.Consistency/Eventualconsistency
5.Availability(24x7)(Failuretransparency)
6.Partitiontolerance(BothH/wandS/w)
7.Validatingbigdata(Dataquality):accuracy,
completenessandtimeliness.

30. Importance

31. Importance of BDA
•Costreduction:Cost-effectivestoragesystemfor
hugedatasets.[Hadoopandcloudbased
analytics]
•Faster,betterdecisionmaking:Provideswaysto
analyzeinformationquicklyandmakedecisions.
•[Hadoopprocessingspeedandin-memory
analytics,combinedwiththeabilitytoanalyzenew
sourcesofdata]

32. Importance of BDA
•Newproductsandservices:Evaluationof
customerneedsandsatisfactionthroughBDA.
–businesscancreatenewproductstomeet
customers’needs.
–WithBDA,morecompaniesarecreatingnew
products,includingnewrevenueopportunities,
moreeffectivemarketing,bettercustomer
service.
–Example:Automatedcar,healthcareappsand
entertainmentapps,Bankapps,Govt.apps,

33. Technologies needed to meet challenges
posed by big data
1.Cheapandabundantstorage.
2.Fasterprocessortohelpwithquickerprocessingof
bigdata.
3.Affordableopen-source,distributedbigdata
platformssuchasHadoop.
4.Parallelprocessing,clustering,virtualization,large
gridenvironments,highconnectivity,high
throughputsandlowlatency.
5.Cloudcomputingandotherflexibleresource
allocationarrangements.

36. Data Science
•Datascienceisaninterdisciplinaryfieldthatuses
scientificmethods,processes,algorithmsand
systemstoextractknowledgeandinsights
fromstructuredandunstructureddata.
•Itemploystechniquesandtheoriesdrawnfrom
manyfieldswithinthebroadareas
ofmathematics,statistics,informationtechnology
includingmachine learning,probability
models,classification,clusteranalysis,data
mining,databases,patternrecognition
andvisualization.

37. Data Science
•DataScienceisprimarilyusedtomake
predictionsand decisions using
predictive,prescriptiveanalyticsandmachine
learning.
•Prescriptiveanalyticsisallaboutproviding
advice.itnotonlypredictsbutsuggestsarange
ofprescribedactionsandassociatedoutcomes.

38. Data science –development of data
product
•A"dataproduct"isatechnicalassetthat:(1)
utilizesdataasinput,and(2)processesthatdata
toreturnalgorithmically-generatedresults.
•Theclassicexampleofadataproductisa
recommendationengine,whichingestsuserdata,
andmakespersonalizedrecommendationsbased
onthatdata

39. Examples of data products:
•Amazon'srecommendationenginessuggest
itemsforyoutobuy.Netflixrecommendsmovies
toyou.Spotifyrecommendsmusictoyou.
•Gmail'sspamfilterisdataproduct–analgorithm
behindthescenesprocessesincomingmailand
determinesifamessageisspamornot.
•Computervisionusedforself-drivingcarsisalso
dataproduct–machinelearningalgorithmsare
abletorecognizetrafficlights,othercarsonthe
road,pedestrians,etc.

40. The role of Data Science
•Theself-drivingcarscollectlivedatafromsensors,
includingradars,camerasandlaserstocreateamapofits
surroundings.Basedonthisdata,ittakesdecisionslike
whentospeedup,whentospeeddown,whento
overtake,wheretotakeaturn–makinguseofadvanced
machinelearningalgorithms.
•DataSciencecanbeusedinpredictiveanalytics:Datafrom
ships,aircrafts,radars,satellitescanbecollectedand
analyzedtobuildmodels.Thesemodelswillnotonly
forecasttheweatherbutalsohelpinpredictingthe
occurrenceofanynaturalcalamities.

41. Domains of Data Science
Infographic

42. Difference between Data Analysis and
Data Science
•DataAnalysisincludesdescriptiveanalyticsand
predictiontoacertainextent.
•Ontheotherhand,DataScienceismoreabout
PredictiveAnalyticsandMachineLearning.
•DataScienceisamoreforward-lookingapproach,an
exploratorywaywiththefocusonanalyzingthepastor
currentdataandpredictingthefutureoutcomeswith
theaimofmakinginformeddecisions.

43. Use-cases of for Data Science
•Internetsearch
•DigitalAdvertisements
•Recommender Systems
•Image Recognition
•Speech Recognition
•Gaming
•Price Comparison Websites
•Airline Route Planning
•FraudandRiskDetection
•Medicaldiagnosis,etc.

44. Data Science is multi-disciplinary

45. Data Science process
1.Collectingrawdatafrommultipledisparatedata
sources.
2.Processingthedata
3.Integratingthedataandpreparingcleandatasets
4.Engaginginexplorativedataanalysisusingmodel(ML
model)andalgorithms.
5.Preparingpresentationsusingdatavisualization.
6.Communicatingthefindingstoallstakeholders
7.Makingfasterandbetterdecisions.

46. Business Acumen (wisdom) skills of
Data Scientist
•Understanding of domain
•Business strategy
•Problem solving
•Communication
•Presentation
•Thirst for knowledge

47. Technology Expertise of Data Scientist
•GooddatabaseknowledgesuchasRDBMS
•GoodNoSQLdatabaseknowledgesuchasMongoDB,
Cassandra,Hbase,etc.
•LanguagessuchasJava,Python,R,C++,etc.
•Open-sourcetoolssuchasHadoop.
•Datawarehousing
•DataMining,Patternrecognition,algorithms
•Excellentunderstandingofmachinelearningtechniques
andalgorithms,suchasK-means,Regression,kNN,Naive
Bayes,SVM,PCA,Decisiontree,Tableau,Flare,Google
visualizationAPIs,textanalytics,DL,NLP,AIetc.

48. Mathematics Expertise of Data Scientist
•Probability
•Statistics
•LinearAlgebra
•Calculus

49. Data Scientist
•Adatascientistisaprofessionalresponsiblefor
collecting,analyzingandinterpretinglarge
amountsofdatatoidentifywaystohelpa
businesstoimproveoperationsandgaina
competitiveedgeovercompetitors.
•They're part mathematician,part
computerscientistandparttrend-spotter.

50. Responsibilities of Data Scientist
1.Prepareandintegrateslargeandvarieddatasets
anddeveloprelevantdatasetsforanalysis.
2.Thoroughlycleanandprunedatatodiscard
irrelevantinformation
3.Appliesbusiness/domainknowledgetoprovide
context.
4.Employsablendofanalyticaltechniquesto
developmodelsandalgorithmstounderstand
thedata,interpretrelationships,spottrends
andunveilpatterns.

51. Responsibilities of Data Scientist
5.Communicatesorpresentsfindingsorresultsin
thebusinesscontextinalanguagethatis
understoodbythedifferentbusiness
stakeholders.
6.Inventnewalgorithmstosolveproblemsand
buildnewtoolstoautomatework.(IIT)

52. Responsibilities of Data Scientist
7.Employsophisticatedanalyticsprograms,
machinelearningandstatisticalmethodsto
preparedataforuseinpredictiveand
prescriptivemodeling.
8.Exploreandexaminedatafromavarietyof
anglestodeterminehiddenweaknesses,trends
and/oropportunities

53. Terminologies/Technologies used in big
data environment
1.In-memoryanalytics
2.In-databaseprocessing
3.Symmetricmultiprocessorsystem(SMP)
4.Massivelyparallelprocessing
5.Parallelanddistributedsystems
6.Sharednothingarchitecture

54. In-memory analytics
•In-memoryanalyticsisanapproachtoqueryingdata
thatresidesinacomputer'srandomaccess
memory(RAM),asopposedtoqueryingdatathatis
storedonphysicaldisks.
•Thisresultsinreducedqueryresponsetimes,allowing
analyticapplicationstosupportfasterbusiness
decisions.
•In-memoryanalyticsisachievedthroughadoptionof64-
bitarchitectures,whichcanhandlemorememoryand
largerfilescomparedto32-bit.

55. In-database processing
•In-databaseanalyticsisatechnologythatallowsdata
processingtobeconductedwithinthedatabaseby
buildinganalyticlogicintothedatabaseitself.
•In-databaseprocessing,sometimesreferredtoasin-
databaseanalytics,referstotheintegrationof
dataanalyticsintodatawarehousingfunctionality.
•Iteliminatesthetimeandeffortrequiredtotransform
dataandmoveitbackandforthbetweenadatabaseand
aseparateanalyticsapplication.
•Example:creditcardfrauddetection,Bankrisk
management,etc.

56. Symmetric multiprocessor system (SMP)
•SMP(symmetricmultiprocessing)istheprocessingof
programsbymultipleprocessorsthatshareacommon
operatingsystemandmemory.
•Insymmetric(or"tightlycoupled")multiprocessing,the
processorssharememoryandtheI/Obusordatapath.
•Asinglecopyoftheoperatingsystemisinchargeofall
theprocessors(homogeneous).

58. Massively parallel processing (MPP)
•MPP(massivelyparallelprocessing)isthe
coordinatedprocessingofaproblem(program)by
multipleprocessorsthatworkondifferentpartsofthe
programinparallel,witheachprocessorhavingitsown
operatingsystemanddedicatedmemory.
•TheMPPprocessorscommunicateusingmessage
passing.
•Typically,thesetupforMPPismorecomplicated,
requiringthoughtabouthowtopartitionacommon
databaseamongprocessorsandhowtoassignwork
amongtheprocessors.AnMPPsystemisalsoknownas
a"looselycoupled"or"sharednothing"system.

59. Massively parallel processing (MPP)
•AnMPPsystemisconsideredbetterthana
symmetricallyparallelsystem(SMP)forapplications
thatallowanumberofdatabasestobesearchedin
parallel.Theseincludedecisionsupportsystem,data
warehouseandbigdataapplications.

60. Massively parallel processing

61. Parallel Systems

62. Distributed Systems

63. Shared nothing architecture

64. Shared Nothing architecture (SNA)
•Asharednothingarchitecture(SN)isadistributed
computingarchitectureinwhicheachnodeisindependent
andself-sufficient.Morespecifically,noneofthenodesshare
memoryordiskstorage.
•shared-nothingisoftencalledmassivelyparallelprocessor
(MPP).Manyresearchprototypesandcommercialproducts
haveadoptedtheshared-nothingarchitecturebecauseithas
thebestscalability.
•Intheshared-nothingarchitecture,eachnodeismadeof
processor,mainmemoryanddiskandcommunicateswith
othernodesthroughtheinterconnectionnetwork.Eachnode
isunderthecontrolofitsowncopyoftheoperatingsystem.

65. SNA advantages
1.Faultisolation
2.Scalability
3.Absenceofsinglepointoffailure
4.Self-healingcapabilities

66. CAP Theorem

67. CAP Theorem
•TheCAPtheorem,alsonamedBrewer'stheoremafter
computerscientistEricBrewer,statesthatitisimpossiblefor
adistributeddatastoretosimultaneouslyprovidemorethan
twooutofthefollowingthreeguarantees:
1.Consistency:Everyreadfetchesthelastwrite.
2.Availability:Everyrequestgetsaresponseon
success/failure.
3.Partitiontolerance:Systemcontinuestoworkdespite
messagelossorpartialfailureorN/Wpartition.
•Distributeddatastore:Itisisacomputernetworkwhere
informationisstoredonmorethanonenode,oftenin
areplicatedfashion.Itisalsoknownasdistributed
database.

68. CAP Theorem

69. Possible combinations of CAP for databases
1.AvailabilityandPartitionTolerance(AP)
2.ConsistencyandPartitionTolerance(CP)
3.ConsistencyandAvailability(CA)
Note:Google’sBigTable,Amazon’sDynamoand
Facebook’sCassandrausesoneofthesecombinations.

70. CAP Theorem
Examples

71. BASE
•BASEstandsforBasicallyAvailable,Softstate,Eventual
consistencyandusedtoachievehighavailabilityin
distributedcomputing.
•Basicallyavailableindicatesthatthesystemguarantees
theavailability,intermsoftheCAPtheorem.
•Softstateindicatesthatthestateofthesystemmay
changeovertime.
•Eventualconsistencyindicatesthatthesystemwill
becomeconsistentovertimeorafteracertaintimeall
nodesbecomeconsistent,

72. Few top analytics tools
1.MSExcel
2.SAS
3.IBMSPSSModeler
4.Statistica
5.QlickView
6.Tableau
7.Ranalytics
8.Weka
9.ApacheSpark
10.KNIME,Rapidminer
11.Splunkandsoon.