Fundamentals of data science: digital data
lokeshsd14
39 views
82 slides
Oct 01, 2024
Slide 1 of 82
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
About This Presentation
Digital data
Slide Content
Digital Data
Unit -3
Dr. T. NIKIL PRAKASH
ASSISTANT PROFESSOR
DEPARTMENT OF INFORMATION TECHNOLOGY,
ST. JOSEPH’S COLLEGE (AUTONOMOUS),
TIRUCHIRAPPALLI-02
Unit -3
Dr. T. NIKIL PRAKASH
ASSISTANT PROFESSOR
DEPARTMENT OF INFORMATION TECHNOLOGY,
ST. JOSEPH’S COLLEGE (AUTONOMOUS),
TIRUCHIRAPPALLI-02
Digital Data
•Data is present in homogeneous sources as well as in
heterogeneous sources.
•The need of the hour is to understand, manage,
process, and take the data for analysis to draw
valuable insights
•Types of Digital Data
•Digital data can be structured, semi-structured or unstructured
data.
•Data is present in homogeneous sources as well as in
heterogeneous sources.
•The need of the hour is to understand, manage,
process, and take the data for analysis to draw
valuable insights
•Types of Digital Data
•Digital data can be structured, semi-structured or unstructured
data.
•Structured data
•When data follows a pre-defined schema/structure we say
it is structured data.
•This is the data that is in an organized form (e.g., in rows
and columns) and be easily used by a computer program.
•Relationships exist between entities of data, such as classes
and their objects.
•About 10% data of an organization is in this format.
•Data stored in databases is an example of structured data
•When data follows a pre-defined schema/structure we say
it is structured data.
•This is the data that is in an organized form (e.g., in rows
and columns) and be easily used by a computer program.
•Relationships exist between entities of data, such as classes
and their objects.
•About 10% data of an organization is in this format.
•Data stored in databases is an example of structured data
Sources of Structured Data
•SQL Databases-Oracle DB,
•Spreadsheets such as Excel
•OLTP Systems
•Online forms
•Sensors such as GPS or RFID tags
•Network and Web server logs
•Medical devices
•SQL Databases-Oracle DB,
•Spreadsheets such as Excel
•OLTP Systems
•Online forms
•Sensors such as GPS or RFID tags
•Network and Web server logs
•Medical devices
Ease of working with structured data
•Structureddataiseasiertoworkwiththanunstructureddatabecauseit's
alreadyformattedandhasaclearstructure:
•Easytoanalyzeandmanipulate
•Structureddataiseasyforbothhumansandmachinestoworkwithbecauseit's
alreadyformatted.
•Easytosearchandquery
•Structureddata'sorganizednaturemakesiteasytomanipulateandquery.
•Easytouse
•Structureddatacanbeusedbyaveragebusinessuserswhounderstandthetopic
thedatarelatesto.
•Easytostore
•StructureddatacanbestoredintabularformatslikeExcelsheetsorSQL
databases,whichrequirelessstoragespace.
•Easytoscale
•Structureddatacanbestoredindatawarehouses,whichmakesithighlyscalable
•Structureddataiseasiertoworkwiththanunstructureddatabecauseit's
alreadyformattedandhasaclearstructure:
•Easytoanalyzeandmanipulate
•Structureddataiseasyforbothhumansandmachinestoworkwithbecauseit's
alreadyformatted.
•Easytosearchandquery
•Structureddata'sorganizednaturemakesiteasytomanipulateandquery.
•Easytouse
•Structureddatacanbeusedbyaveragebusinessuserswhounderstandthetopic
thedatarelatesto.
•Easytostore
•StructureddatacanbestoredintabularformatslikeExcelsheetsorSQL
databases,whichrequirelessstoragespace.
•Easytoscale
•Structureddatacanbestoredindatawarehouses,whichmakesithighlyscalable
•Semi-structureddata:
•Semi-structureddataisalsoreferredtoasself-describing
structure.
•Thisisthedatawhichdoesnotconformtoadatamodel
buthassomestructure.
•However,itisnotinaformwhichcanbeusedeasilybya
computerprogram.
•About10%dataofanorganizationisinthisformat;for
example,HTML,XML,JSON,emaildataetc.
•Semi-structureddataisalsoreferredtoasself-describing
structure.
•Thisisthedatawhichdoesnotconformtoadatamodel
buthassomestructure.
•However,itisnotinaformwhichcanbeusedeasilybya
computerprogram.
•About10%dataofanorganizationisinthisformat;for
example,HTML,XML,JSON,emaildataetc.
Source of semistructureddata
•Semi-structureddataisdatathatisnotcapturedorformatted
inconventionalways,butitdoeshavesomestructural
elements.Itcancomefrommanysources,including:
•Emails
•Markuplanguages
•Binaryexecutables
•TCP/IPpackets
•Zippedfiles
•Dataintegratedfromdifferentsources
•Webpages
•Logfiles
•NoSQLdatabases
•Electronicdatainterchange(EDI)
•Semi-structureddataisdatathatisnotcapturedorformatted
inconventionalways,butitdoeshavesomestructural
elements.Itcancomefrommanysources,including:
•Emails
•Markuplanguages
•Binaryexecutables
•TCP/IPpackets
•Zippedfiles
•Dataintegratedfromdifferentsources
•Webpages
•Logfiles
•NoSQLdatabases
•Electronicdatainterchange(EDI)
•Example
•<!DOCTYPEhtml>
<html>
<body>
<h1>My First Heading</h1>
<p>My first paragraph.</p>
</body>
</html>
•<!DOCTYPEhtml>
<html>
<body>
<h1>My First Heading</h1>
<p>My first paragraph.</p>
</body>
</html>
•Unstructured data:
•This is the data which does not conform to a data model or is not
in a form which can be used easily by a computer program.
•About 80% data of an organization is in this format; for example,
memos, chat rooms, PowerPoint presentations, images, videos,
letters. researches, white papers, body of an email, etc.
•This is the data which does not conform to a data model or is not
in a form which can be used easily by a computer program.
•About 80% data of an organization is in this format; for example,
memos, chat rooms, PowerPoint presentations, images, videos,
letters. researches, white papers, body of an email, etc.
Issues of Unstructured Data
•Storageandmanagement
•Unstructureddataisdifficulttostoreandmanagebecauseitcomesinmany
formats,suchastext,video,audio,andsocialmediacontent.Itcanalsobe
difficulttonavigatethroughthelargevolumeofunstructureddata.
•Processing
•Unstructureddatacanbetime-consumingandresource-intensiveto
process.Traditionaldatastorageoptionsmayalsobeinflexibleandunable
toadapttounstructureddata.
•Analysis
•Unstructureddataisnotorganizedinapredefinedmanner,makingit
difficulttoprocessandanalyzeusingtraditionalmethods.
•Cyber-attacks
•Unstructureddatacanmakesystemsmorevulnerabletocyber-attacks.
•Storageandmanagement
•Unstructureddataisdifficulttostoreandmanagebecauseitcomesinmany
formats,suchastext,video,audio,andsocialmediacontent.Itcanalsobe
difficulttonavigatethroughthelargevolumeofunstructureddata.
•Processing
•Unstructureddatacanbetime-consumingandresource-intensiveto
process.Traditionaldatastorageoptionsmayalsobeinflexibleandunable
toadapttounstructureddata.
•Analysis
•Unstructureddataisnotorganizedinapredefinedmanner,makingit
difficulttoprocessandanalyzeusingtraditionalmethods.
•Cyber-attacks
•Unstructureddatacanmakesystemsmorevulnerabletocyber-attacks.
Deals with unstructured data
Introduction to Big Data
•The "Internet of Things" and its widely ultra-
connected nature are leading to a burgeoning
rise in big data.
•There is no dearth of data for today's
enterprise.
•On the contrary, they are mired in data and
quite deep at that.
•Data is widely available.
•The "Internet of Things" and its widely ultra-
connected nature are leading to a burgeoning
rise in big data.
•There is no dearth of data for today's
enterprise.
•On the contrary, they are mired in data and
quite deep at that.
•Data is widely available.
SomeexamplesofBigData:
•TherearesomeexamplesofBigDataAnalyticsindifferent
areassuchasretail,ITinfrastructure,andsocialmedia.
•Retail:Asmentionedearlier,BigDatapresentsmany
opportunitiestoimprovesalesandmarketinganalytics.
•AnexampleofthisistheU.S.retailerTarget.
•Afteranalyzingconsumerpurchasingbehavior,Target's
statisticiansdeterminedthattheretailermadeagreatdeal
ofmoneyfromthreemainlife-eventsituations.
•Marriage,whenpeopletendtobuymanynewproducts
•Divorce,whenpeoplebuynewproductsandchangetheir
spendinghabits
•TherearesomeexamplesofBigDataAnalyticsindifferent
areassuchasretail,ITinfrastructure,andsocialmedia.
•Retail:Asmentionedearlier,BigDatapresentsmany
opportunitiestoimprovesalesandmarketinganalytics.
•AnexampleofthisistheU.S.retailerTarget.
•Afteranalyzingconsumerpurchasingbehavior,Target's
statisticiansdeterminedthattheretailermadeagreatdeal
ofmoneyfromthreemainlife-eventsituations.
•Marriage,whenpeopletendtobuymanynewproducts
•Divorce,whenpeoplebuynewproductsandchangetheir
spendinghabits
Characteristics of Data:
1.Composition:Thecompositionofdatadealswiththestructureof
data,thatis,thesourcesofdata,thegranularity,thetypes,andthe
natureofdataastowhetheritisstaticorreal-timestreaming.
2.Condition:Theconditionofdatadealswiththestateofdata,thatis,
"Canoneusethisdataasisforanalysis?"or"Doesitrequirecleansing
forfurtherenhancementandenrichment?"
3.Context:Thecontextofdatadealswith"Wherehasthisdatabeen
generated?""Whywasthisdatagenerated?"Howsensitiveisthis
data?""Whataretheeventsassociatedwiththisdata?"andsoon.
•Smalldata(dataasitexistedpriortothebigdatarevolution)isabout
certainty.
•Itisaboutknowndatasources;itisaboutnomajorchangestothe
compositionorcontextofdata.
1.Composition:Thecompositionofdatadealswiththestructureof
data,thatis,thesourcesofdata,thegranularity,thetypes,andthe
natureofdataastowhetheritisstaticorreal-timestreaming.
2.Condition:Theconditionofdatadealswiththestateofdata,thatis,
"Canoneusethisdataasisforanalysis?"or"Doesitrequirecleansing
forfurtherenhancementandenrichment?"
3.Context:Thecontextofdatadealswith"Wherehasthisdatabeen
generated?""Whywasthisdatagenerated?"Howsensitiveisthis
data?""Whataretheeventsassociatedwiththisdata?"andsoon.
•Smalldata(dataasitexistedpriortothebigdatarevolution)isabout
certainty.
•Itisaboutknowndatasources;itisaboutnomajorchangestothe
compositionorcontextofdata.
Definition of Big Data:
•Bigdataishigh-velocityandhigh-variety
informationassetsthatdemandcosteffective,
innovativeformsofinformationprocessingfor
enhancedinsightanddecisionmaking.
•Bigdatareferstodatasetswhosesizeis
typicallybeyondthestoragecapacityofand
alsocomplexfortraditionaldatabasesoftware
tools
•Bigdataisanythingbeyondthehuman&
technicalinfrastructureneededtosupport
storage,processingandanalysis
•Bigdataishigh-velocityandhigh-variety
informationassetsthatdemandcosteffective,
innovativeformsofinformationprocessingfor
enhancedinsightanddecisionmaking.
•Bigdatareferstodatasetswhosesizeis
typicallybeyondthestoragecapacityofand
alsocomplexfortraditionaldatabasesoftware
tools
•Bigdataisanythingbeyondthehuman&
technicalinfrastructureneededtosupport
storage,processingandanalysis
•Variety:Datacanbestructureddata,semi-structureddata
andunstructureddata.
•Datastoredinadatabaseisanexampleofstructureddata.
•HTMLdata,XMLdata,emaildata,CSVfilesarethe
examplesofsemi-structureddata.
•Powerpointpresentation,images,videos,researches,white
papers,bodyofemailetc.aretheexamplesofunstructured
data.
•Velocity:Velocityessentiallyreferstothespeedatwhich
dataisbeingcreatedinreal-time.
•Wehavemovedfromsimpledesktopapplicationslike
payrollapplicationtoreal-timeprocessingapplications.
•Volume:VolumecanbeinTerabytesorPetabytesor
Zettabytes.
andunstructureddata.
•Datastoredinadatabaseisanexampleofstructureddata.
•HTMLdata,XMLdata,emaildata,CSVfilesarethe
examplesofsemi-structureddata.
•Powerpointpresentation,images,videos,researches,white
papers,bodyofemailetc.aretheexamplesofunstructured
data.
•Velocity:Velocityessentiallyreferstothespeedatwhich
dataisbeingcreatedinreal-time.
•Wehavemovedfromsimpledesktopapplicationslike
payrollapplicationtoreal-timeprocessingapplications.
•Volume:VolumecanbeinTerabytesorPetabytesor
Zettabytes.
Introduction to big data analytics
•BigDataAnalyticsis...
•Technology-enabledanalytics:Manydataanalyticsand
visualizationtoolsareavailableinthemarkettodayfrom
leadingvendorssuchasIBM,Tableau,SAS,R
Analytics,Statistical,WorldProgrammingSystems
(WPS),etc.tohelpprocessandanalyzethebigdata.
•Aboutgainingameaningful,deeper,andricherinsight
intoyourbusinesstosteeritintherightdirection.
•Understandingthecustomer'sdemographicstocross-sell
andup-selltothem,betterleveragingtheservicesof
yourvendorsandsuppliers,etc.
•BigDataAnalyticsis...
•Technology-enabledanalytics:Manydataanalyticsand
visualizationtoolsareavailableinthemarkettodayfrom
leadingvendorssuchasIBM,Tableau,SAS,R
Analytics,Statistical,WorldProgrammingSystems
(WPS),etc.tohelpprocessandanalyzethebigdata.
•Aboutgainingameaningful,deeper,andricherinsight
intoyourbusinesstosteeritintherightdirection.
•Understandingthecustomer'sdemographicstocross-sell
andup-selltothem,betterleveragingtheservicesof
yourvendorsandsuppliers,etc.
•Aboutacompetitiveedgeoveryourcompetitors
byenablingyouwithfindingsthatallowquicker
andbetterdecision-making.
•Atighthandshakebetweenthreecommunities:
IT,businessusers,anddatascientists.
•Workingwithdatasetswhosevolumeandvariety
exceedthecurrentstorageandprocessing
capabilitiesandinfrastructureofyourenterprise
byenablingyouwithfindingsthatallowquicker
andbetterdecision-making.
•Atighthandshakebetweenthreecommunities:
IT,businessusers,anddatascientists.
•Workingwithdatasetswhosevolumeandvariety
exceedthecurrentstorageandprocessing
capabilitiesandinfrastructureofyourenterprise
Big Data Technologies
•Followingaretherequirementsoftechnologiesto
meetchallengesofbigdata:
•Thefirstrequirementischeapandamplestorage.
•Weneedfasterprocessorstohelpwithquickerprocessingofbig
data.
•Affordableopen-sourcedistributedbigdataplatforms,suchas
Hadoop.
•Parallelprocessing,clustering,virtualization,largegrid
environments(todistributeprocessingtoanumberofmachines),
highconnectivity,andhighthroughputs(rateatwhichsomething
isprocessed).
•Cloudcomputingandotherflexibleresourceallocation
arrangements.
•Followingaretherequirementsoftechnologiesto
meetchallengesofbigdata:
•Thefirstrequirementischeapandamplestorage.
•Weneedfasterprocessorstohelpwithquickerprocessingofbig
data.
•Affordableopen-sourcedistributedbigdataplatforms,suchas
Hadoop.
•Parallelprocessing,clustering,virtualization,largegrid
environments(todistributeprocessingtoanumberofmachines),
highconnectivity,andhighthroughputs(rateatwhichsomething
isprocessed).
•Cloudcomputingandotherflexibleresourceallocation
arrangements.
•BigDataTechnologiesInclude:
•ApacheKafka
•Abigdatatechnologythatenablesuserstoprocessdatainmotionand
quicklydeterminewhatworksandisnot.
•Tableau
•Apopulardataengineeringtoolthatgathersdatafrommultiplesources
usingadrag-and-dropinterfaceandallowsdataengineerstobuild
dashboardsforvisualization.
•Predictiveanalytics
•Akeycomponentofbigdatathatinvolvesstatisticalmodels,machine
learningalgorithms,andothertechniquestoanalyzelargeandcomplex
datasets.
•Splunk
•Abigdataplatformthatsimplifiescollectingandmanagingmassive
volumesofmachine-generateddata.
•ApacheKafka
•Abigdatatechnologythatenablesuserstoprocessdatainmotionand
quicklydeterminewhatworksandisnot.
•Tableau
•Apopulardataengineeringtoolthatgathersdatafrommultiplesources
usingadrag-and-dropinterfaceandallowsdataengineerstobuild
dashboardsforvisualization.
•Predictiveanalytics
•Akeycomponentofbigdatathatinvolvesstatisticalmodels,machine
learningalgorithms,andothertechniquestoanalyzelargeandcomplex
datasets.
•Splunk
•Abigdataplatformthatsimplifiescollectingandmanagingmassive
volumesofmachine-generateddata.
•Datavisualization
•Anintegralpartofanybigdataanalyticsprojectthat
allowsuserstocreatecharts,graphs,andothervisual
representationsoftheirdata.
•TensorFlow
•Apredictiveandgenericdeeplearninglibrarythatuses
bigdatatoofferitsextensivecapabilitiestocomputer
systems.
•KNIME
•Abigdatatoolthatgivesuserstheabilitytoreportand
integratedataacrossdifferentsources.
•MapReduce
•AprogrammingmodelthatiscommonlyusedinBig
DataAnalyticstoprocessandanalyzelargedatasets.
•Anintegralpartofanybigdataanalyticsprojectthat
allowsuserstocreatecharts,graphs,andothervisual
representationsoftheirdata.
•TensorFlow
•Apredictiveandgenericdeeplearninglibrarythatuses
bigdatatoofferitsextensivecapabilitiestocomputer
systems.
•KNIME
•Abigdatatoolthatgivesuserstheabilitytoreportand
integratedataacrossdifferentsources.
•MapReduce
•AprogrammingmodelthatiscommonlyusedinBig
DataAnalyticstoprocessandanalyzelargedatasets.
Tags
Categories
GeneralDownload
Download Slideshow Get the original presentation fileQuick Actions
Statistics
- Views 39
- Slides 82
- Age 427 days
Related Slideshows
22
Pray For The Peace Of Jerusalem and You Will Prosper
RodolfoMoralesMarcuc
32 views
26
Don_t_Waste_Your_Life_God.....powerpoint
chalobrido8
33 views
31
VILLASUR_FACTORS_TO_CONSIDER_IN_PLATING_SALAD_10-13.pdf
JaiJai148317
31 views
14
Fertility awareness methods for women in the society
Isaiah47
30 views
35
Chapter 5 Arithmetic Functions Computer Organisation and Architecture
RitikSharma297999
27 views
5
syakira bhasa inggris (1) (1).pptx.......
ourcommunity56
29 views