Digital Types

Describes types of digital data

1. CHAPTER 01: TYPES OF DIGITAL DATA

2. Data
•Anydatathatcanbeprocessedbydigital
computerandstoredinthesequencesof0'sand
1's(Binarylanguage)isknownsasdigitaldata.
•Wheneveryousendanemail,readasocialmedia
post,ortakepictureswithyourdigitalcamera,
youareworkingwithdigitaldata.
•Ingeneral,datacanbeanycharacter,text,
numbers,voicemessages,SMS,WhatsApp
messages,pictures,sound,orvideo.

3. Data
•Byteisthebasicunitofinformation
incomputerstorageandprocessing,andis
composedofeightbits;akilobyteis1,000bytes;
onemegabyteis1,000kilobytes.(GB,TB,PB,EB,
ZB,YB)
•Digitizingistheprocessofconvertinginformation
intodigitalformandisnecessaryforacomputerto
beabletoprocessandstoretheinformation.

4. Data
•It is an invaluable asset of any enterprise (big or small).
•Data is present internal to the enterprise and also exists
outside the firewalls of the enterprise.
•Data may be in homogeneous or heterogeneous.
•Need of the hour is to
–Understand, manage, process,
–and take the data for analysis
–to draw valuable insights.

5. Types of digital data
1.StructuredData:datastoredintheformof
rowsandcolumns(databases,Excel)
2.Un-structuredData:Nopre-definedschema
(PPTs,images,Videos,pdfs)
3.Semi-structuredData:Hybridschema(JSON,
HTML,XML,Email,andsoon),

7. Distribution of digital data (in %)
(by Gartner)
80
10
10
Unstructured
Semi-structured
Structured

8. Structured Data
•Data which is in an organized form (In rows & columns).
•Computer programs can use this data easily.
•Relationships exists between entities of data.
•Example
–Data stored in databases
–ERP
–CRM
–DW
–Data Cube

9. Structured Data
•Thedataconformstoapre-definedschemaorstructure
isknownasstructureddata.
•Thedatacanbeprocessed,stored,andretrievedina
fixedformat.Thisdatacanbeprocessedeasilyby
programs.
•Conformstoarelationaldatamodel.
•Structureddataisorganizedinsemanticchunks/entities
withsimilarentitiesgroupedtogethertoform
relations/tables.

10. structured Data
•Descriptionsforallentitiesinagroup
•Havethesamedefinedformat
•Haveapredefinedlength
•Followthesameorder.

11. Example

12. Sources of Structured Data
Structured
Data
OLTP
systems
Excel
Databases

13. Ease with structured data
Ease with
structured
data
Security
Insert/Update/
delete
Scalability
Transaction
processing
(ACID)
Indexing/
Searching

14. Database (RDBMS)
•Oracle Corp. –Oracle
•IBM –DB2, IBM-Informix
•Microsoft –SQL
•EMC –Greenplum
•Teradata –Teradata
•Open source-MySQL, PostgresSQL
•Sqlite
•Sequel Pro
•Amazon Aurora
•SAP SQL Anywhere, SAP IQ (Sybase)

15. Semi-structured Data
•Data which does not conform to a data model but has
some structure.
•Computer programs can not use this data easily.
•Example
–emails
–XML
–HTML
–JSON, and so on.

16. Semi-structured data (SSD)
•Itisreferredtoasselfdescribingstructure.
•Itisaformofstructureddatathatdoesnot
conformwiththeformalstructureofdatamodels
associatedwithrelationaldatabasesorother
formsofdatatables.
•Itusesmetadataandtagstoprovidesemantic
information.

17. Characteristics of semi-structured data
(SSD)
•Doesnotconformtoadatamodel
•Cannotbestoredintheformofrowsandcolumns
asinadatabase.
•Thetagsandelementsareusedtodescribedata.
•Attributesinagroupmaynotbethesame.
•Similarentitiesaregrouped.
•Sizeofthesameattributesinagroupmaydiffer
•Typeofsameattributesingroupmaydiffer.
•EvolvingSchema
•Schemaanddataaretightlycoupled.

18. Example (Names & Emails)
•Onewayis:
Name:RajuPatil
Email:[email protected],[email protected]
•Anotherwayis:
FirstName:Raju
LastName:Patil
Email:[email protected]

20. Sources of SSD
•Email
•XML
•TCP/IP
•Zipped files
•Mark-up languages
•Integration of data from heterogeneous sources.

21. Example: Email format
To: <Name>
From: <Name>
Subject: <Text>
CC: <Name>
Body: <Text, Graphics, Images, etc.><Name>

22. ABC Healthcare Blood Test Report
Date
<> ----
Department
<> -----
Patient Name
<>
Attending Doctor
<>
Hemoglobin
content
<>
Patient Age
<>
RBC count
<>
WBC count
<>
Platelet count
<>
Diagnosis <notes>
Conclusion <notes>

23. XML & JSON

24. Integration of data from heterogeneous
sources
User
Mediator : Uniform access to multiple data sources
OODBMSRDBMS
Legacy
system
Structured
file

26. Getting to know Unstructured data
•Overthepastfewdays,Dr.BenandDr.Stanley
hadbeenexchanginglongemailsabouta
particularcaseofgastro-intestinalproblem.
•EmailcontainsprocedurepracticedbyDr.Stanley,
aboutcombinationofdrugsthathassuccessfully
curedgastro-intestinaldisordersinpatients.
•Dr.Markhasapatientinthe“GoodLife”
emergencyunitwithquitesimilarcaseofgastro-
intestinaldisorder.

27. Unstructured Data
•Unstructureddatareferstothedatathatlacksany
specificformorstructure.
•Thismakesitverydifficultandtime-consumingto
processandanalyzeunstructureddata.
•Data which does not conform to any data model is USD.
•Computer programs can not use this data directly.
•About 80-90% data of an organization is in this format.
•Anenormousamountofknowledgeishiddeninthis
data.
•Hencefindingusefulknowledge/insightfromUSDisvery
crucial.

28. Unstructured Data
•Unstructureddataisagenericlabelfordescribingdata
thatisnotcontainedinadatabaseorsomeothertype
ofdatastructure.
•Unstructureddatacanbetextualornon-textual.
•Textualunstructureddataisgeneratedinmedialike
emailmessages,PowerPointpresentations,Word
documents,commentsinsocialmedia,etc.
•Non-textualunstructureddataisgeneratedinmedia
likeimages,CCTVfootage,audiofilesandvideofiles.
•Anythinginanon-databaseformisunstructureddata.

29. Unstructured Data
•Twotypes:
1.Bitmapobjects:image,video,oraudiofiles
2.Textualobjects:word,emails,pptsandsoon.

30. Unstructured Data
•Example
–Memos, QR code (Quick Response), Blogs
–Chat rooms, Tweets, Comments, likes, tags
–PPTs, emoji's, emoticons (emotion icons)
–Images, log files, social media posts
–Videos, sensor data (raw), weather data
–Doc files, geospatial data, surveillance data
–Body of email , GPS data, sensor data, etc.
–WhatsApp messages, CCTV footage and so on.

31. Getting to know Unstructured data

32. Characteristics of Unstructured data
•Thisdatacannotbestoredintheformofrows
andcolumnsasinadatabaseanddoesnot
conformtoanydatamodel.
•Itisdifficulttodeterminethemeaningofthe
data.
•Itdoesnotfollowanyruleorsemantics,i.e.Not
inanyparticularformatorsequence.
•Noteasilyusablebyaprogram.

33. Sources of Unstructured data
•Web pages
•Audio and Videos
•Images
•Body of an email
•Word document
•PPT and reports
•Chats and text messages
•Social media data
•White papers
•Surveys
•SMS
•Free form text
•Server Log files
•Product reviews

34. Web page is unstructured data
Web Page
Multimedia Image
Database
Text
XML

35. Challenges
•Storagespace:AlotofspaceisrequiredtostoreUSD.
•Scalability:Asthedatagrows,scalabilitybecomesan
issueandthecostofstoringUSDincreases.
•Retrieveinformation:Difficulttoretrieverequired
informationfromUSD
•Security:Ensuringsecurityisdifficultduetovaried
sourcesofdata.E.g.emails,webpages,etc.
•Indexing&searching:Verydifficultanderror-prone
asthestructureoftheUSDisnotclear.

36. Challenges
•Interpretation:USDisnoteasilyinterpretedby
conventionalsearchalgorithms.
•Classification:Differentnamingconventions
followedacrosstheorganizationmakeitdifficultto
classifydata.
•Derivingmeaning:Computerprogramscannot
automaticallyderivemeaningorstructurefromUSD.
•Fileformats:Increasingnumberoffileformats
makesitdifficulttointerpretdata.

37. Portion of Unstructured data
USD
SD

38. Dealing with USD
1.Datamining
2.Textmining/TextAnalytics
3.NLP
4.Noisytextanalytics
5.Manualtaggingwithmetadata
6.Partofspeechtagging
7.UIMA
8.WebScraping
Possible
Solutions

39. Data Mining
•Itisthecomputingprocessofdiscoveringpatterns
inlargedatasetsinvolvingmethodsatthe
intersectionofAI,machinelearning&
DL,statistics,anddatabasesystems.
•Popularalgorithms:
–Associationrulemining(MBA)
–RegressionAnalysis(Y=mX+c)
–Collaborativefiltering

40. Collaborative filtering
•collaborativefilteringusessimilaritiesbetweenusersand
itemssimultaneouslytoproviderecommendations.
•Itisamethodofmakingautomaticpredictions(filtering)
abouttheinterestsofauserbycollectingpreferences
ortasteinformationfrommanyusers(collaborating).
•Collaborativefilteringworksonafundamental
principle:youarelikelytolikewhatsomeonesimilarto
youlikes.

41. Collaborative filtering
•Collaborativefiltering(CF)isatechniquecommonlyused
•Collaborativefiltering(CF)isatechniqueused
byrecommendersystemstobuildpersonalized
recommendationsontheWeb.
•CompaniesthatemployCFmodelincludeAmazon,
Facebook,Twitter,LinkedIn,Spotify,GoogleNews,
Netflix,iTunes.

42. Collaborative filtering

43. Text analytics or text mining
•Itistheprocessofconverting
unstructuredtextdataintomeaningfuldatafor
analysis,tomeasurecustomeropinions,product
reviews,feedbackandsentimentalanalysisto
supportfactbaseddecisionmaking.
•Usesmanylinguistic,statistical,andmachine
learningtechniquessuchasclustering,pattern
recognition,tagging,associationanalysis,
predictiveanalytics,etc.

44. Text analytics or text mining
•Ithelpsorganizationstofindpotentiallyvaluable
businessinsightsincorporatedocuments,customer
emails,callcenterlogs,surveycomments,social
networkposts,medicalrecordsandothersourcesof
text-baseddata.
•Textminingcapabilitiesarealsobeingincorporated
intoAIchatbots/virtualagentsthatcompaniesdeploy
toprovideautomatedresponsestocustomersaspart
oftheirmarketing,salesandcustomerservice
operations.

45. Natural Language Processing (NLP)
•Naturallanguageprocessing(NLP)istheabilityofa
computerprogramtounderstandhumanlanguageas
itisspoken.NLPisacomponentofartificial
intelligence(AI).
•Itisafieldofcomputerscience,artificial
intelligenceandcomputationallinguisticsconcerned
withtheinteractionsbetweencomputersandhuman
(natural)languages(HCIdomain).
•NLPstrivestobuildmachinesthatunderstandand
respondtotextorvoicedata.

46. Natural Language Processing (NLP)

47. Noisy text analytics
•Itistheprocessofextractingstructuredorsemi-
structuredinformationfromnoisyunstructuredtextdata
suchasonlinechat,textmessages,emails,message
boards,blogs,wikis,etc.
•Thenoisyunstructureddatacomprisesoneormoreof
thefollowings:
–Spellingmistakes,
–Acronyms
–Non-standardwords(HBD,K,GN,GM,VGM,etc.)
–Missingpunctuations,
–Missinglettersandsoon.

48. Manual tagging with metadata
•Itistheprocessoftaggingmanuallywithadequate
metadatatoprovidethesemanticstounderstand
unstructureddata.
.
Road Accident

49. Part of Speech Tagging
•ItisalsocalledasPOSorPOSTorgrammatical
tagging.
•Itistheprocessofreadingtextandtaggingeach
wordinthesentenceasbelongingtoaparticular
partofspeechsuchas“noun”,“verb”,“adjective”,
“pronoun”,etc.
.

50. Unstructured Information
Management Architecture(UIMA)
•ItisanopensourceplatformfromIBM,which
integratesdifferentkindsofanalysisenginesto
provideacompletesolutionforknowledge
discoveryfromUSD.
•ItbridgethegapbetweenstructuredandUSD.

51. Uses of UIMA
•Usedtoconvertunstructureddatasuchas
repairlogs and servicenotes
intorelationaltables.
•Thesetablescanthenbeused
byautomatedtoolstodetectmaintenanceor
manufacturingproblems.

52. Uses of UIMA
•Usedinmedicalcontextstoanalyzeclinicalnotes,
suchastheClinicalTextAnalysisandKnowledge
ExtractionSystem(ApacheCTAKES).
•CTAKESisanopen-sourceNaturalLanguage
Processing(NLP)systemthatextractsclinical
informationfromelectronichealth/medical
recordfree-text(Usersarefreetotypewhatever
theywantinanyform).

53. UIMA block diagram
Users
Acquired from
various
sources
Subjected to
semantic
analysis
Structured
information
access
Query and
presentation
Structured
information
Analysis
Delivery
USD
Transformed into

54. Web Scraping

56. Big Data
•Bigdataisatermthatdescribeslarge,hard-
to-managevolumesofdata–bothstructured
andunstructured-noneoftraditionaldata
managementtoolscanstoreitorprocessit
efficiently.
•expertsnowpredictthat74zettabytesof
datawillbeinexistenceby2021.

57. Big Data
•Everyday,wecreate2.5quintillion(10
18
)
bytesofdata—90%ofthedataintheworld
todayhasbeencreatedinthelasttwoyears
alone.
•Thisdatacomesfromeverywhere:sensors
usedtogatherclimateinformation,poststo
socialmediasites,digitalpicturesandvideos,
purchasetransactionrecords,andcellphone
GPSsignals,WhatsApp,IOTandsoon.

58. Characteristics of Data
•Composition:Dealswithstructureofdata,i.e.,
sourcesofdata,thegranularity(Ex.Postal
address),thetypes,natureofdata(Staticorreal-
time).
•Condition:Dealswiththestateofdata,thatis,
“Canoneusedataasitisforanalysis?”or“Doesit
requirecleansingforfurtherenhancementand
enrichment?”.

59. Characteristics of Data
•Context:Dealswith
–Where,thisdatahasbeengenerated?
–Whythisdatagenerated?
–Howsensitiveisthisdata?
–Whataretheeventsassociatedwiththisdata?
–Andsoon.

60. Gartner
•Isaglobalresearchandadvisoryfirm
providinginsights,advice,andtoolsfor
leadersinIT,Finance,HR,CustomerService
andSupport

61. Big data definition-Gartner
•Bigdataishigh-volume,high-velocity,andhigh-
varietyinformationassetsthatdemandcost
effective,innovativeformsofinformation
processingforenhancedinsightanddecision
making.
•Costeffectiveandinnovativeformsof
informationprocessing:Talksaboutembracing
newtechniquesandtechnologiestocapture,
store,process,persevere,integrateandvisualize
thebigdata(3vs).

62. Definition of Big data by Gartner
•Enhancedinsightanddecisionmaking:Talks
aboutderivingdeeper,richer,andmeaningful
insightsandthenusingtheseinsightstomake
fasterandbetterdecisionstogainbusinessvalue
andthusacompetitiveedge.

63. Big data formula
DATA
Enhanced
Business
Value
Information
Actionable
Intelligence
Better
Decisions

64. Challenges with Big Data
•Capture
•Storage(Solution:CloudComputing)
•Curation(Managementofdata+Dataretention)
•Search
•Analysis
•Transfer
•Visualization
•Privacyviolations

65. 3 Vs

66. 3 V’s of Big data
•ThedatathatisbiginVolume,Velocityand
Varietyisknownasbigdata.

67. Sources of big data
•Archives:Archivesofscanneddocuments,
customercorrespondencerecords,patient’s
healthrecords,student’sadmissionrecords,
students’assessmentrecordsandsoon.
•Sensordata:Carsensors,smartelectricmeters,
officebuildings,washingm/c,otherelectronic
appliancesandsoon.
•Machinelogdata:Eventlogs,applicationlogs,
auditlogs,serverlogs,etc.

68. Sources of big data
•Publicweb:Wikipedia,Weather,regulatory,census,etc.
•Datastorage:Filesystems,SQLdatabase,NoSQL
database(MongoDB,Cassandra)andsoon.
•Media:Audio,Video,image,etc.
•Docs:CSV,worddocs,PDF,PPT,XLS,etc.
•BusinessApps:ERP,CRM,HR,GoogleDocs,etc.
•Socialmedia:Twitterblogs,Facebook,LinkedIn,
YouTube,Instagram,etc.
•IOT

69. Other characteristics of big data
•VeracityandValidity:Referstotheaccuracy
(quality)andcorrectnessofthedata.
•Volatility:Dealswithhowlongthedataisvalid?,
andhowlongshoulditbestored?.(OTP,Aadhar
No.,PW)
•Variability:Dataflowscanbehighlyinconsistent
withperiodicpeaks.(Intotal7V’sofbigdata)

70. Why Big data
More confidence in decision making
MoreData
More Accurate analysis
Greateroperationalefficiency,costreduction,time
reduction,newproductdevelopment,optimized
offerings,etc.

71. Three reasons for leveraging big data
1.CompetitiveAdvantage.
2.Decisionmaking
3.Tocreatenewbusinessvalueoutofdata.

72. Typical data warehouse Environment

73. Typical Hadoop Environment
•ItisdifferentfromDWenvironment.
•Heredatasourcesareweblogs,images,audios,
videos,socialmedia,docfiles,pdfs,etc.

74. Hadoop Environment

78. Big data & DW coexistence

79. Big data & DW coexistence