Managing and Controlling Data Proliferation.pdf

Discover BoxLang in our introductory workshop, where participants explored its innovative platform and learned to harness its power for efficient web development. Whether new to BoxLang or deepening their skills, attendees gained practical insights and hands-on experience. The workshop showcased how BoxLang streamlines development workflows and unlocks new possibilities in web application creation.

1. INTO THE BOX 2024
THE NEW ERA OF
MODERN DEVELOPMENT

2. Eberly Hall Side A
PRESENTED BY
Curt Gratz
TAMING THE DATA SPRAWL

3. Curt Gratz
SPEAKER EN ITB 2024
‣Less of a Werido then Luis
‣Husband
‣Dad
‣Coach (CC and Track)
‣Co-Owner of CKH Consulting
‣Runner
‣BoxLang developer
‣Polyglot

4. ‣"Data is the new oil. Like oil, data can bring tremendous wealth, but if
uncontrolled, it can also be very messy." - Clive Humby, British data scientist

5. MadLibs
‣Give me a Company Name
‣Give me a thing
‣Give me a programming language
‣Give me a database
‣Give me a noun
‣Give me accounting software
‣Give me a cloud provider
‣Give me a noun

6. MadLibs
‣Give me another databases
‣Give me a noun
‣and another noun
‣Give me another databases
‣Give me a logging platform
‣Give me a programming language
‣Give me another databases
‣Give me a storage technology

7. MadLibs
‣Give me another company name
‣Give me a thing
‣Give me a search solution
‣Give me another database
‣Give me another accounting software
‣Give me another storage solution
‣Give me an programming language
‣Give me a cache
‣Give me an analytics platform

8. Taming the Data Sprawl

9. So, why does this suck
Taming the Data Sprawl

10. Step 1) Identify the problem
Taming the Data Sprawl

11. Step 2) Fix it
Taming the Data Sprawl

12. Taming the Data Sprawl
REPEAT

13. Taming the Data Sprawl

14. The End, thank you for coming 

to my TED talk
Taming the Data Sprawl

15. Taming the Data Sprawl
Fix it, Step 1
‣Eliminate
‣Remove duplicate
sources
‣Consolidate
‣Remove any unused
data

16. Taming the Data Sprawl
Fix it, Step 2
‣Govern It
‣Know
‣Validate
‣Own
‣Secure
‣Clean

17. Taming the Data Sprawl
Fix it, Step 3
‣De-Duplicate
‣What’s a match
‣Victim/Survivorship
‣Merging

18. Taming the Data Sprawl
Fix it, Step 4

19. Taming the Data Sprawl
Fix it, Step 5
REPEAT

20. Taming the Data Sprawl

21. Taming the Data Sprawl
Data Governance
“Right Information at the right time to the right people to make the right
decisions”

22. Taming the Data Sprawl

23. Taming the Data Sprawl

24. Taming the Data Sprawl
Challenges
Best Practices

25. Taming the Data Sprawl
Multiple versions of truth
Single version of truth

26. Taming the Data Sprawl
Time wasted on data gathering
Time for analysis

27. Taming the Data Sprawl
Project-driven approach to data
Common data definitions leveraged across
applications

28. Taming the Data Sprawl
Unclear data ownership rules
Well defined data ownership/stewardship rules

29. Taming the Data Sprawl
Inconsistent and incomplete information (poor
quality)
Well defined data with robust data quality
processes

30. Taming the Data Sprawl
“Work-arounds” from early implementations
Thought out data architecture and wholistic
approach (lower TCO)

31. Taming the Data Sprawl
Challenging to correct data history/errors
Detailed data repository and “single source”

32. Taming the Data Sprawl
Development focus on providing detail data
Development focus on providing quality data

33. Taming the Data Sprawl
THE DATA USER’S BILL OF
RIGHTS
Data users have the right to know what the data means
1. The right to know the definition of the data.
2. The right to know where the data came from.
3. The right to know how the data was calculated or manipulated.

34. Taming the Data Sprawl
THE DATA USER’S BILL OF
RIGHTS
Data users have the right to know how risks to the data have (or have not) been
managed
4. The right to know what Security risks weren't eliminated.
5. The right to know what Quality risks weren't eliminated.
6. The right to know what Privacy risks weren't eliminated.
7. The right to know what Compliance requirements influenced data
processing and usage.

35. Taming the Data Sprawl
THE DATA USER’S BILL OF
RIGHTS
Data users have the right to know who made decisions about managing the data,
according to what rules
8. The right to know who made data-related decisions.
9. The right to know what decision-making checks-and-balances were in place.
10. The right to know how issues have been and will be resolved.

36. Taming the Data Sprawl

37. Taming the Data Sprawl

38. Taming the Data Sprawl
Data Quality
”Quality is never an accident; it is
always the result of high intention,
sincere effort, intelligent direction and
skillful execution" - John Ruskin

39. Taming the Data Sprawl
Data Quality
‣Inaccurate Data
‣Incomplete Data
‣Inconsistant Data
‣Outdated Data
‣Invalid Data

40. Taming the Data Sprawl
Inaccurate Data
‣3rd Party data validation
‣Avoid Freeform
fi
fields
‣Regular Data Audits

41. Taming the Data Sprawl
Incomplete Data
‣Checks for completeness before submission
‣Data audits to capture missing data
‣Processes to stop actions without needed data

42. Taming the Data Sprawl
Inconsistant Data
‣Avoid freeform data
‣Require formats
‣Audit and adapt

43. Taming the Data Sprawl
Outdated Data
‣Checks against 3rd party systems
‣Data retention policies
‣Procedures to force rechecking stale data

44. Taming the Data Sprawl
Invalid Data
‣Data validation rules on entry
‣Avoid freeform data
‣Audit and clean
‣Process for identifying

45. Taming the Data Sprawl
Source of Truth
“We find the truth only when are searching for it.”

46. Taming the Data Sprawl
Data Management Association (DAMA)
www.dama.org
The Data Administrator Newsletter
www.tdan.com
The Data Governance Institute (DGI)
www.datagovernance.com

47. Taming the Data Sprawl
“Thank you.”
–hold up applause sign here
- First talk in given in 2024 with no AI reference

48. Taming the Data Sprawl
‣Email - [email protected]
‣Blog - https://ckhconsulting.com/blog/
‣Twitter - gratzc
‣LinkedIn - gratzc
‣League of Legends - gratzc
Contact Info

49. INTO THE BOX 2024
THANK YOU TO OUR
SPONSORS
INTO THE BOX 2024