Git mercurial - Git basics , features and commands

Git mercurial - Git basics , features and commands

1. Beyond code: Versioning
data with Git and Mercurial
Stephanie Collett and Martin Haye
California Digital Library, University of California

2. Not on Agenda

6. Agenda
•Background
•Case Study #1: eScholarship
Backup
•Case Study #2: Zephir Metadata
•Summary

7. Code
Version Control
Repository

8. Data/Metadata
Version Control
Repository

15. Why distributed?

17. Case #1
eScholarship Data/Metadata Backup

18. eScholarship

20. ~50k scholarly works

21. XML Metadata
10 files
per work}

22. XML Metadata
~500,000
files total}

23. Single Mercurial
Repository
XML Metadata

24. Working
Repository
Backup
Repository
Nightly Sync
(hg push)

26. Single Mercurial
Repository
XML Metadata

27. .hgignore
Single Mercurial
Repository
XML Metadata

28. Nightly Sync
(rsync)
Working
Storage
Backup
Storage
}{

29. 30-60 minutes
for the batch job

30. Date
Annotation
Change
Logs
Commit
History}

31. Case #2
Zephir Metadata Management System

34. Zephir

35. record/
File system

36. record/
File system
marc.xml

37. record/
File system
marc.xml
attrbutes.xml
summary.xml
transform.xsl

38. record/
marc.xml
attrbutes.xml
summary.xml
transform.xsl
.git/
File system

39. 10 million
/pairtree/ab/cd/e/record/.git
/pairtree/ab/cd/ea/record/.git
/pairtree/ab/cd/ez/record/.git
/pairtree/ab/cd/f2/record/.git
/pairtree/ab/cd/f9/record/.git
/pairtree/ab/cd/ff/record/.git
/pairtree/ab/cd/fm/record/.git
/pairtree/ab/cd/fq/record/.git
/pairtree/ab/cd/gi/record/.git
/pairtree/ab/cd/gw/record/.git
/pairtree/ab/cd/gz/record/.git
/pairtree/ab/cd/hs/record/.git
/pairtree/ab/cd/ht/record/.git
/pairtree/ab/cd/i/record/.git
...
...
}

40. Individually

41. Versioning
+ Audit Trail
+ Diffing
+ Debugging

42. Collectively

43. record/
marc.xml

44. 1 file, ~4k

45. marc.xml
attrbutes.xml
summary.xml
transform.xsl
record/

46. 4 file, ~36k

47. .git/
branches/
config
description
HEAD
hooks/
index
info/
objects/
refs/

48. record/ + record/.git
43 files, ~132k

49. record/ + record/.git
~132k x 10 million

50. record/ + record/.git
43 files x 10 million

52. Command Line
vs.
API

53. Grit Gem (Git)
vs.
Rugged Gem (Libgit2)

54. Grit Gem (Git)

57. Rugged Gem (Libgit2)

60. Grit vs. Rugged
•add files
•commit
•add files
•determine changes
•determine parent
•commit
•replace HEAD

62. Summary

63. Commit
Add
Remove
Log
Diff

64. vs.

67. texty data, small files
100-10,000 files per
repository

68. If it looks like code,
even if it's data,
it will probably work