Memento: Time Travel for the Web

Presented at the Microsoft Research Faculty Summit 2010

http://research.microsoft.com/en-us/events/fs2010/

1. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
The Memento Team
Herbert Van de Sompel
Michael L. Nelson
Robert Sanderson
Lyudmila Balakireva
Scott Ainsworth
Harihar Shankar
Memento: Time Travel for the Web
Memento is partially funded by the
Library of Congress

2. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
Memento wants to make navigating the Web’s Past Easy
2
http://www.mementoweb.org
http://groups.google.com/group/memento-dev

3. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
Recap of the Basics …
3

4. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
W3C Web Architecture: Resource – URI - Representation
Resource
Representation
Represents
URI
Identifies
dereference
4

5. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
dereferencecontent negotiation
W3C Web Architecture: Resource – URI - Representation
Resource
URI
Identifies
Representation 1
Represents
Representation 2Represents
5

6. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
Problem Statement …
6

7. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
Resources
7

8. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
Resources have Representations
8

9. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
Resources have Representations that Change over Time
9

10. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
Only the Current Representation is Available from a Resource
10

11. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
Old Representations are Lost Forever
11

12. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
Archived Resources Exist
12

13. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
Archived Resources
http://web.archive.org/web/20010911203610/http://ww
w.cnn.com/ archived resource for http://cnn.com
http://en.wikipedia.org/w/index.php?title=September_1
1_attacks&oldid=282333 archived resource for
http://en.wikipedia.org/wiki/September_11_attacks
Sep 11 2001, 20:36:10 UTC Dec 20 2001, 4:51:00 UTC
13

14. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
Finding Archived Resources
Go to http://www.archive.org/ and search
http://cnn.com
On http://web.archive.org/web/*/http://cnn.com, select
desired datetime
14

15. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
Finding Archived Resources
Go to
http://en.wikipedia.org/wiki/September_11_attacks
and click History
Browse History
15

16. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
Navigating Archived Resources
http://en.wikipedia.org/w/index.php?title=September_1
1_attacks&oldid=282333 archived resource for
http://en.wikipedia.org/wiki/September_11_attacks3
Dec 20 2001, 4:51:00 UTC
http://en.wikipedia.org/wiki/The_Pentagon
current
Pentagon
16

17. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
Navigating Archived Resources
http://web.archive.org/web/20010911203610/http://ww
w.cnn.com/ archived resource for http://cnn.com
http://web.archive.org/web/20010911213855/www.cnn
.com/TECH/space/
Sep 11 2001, 20:36:10 UTC Sep 11 2001, 21:38:55 UTC
SPACE
17

18. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
Current and Past Web are Not Integrated
18
• Current and Past Web
based on same technology.
• But, going from Current to
Past Web is a matter of
(manual) discovery.
• Memento wants to make
going from Current to Past
Web a (HTTP) protocol
matter.
• Memento wants to integrate
Current And Past Web.

19. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
The Memento Approach …
19

20. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
Navigate the Web of the Past
http://en.wikipedia.org/wiki/
Web_Archiving
20

21. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
Navigate the Web of the Past
http://en.wikipedia.org/wiki/
Web_Archiving
Oct 11 2009, 05:30:33 UTC
Set browser time dial to …
21

22. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
Navigate the Web of the Past
http://en.wikipedia.org/wiki/
Web_Archiving
Oct 01 2009, 16:30:00 UTC
From Wikipedia History
Oct 11 2009, 05:30:33 UTC
Set browser time dial to …
22

23. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
Navigate the Web of the Past
http://en.wikipedia.org/wiki/
Web_Archiving
Robots Exclusion ProtocolOct 11 2009, 05:30:33 UTC
Oct 01 2009, 16:30:00 UTC
From Wikipedia History
Oct 11 2009, 05:30:33 UTC
Set browser time dial to …
23

24. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
Navigate the Web of the Past
Oct 11 2009, 05:30:33 UTC
http://en.wikipedia.org/wiki/
Robots_exclusion_protocol
24
Oct 11 2009, 05:30:33 UTC
Browser time dial still at …

25. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
Navigate the Web of the Past
Oct 11 2009, 05:30:33 UTC
http://en.wikipedia.org/wiki/
Robots_exclusion_protocol
Sep 15 2009, 20:49:00 UTC
From Wikipedia History
25
Oct 11 2009, 05:30:33 UTC
Browser time dial still at …

26. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
Navigate the Web of the Past
Oct 11 2009, 05:30:33 UTC
http://en.wikipedia.org/wiki/
Robots_exclusion_protocol
Robots Exclusion Oct 11 2009, 05:30:33 UTC
Sep 15 2009, 20:49:00 UTC
From Wikipedia History
26
Oct 11 2009, 05:30:33 UTC
Browser time dial still at …

27. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
Navigate the Web of the Past
Oct 11 2009, 05:30:33 UTC
http://www.robotstxt.org/
27
Oct 11 2009, 05:30:33 UTC
Browser time dial still at …

28. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
Navigate the Web of the Past
Oct 11 2009, 05:30:33 UTC
http://www.robotstxt.org/
Nov 09 2007, 06:21:04 UTC
From Internet Archive
28
Oct 11 2009, 05:30:33 UTC
Browser time dial still at …

29. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
How does Memento achieve this?
There are two components to the Memento Solution:
•Component 1: Navigation towards an archived
resource via its original resource, by leveraging
content negotiation.
•Component 2: A discovery API for archives that
allows requesting a list of all archived versions it
holds for a resource with a given URI.
29

30. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
How does Memento achieve this?
•Component 1: Navigation towards an archived
resource via its original resource, by leveraging
content negotiation.
30

31. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
The Web without a Time Dimension
31

32. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
The Web without a Time Dimension
32

33. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
The Web without a Time Dimension
33
Need to use a different URI to access archived versions of a resource and its current version

34. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
The Web with Time Dimension added by Memento
34
In Memento: use URI of the current version to access archived versions, but qualify it with datetime

35. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
The Web with Time Dimension added by Memento
35
… and magically arrive at an archived version

36. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
How does Memento achieve this?
In order to fully understand how Memento introduces
a time dimension to the Web, we present a brief
recap of Transparent Content Negotiation (conneg)
in HTTP.
RFC 2295. Transparent Content Negotiation in HTTP,
http://www.ietf.org/rfc/rfc2295.txt
36

37. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
HTTP GET on URI A
37

38. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
GET with conneg on URI T – Server Choice – 302 Found – Step 1
38
transparently
negotiable
resource

39. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
GET with conneg on URI T – Server Choice – 302 Found – Step 2
39

40. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
GET with conneg on URI T – Server List – 406 Not Acceptable
40

41. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
How does Memento do This?
•Component 1: Navigation towards an archived
resource via its original resource, by leveraging
content negotiation.
41

42. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
Terminology Intermission
We introduce the term Memento to refer to an
archived version of a resource.
A Memento for a resource URI-R (as it existed)
at time t
i
is a resource URI-M
i
[URI-R@t
i
] for
which the representation at any moment
past its creation time t
c
is the same as the
representation that was available from URI-
R at time t
i
, with t
c
>= t
i
. Implicit in this
definition is the notion that, once created, a
Memento always keeps the same
representation.
42

43. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
DT-conneg: Content Negotiation in the datetime dimension
•RFC 2295 introduces conneg in the following dimensions: media type,
language, compression, character set, e.g.:
-HTTP Request:
oAccept-Language: en-US
oHTTP Response:
oContent-Language: en-US
•Inspired by RFC 2295, Memento introduces datetime conneg:
-HTTP Request:
oAccept-Datetime: Mon, 12 Oct 2009 14:20:33 GMT
oHTTP Response:
oContent-Datetime: Sun, 11 Oct 2009 11:18:05 GMT
43

44. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
DT-conneg: Content Negotiation in the datetime dimension
•This means that somewhere, we will need transparently negotiable
resources (cf. slides 38-40) that supports the datetime dimension to
get to appropriate Mementos.
•This will be discussed for 2 classes of servers:
oWeb servers without internal archival capabilities;
oWeb servers with internal archival capabilities.
44

45. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
Servers Without Internal Archival Capabilities
•This type includes:
oServers that are crawled by a web archive, such as the
Internet Archive
oServers with an associated transactional archive
•These servers are not aware of the details of Mementos of their
resources held by external archives.
•These servers do not have the essential information (URI-Ms,
and associated datetimes) to respond to a DT-conneg request.
•But they can be constructive by pointing (HTTP Link) a client to
an archive that can respond to the DT-conneg request.
oUnconditionally do this for resources for which Mementos are
conceivably available in the archive.
45

46. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
http://lanlsource.lanl.gov/
hello
current
http://mementoarchive.lanl.gov/store/ta/20091021
120001/http://lanlsource.lanl.gov/hello
Oct 04 2009, 12:00:01 UTC
Oct 21 2009, 12:00:01 UTC
Oct 10 2009, 12:00:03 UTC
46

47. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
original
resource
Mementos
original server archival server
47

48. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
original
resource
variant
resources
Mementos
transparently
negotiable
resource
DT-conneg with URI-G to get URI-M
original server archival server
48

49. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
original
resource
variant
resources
Mementos
DT-conneg with URI-G to get URI-M
transparently
negotiable
resource
original server archival server
49
HTTP
Link

50. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
Terminology Intermission
We introduce the term TimeGate to refer to a
transparently negotiable resource that supports the
datetime dimension.
A TimeGate for an original resource URI-R is a
transparently negotiable resource URI-
G[URI-R] for which all variant resources are
Mementos URI-M
i
[URI-R@t
i
] of the
resource URI-R. Since multiple archives
may host versions of URI-R, multiple
TimeGates may exist for any given
resource, i.e. one per archive.
50

51. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
original
resource
variant
resources
Mementos
DT-conneg with URI-G to get URI-M
transparently
negotiable
resource
TimeGate
original server archival server
51
HTTP
Link

52. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
Servers With Internal Archival Capabilities
•This type includes:
oContent Management Systems
oVersion Control Systems
oServers that archive resource representations in the cloud
and keep track of the URIs and datetimes of remotely
archived resources.
•These servers have all the essential information (URI-Ms, and
associated datetimes) to respond to a DT-conneg request.
•The previous architectural solution is maintained to enforce strict
distinction between handling requests for current and past
representations.
52

53. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
http://en.wikipedia.org/wiki/
September_11_attacks
current
Dec 20 2001, 4:51:00 UTC
Dec 31 2004, 20:46:00 UTC
Dec 20 2008, 22:21:00 UTC
http://en.wikipedia.org/w/index.php?
title=September_11_attacks&oldid=259237305
53

54. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
Mementos
original
resource
original server
54

55. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
original
resource
variant
resources
Mementos
DT-conneg with URI-G to get URI-M
transparently
negotiable
resource
TimeGate
original server
55
HTTP
Link

56. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
A Memento HTTP Navigation involving an Aggregator
56

57. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
A Memento HTTP Navigation involving an Aggregator
57
Scenario
• www.digitalpreservation.gov points at TimeGate provided by
an Aggregator
• URI-R, URI-G, URI-M on different servers

58. Memento HTTP Flow
HEAD R, Accept-Datetime
302M, Vary, TCN, LinkR,B,M
200, Content-Datetime, LinkR,B,M
GET G, Accept-Datetime
GET M, Accept-Datetime
LinkG

59. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
Memento HTTP Flow: URI-R
HEAD R, Accept-Datetime
HEAD / HTTP/1.1
Host: www.digitalpreservation.gov
Accept-Datetime: Sat, 10 Oct 2009 00:00:00 GMT
Connection: close
59

60. Memento HTTP Flow
HEAD R, Accept-Datetime
302M, Vary, TCN, LinkR,B,M
200, Content-Datetime, LinkR,B,M
GET G, Accept-Datetime
GET M, Accept-Datetime
LinkG

61. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
Memento HTTP Flow: Success – URI-R
HTTP/1.1 200 OK
Date: Thu, 21 Jan 2010 00:02:12 GMT
Server: Apache
Link:
<http://mementoproxy.lanl.gov/aggr/timegate/http ://www.digitalpreservation.gov/> ;
rel=“timegate”
Content-Length: 255
Connection: close
Content-Type: text/html; charset=iso-8859-1
61
LinkG

62. Memento HTTP Flow
HEAD R, Accept-Datetime
302M, Vary, TCN, LinkR,B,M
200, Content-Datetime, LinkR,B,M
GET G, Accept-Datetime
GET M, Accept-Datetime
LinkG

63. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
GET G, Accept-Datetime
Memento HTTP Flow: URI-G
GET /aggr/timegate/http://www.digitalpreservation.gov / HTTP/1.1
Host: mementoproxy.lanl.gov
Accept-Datetime: Sat, 10 Oct 2009 00:00:00 GMT
Connection: close
63

64. Memento HTTP Flow
HEAD R, Accept-Datetime
302M, Vary, TCN, LinkR,B,M
200, Content-Datetime, LinkR,B,M
GET G, Accept-Datetime
GET M, Accept-Datetime
LinkG

65. Memento HTTP Flow: Success – URI-G
302M, Vary, LinkR,B,M
HTTP/1.1 302 Found
Date: Thu, 21 Jan 2010 00:06:50 GMT
Server: Apache
TCN: choice
Vary: negotiate, accept-datetime
Location: http://wayback.archive-it.org/1610/20090928171405/http://
www.digitalpreservation.gov/
Link: <http://www.digitalpreservation.gov/>; rel="original",
<http://mementoproxy.lanl.gov/aggr/timebundle/http ://www.digitalpreservation.gov/>;
rel="timebundle”,
<http://wayback.archive
-it.org/256/20051108162921/http://www.digitalpreservation.gov/>;
rel=“first-memento”; datetime=“Tue, 08 Nov 2005 00:00:00 GMT ”,
<http://webcitation.org/query?id=1257028234035091>;
rel=“next-memento”; datetime=”Sat, 31 Oct 2009 18:30:35 GMT ”,
<http://webcitation.org/query?id=1213058061345794>;
rel=“prev-memento”; datetime="Mon, 09 Jun 2008 20:34:23 GMT ”,
<http://wayback.archive
-it.org/256/20100120102000/http://www.digitalpreservation.gov/>;
rel=“last-memento”; datetime=”Wed, 20 Jan 2010 10:20:00 GMT ”
Content-Length: 0
Connection: close

66. Memento HTTP Flow
HEAD R, Accept-Datetime
302M, Vary, TCN, LinkR,B,M
200, Content-Datetime, LinkR,B,M
GET G, Accept-Datetime
GET M, Accept-Datetime
LinkG

67. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
GET M, Accept-Datetime
Memento HTTP Flow: URI-M
GET /1610/20090928171405/http://www.digitalpreservation.gov/ HTTP/1.1
Host: wayback.archive-it.org
Accept-Datetime: Sat, 10 Oct 2009 00:00:00 GMT
Connection: close
67

68. Memento HTTP Flow
HEAD R, Accept-Datetime
302M, Vary, TCN, LinkR,B,M
200, Content-Datetime, LinkR,B,M
GET G, Accept-Datetime
GET M, Accept-Datetime
LinkG

69. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
Memento HTTP Flow: Success – URI-M
200, Content-Datetime, LinkR,B,M
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
X-Archive-Orig-Accept-Ranges: bytes
…
Content-Type: text/html;charset=utf-8
Content-Length: 23364
Date: Thu, 21 Jan 2010 00:09:40 GMT
Content-Datetime: Mon, 28 Sep 2009 17:14:05 GMT
Link: <http://www.digitalpreservation.gov />; rel="original",
<http://wayback.archive-
it.org/web/timebundle/http://www.digitalpreservation.gov/ >;
rel="timebundle”,
<http://wayback.archive
-it.org/256/20051108162921/http://www.digitalpreservation.gov/>;
rel=“first-memento”; datetime=“Tue, 08 Nov 2005 00:00:00 GMT ”,
<http://wayback.archive
-it.org/256/20100120102000/http://www.digitalpreservation.gov/>;
rel=“last-memento”; datetime=”Wed, 20 Jan 2010 10:20:00 GMT ”
Connection: close
69
Link header values are local
to wayback.archive-it.org
and different than those
provided by URI-G

70. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
The Web with Time Dimension added by Memento
70

71. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
Why Care About The Past?
From an anonymous reviewer (emphasis mine):
"Is there any statistics to show that many or a good number of Web
users would like to get obsolete data or resources? "

72. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
Replaying the Experience…
…can be more compelling than a summary

73. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010

74. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
vs.
(thanks to Michele Weigle for the following Memento selection)

75. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010

76. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010

77. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010

78. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010

79. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010

80. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010

81. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010

82. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010

83. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010

84. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010

85. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010

86. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010

87. Memento: Time Travel for the Web
Microsoft Research Faculty Summit, July 12-13, 2010
Memento wants to make navigating the Web’s Past Easy
87
http://www.mementoweb.org
http://groups.google.com/group/memento-dev