Common Crawl: An Open Repository of Web Data

Talk given by Lisa Green from the Common Crawl Foundation at the Hadoop User Group UK meetup on 10 October in London

1. What Does The Data World Mean to Society? Lisa Green 1 October 2012 London HUG Lisa Green 10 October 2012 Common Crawl : An Open Repository of Web Data

2. Photo license: Public Domain Origin: http:// en.wikipedia.org / wiki /File:Floppy_disk_2009_G1.jpg

3. Photo license: CC-BY-SA Origin: http:// en.wikipedia.org / wiki /File:Wikimedia_Foundation_Servers-8055_08.jpg

4. Image license: CC-BY Origin: http:// en.wikipedia.org / wiki /File:Internet_map_1024.jpg

5. Still Nascent Still Nascent Even cheaper storage Even cheaper compute Education Open Data Still Nascent Even cheaper storage Even cheaper compute Education Still Nascent Even cheaper storage Even cheaper compute Still Nascent Even cheaper storage Image license: CC-BY Credit: NASA, ESA, and the Hubble Heritage Team ( STScI /AURA)

6. Proprietary Commercial Gratis Libre

7. Gil Elbaz

8. Common Crawl Data ~8 Billion web pages ~120 TB 2008-2012 ARC files, JSON metadata, text files Available to anyone

9. ARC Files - Raw Content Metadata S tatus information HTTP response code File names & offsets of ARC files HTML title HTML meta tags RSS /Atom information All anchors/hyperlinks Text Files - Text Only http:// commoncrawl.org /get- started

10. http:// webdatacommons.org Change between 2010 and 2012 URLs with embedded data +6% Microdata +14% RDFa +26%

11. 22% of Web pages contain Facebook URLs 8 % of Web pages implement Open Graph tags

12. A corpus of anchortext - WikipediaConcept -Count from the CommonCrawl dataset, to benefit research on WSD, NLP and IR. Explicit Topic Modeling: Given a concept (represented as a wikipedia page), it can tell what are the most common terms people use to describe the concept. Given a sentence, it can help identify entities (person, location , organization) in the sentence and map them onto Wikipedia concepts. http:// wikientities.appspot.com

13. Mapping French websites related to Open Data

14. Other Use Examples Apache Giraph Testing Maplight Tineye Factual Sentiment Analysis Projects

15. In Development N-gram and Link G raph E xtracts Pig R eader More F requent F ull Crawls Focused S ubset C rawls at High Frequency Open Educational Resources

16. What Does The Data World Mean to Society? Lisa Green 1 October 2012 Lisa Green [email protected] www.commoncrawl.org @ commoncrawl @ boudicca Thank You London HUG