Open Cloud Consortium Overview (01-10-10 V6)

This is a brief overview of the Open Cloud Consortium.

1. Open Cloud Consortium: An Update Robert Grossman Open Cloud Consortium January 10, 2010 www.opencloudconsortium.org

2. Part 1. Overview of the Open Cloud Consortium (OCC) www.opencloudconsortium.org 2

3. 501(3)(c) Not-for-profit corporation Supports the development of standards, interoperability frameworks, and reference implementations. Manages testbeds: Open Cloud Testbed and Intercloud Testbed. Manages cloud computing infrastructure to support scientific research: Open Science Data Cloud. Develops benchmarks. 3 www.opencloudconsortium.org

4. OCC Members Companies: Aerospace, Booz Allen Hamilton, Cisco, InfoBlox , Open Data Group, Raytheon, Yahoo Universities: CalIT2, Johns Hopkins, MIT Lincoln Lab, Northwestern Univ., University of Illinois at Chicago, University of Chicago Government agencies: NASA Open Source Projects: Sector Project 4

5. OCC Working Groups Large Data Cloud Working Group Open Cloud Testbed Working Group. Intercloud Testbed Working Group Open Science Data Cloud Working Group

6. Storage Services Compute Services Applications Virtual Network Manager Data Services Network Transport Virtual Machine Manager IF-MAP (Metadata) Services Identity Manager IaaS PaaS Apps

7. Part 2. Intercloud Testbed 7 Cloud 1 Cloud 2

8. We have several cloud standards… Infrastructure as a Service Virtual Data Centers (VDC) Virtual Networks (VN) Virtual Machines (VM) Platform as a Service Cloud Compute Services Data/Table Cloud Services Cloud Storage Services Open Virtualization Format (OVF) Open Cloud Computing Interface (OCCI) SNIA Cloud Data Management Interface (CDMI) Large Data Cloud Interoperability Framework

9. Where are the Gaps? Infrastructure as a Service Virtual Data Centers (VDC) Virtual Networks (VN) Virtual Machines (VM) Physical Resources Platform as a Service Cloud Compute Services Data as a Service Open Virtualization Format (OVF) Open Cloud Computing Interface (OCCI) SNIA Cloud Data Management Interface (CDMI) Large Data Cloud Interoperability Framework Naming entities in IaaS & PaaS Bridging IaaS & DaaS Services that span multiple VMs , ….

10. Bridging the Gaps…A Small Step Infrastructure as a Service Virtual Data Centers (VDC) Virtual Networks (VN) Virtual Machines (VM) Physical Resources Platform as a Service Cloud Compute Services Data as a Service Open Virtualization Format (OVF) Open Cloud Computing Interface (OCCI) SNIA Cloud Data Management Interface (CDMI) Large Data Cloud Interoperability Framework Metadata service linking IaaS and DaaS Metadata service naming and linking entities in the IaaS layers

11. Part 3. Large Data Cloud Working Group 11 Standards for integrating and interoperating large data cloud services such as those provided by Hadoop and similar systems.

12. Focus of Working Group 12 Cloud Storage Services Cloud Compute Services (MapReduce, UDF, & other programming frameworks) Table-based Data Services Relational-like Data Services App App App App App App App App App Developing APIs for this framework.

13. Benchmarks for Large Data Clouds Until recently, the only benchmark used was Terasort (sorting 10 billion 100 byte records) Replaced by Gray Sort and Minute Sort Gray Sort tries to maximize TB / min sorted on 100 TB or more of data. Hadoop holds the current Gray Sort and Minute Sort records. Problem: sort is just one of the types of work load for analytic applications

14. MalStone MalGen – generates synthetic data with realistic distributions. MalStone A & B – “stylized” computations that can be used as benchmarks for architectures, software and systems for large data clouds. Open source and available at malgen.googlecode.com 14

15. Part 4. OCC Testbeds

16. Condominium Clouds In a condominium cloud, you buy your own rack or bunch of racks. The racks are managed and operated by the condominium association, in this case the OCC. If your rack is 120 TB, you get the rights to c. 40 TB of storage in the cloud. The rest is a shared resource. The Open Cloud Testbed is a condo cloud managed by the OCC. 16

17. Open Cloud Testbed Phase 2 9 racks 250+ Nodes 1000+ Cores 10+ Gb/s 17 MREN CENIC Dragon Hadoop Sector/Sphere Thrift KVM VMs Eucalyptus VMs C-Wave

18. Part 5. Open Science Data Cloud Working Group 18

19. Open Science Data Cloud 19 A stronomical data Biological data ( Bionimbus ) Networking data

20. For More Information [email protected] www.opencloudconsortium.org