Cloud Computing @Yahoo!

1. Cloud Computing
@Yahoo!
Dekel Tankel
Director, Product Management
Yahoo! Cloud Computing
[email protected]
IGT, June 2009

2. What we’ll cover today…
•Why Cloud?
–Scale and Abstraction; Quality and Agility
–Yahoo!’s unique footprint
•Yahoo!’s Cloud Strategy
–Overview of the Yahoo! Cloud vision and portfolio
–Deep dive on Horizontal & Functional Cloud Services
•The Yahoo! Open Strategy
–Marrying Yahoo!’s “Open Strategy”, its platforms and ethic
with external Cloud services

3. Agility & Innovation
Quality & Stability
Why Cloud? Benefits for Yahoo!
Higher Agility & Stability while maintaining Scale
•Abstraction
–Enable developers to focus on
their applications, not infrastructure
•Accelerating innovation
–Adding new features and products
at an ever faster rate
•Increasing Scale & Availability
–More robustly, more globally,
more completely, for a given budget
Cloud is pushing up the
Operation Excellence Curve

4. Yahoo!’s Unique Cloud:
Unprecedented Scale
•Massive user base and engagement
–500M+ unique users per month
–Hundreds of petabyte of storage
–Hundreds of billions of objects
–Hundred of thousands of requests/sec

•Global
–Tens of globally distributed data centers
–Serving each region at low latencies
•Challenging Users
–Rapidly extracting value from voluminous data
–Downtime is not an option (outages cost $millions)
–Variable usage patterns

5. Yahoo! Cloud Services
ROI & Innovation
Users
Applications
Functional Cloud
Services
Horizontal Cloud
Services
Physical Layer
Y!OS, BOSS,
YQL, APT,
Analytics, …
Storage,
Batch, Edge
Serving,…

6. Yahoo! Cloud Services:
Focus on PaaS offerings
ROI & Innovation
Users
Applications
Functional Cloud
Services
Horizontal Cloud
Services
Physical Layer
IaaS
PaaS
SaaS

7. From Infrastructure to
Shareholders benefit
•Horizontal Cloud
–Focus on open source and collaborative R&D with
industry, academia and government
•Functional Cloud
–Focus on developing "open strategy" frameworks, tools
and services for developers (at Yahoo! and beyond)
•Combined Together
–Leverage our unique scale, assets and data to drive
disruptive innovations in the market and expand Yahoo!’s
competitive differentiation

8. Yahoo! Cloud Strategy in Action:
The Front Page Case Study
•Horizontal Cloud – Storage & Hadoop
–Analyze extremely large content data sets
•Functional Cloud – Content Optimization
–Rate content items based on various parameters
•Applications – Yahoo’s Front Page
–Display “high rating” items to the right users
–Benefit consumers and advertisers and grow
Yahoo!’s revenue

9. Yahoo! Cloud Strategy in Action:
The Inquisitor Case Study
•Horizontal Cloud – Hadoop
–Analyze large search-index data sets

•Functional Cloud - BOSS
–Expose the data in a structured, open, flexible
and “cloud like” way
•Applications - iPhone
TM
Inquisitor
–Leverage BOSS to provide innovative consumer experience
–Benefit consumers and grow Yahoo!’s revenue

10. Horizontal Cloud Services
ROI & Innovation
Users
Applications
Functional Cloud
Services
Horizontal Cloud
Services
Physical Layer

11. Horizontal Cloud Services
•Optimized for Yahoo!-scale
–Yahoo!-internal focus
–Data processing and serving environments
•Drive faster innovation and agility
–Shorter product development cycles
–Reduce labor and costs for infrastructure
•Multi-year effort
–Strategic investment across the company

12. Horizontal Cloud Services:
Conceptual View
Common Approaches to QA, Production Engineering,
Performance Engineering, Datacenter Management, and Optimization
ID & Account
Management
Shared Infrastructure
Provisioning & Virtualization (Xen)
Simple API’s
Operational
Storage
Structured,
unstructured
Batch
Storage &
Processing
Hadoop, PIG
Edge
Content
Services
Caching,
Proxies
Online
Serving
Web, Data
Security and
Authenticatio
n
Metering,
Billing
Monitoring &
QoS

13. Horizontal Cloud Services: Use Cases
Ads
Optimization
Content
Optimization
Search Index
Image/Video
Storage &
Delivery
Machine
Learning
(e.g. Spam filters)
Attachment
Storage

14. Yahoo! Distribution of Hadoop
•Hadoop in a nutshell
–Open source distributed file system & parallel execution
environment to process massive amounts of data
–Started in 2005, became top-level Apache project in 2008
–Simple Design for Horizontal Scaling on commodity HW
•Yahoo! Distribution of Hadoop
–Source distribution of Yahoo!’s implementation of Hadoop
(Based entirely on code found in the Apache Hadoop)
–Tested and deployed at Yahoo!’s massive scale
–Benefit the larger ecosystem , Increase pace of innovation
–http://developer.yahoo.com/hadoop

15. Yahoo! runs the largest
Hadoop Clusters in the World
•25,000+ nodes
–Clusters of up to 4,000 nodes
•4 Tiers of clusters
–Development & Testing, POCs, Science & Research,
Production
•Terasort Benchmarks
–62 seconds to sort One Terabyte (run on 1,500 nodes)
–16.25 hours to sort One Petabyte (run on 3,700 nodes)
•Webmap application
–~490 TB shuffling
–~280 TB output

16. Case Study - Search Assist™
•Database for Search Assist™ is built using Hadoop.
•3 years of log-data, 20-steps of map-reduce
•Leverage Hadoop’s scalability, load balancing and resiliency
•Simplified access, flexibility for rapid innovation (from C++ to Python)
Before Hadoop After Hadoop
Time 26 days 20 minutes
Development Time 2-3 weeks 2-3 days
16

17. Functional Cloud Services
ROI & Innovation
Users
Applications
Functional Cloud
Services
Horizontal Cloud
Services
Physical Layer

18. Functional Cloud Services
•Provides functional capabilities for applications
–Help developers to accomplish integrated web
experiences in a faster and easier way
–Provides common set of functional “building blocks”
• “Powered by” the horizontal cloud services
–Abstracts infrastructure services from the Application
•E.g. Storage, Compute, Serving, Robustness and Scalability
–Self-Served, Global, Managed, Elastic and Metered

19. Functional Cloud Services:
YQL & BOSS
•


A single endpoint service that enables
developers to query, filter and
combine data across
Yahoo! and beyond
http://developer.yahoo.com/yql/console/
Providing Yahoo! Search
infrastructure and technology to
developers and companies to help
them build their own search
experiences
Build your Own Search Service
http://developer.yahoo.com/search/boss/
Yahoo! Query Language

20. Build your Own Search Service (BOSS)
•Yahoo!'s open search web services platform
•Serving hundreds of millions of users across the Web.
•Goal: foster innovation in the search industry
–Build and launch web-scale search products that utilize the
entire Yahoo! Search index.
–Access to Yahoo!'s investments in crawling and indexing,
ranking and relevancy algorithms

21. Yahoo! Query Language (YQL)
•Single endpoint service to query, filter and combine data
across Yahoo! and beyond
–The “Internet API”
•SQL-like SELECT syntax for getting the right data
–Quickly discover available data sources and structure
–Combined data from a single web browser
•Easy-to-use Consol
–http://developer.yahoo.com/yql/console/

22. Y!OS and Cloud

23. 23
Yahoo! Open Stagey (Y!OS): Goals

24. 24
Y!OS and Cloud Strategy
CLOUD SERVICES

25. Open Collaborations
around the globe
•M45 - Yahoo!’s supercomputing cluster
–4,000 cores, 3 TB RAM, 1.5 PB disks, 27 teraflops!
–Operational since November 2007, 4 major Universities
–Focus on highly parallel computing
•Open Cirrus™ with HP & Intel
–A global, multi-data center, open source test bed
–Target to advance cloud computing research & education
–Simulates a real-life, Internet-scale environment
–9 Global sites, more than 50 research projects

26. Questions?
Dekel Tankel
Director, Product Management
Yahoo! Cloud Computing
[email protected]