Hadoop in the Cloud – The What, Why and How from the Experts

Hadoop in the Cloud – The What, Why and How from the Experts

1. Hadoop in the cloud The What, Why and How from the experts SATO Naoki (@ satonaoki ) Azure Technologist Microsoft Japan

2. Hadoop in the Cloud 2 Agenda Why Benefits of running Hadoop in the cloud What Options to run Hadoop in the Cloud Hadoop Clusters in the cloud Cluster Customizations How Architecture of a Cloud deployment

3. Hadoop in the Cloud 3 Agenda Why Benefits of running Hadoop in the cloud What Options to run Hadoop in the Cloud Hadoop Clusters in the cloud Cluster Customizations How Architecture of a Cloud deployment

4. Traditional Hadoop Clusters 4

5. Up-front HW costs Capacity planning Hadoop expertise Challenges with implementing Hadoop

6. Hadoop Clusters in the Cloud 6

7. Benefits of Cloud Unlimited elastic scale Auto geo redundancy No hardware costs Pay only for what you need No HW costs $0 Unlimited scale Pay what you need Deployed in minutes Why Hadoop in the cloud?

8. Distributed Storage Files split across storage Files replicated Nearest node responds Abstracted Administration Hadoop Clusters Extensible APIs to extend functionality Add new capabilities Allow for inclusion in custom environments Automated Failover Unmonitored failover to replicated data Built for resiliency Metadata stored for later retrieval Hyper-Scale Add resources as desired Built to include commodity configs Direct correlation of performance and resources Distributed Compute Distributed processing Resource Utilization Cost-Efficient method calls 8

9. Distributed Storage Files split across storage Files replicated Nearest node responds Abstracted Administration Cloud Extensible APIs to extend functionality Add new capabilities Allow for inclusion in custom environments Automated Failover Unmonitored failover to replicated data Built for resiliency Metadata stored for later retrieval Hyper-Scale Add resources as desired Built to include commodity configs Direct correlation of performance and resources Distributed Compute Distributed processing Resource Utilization Cost-Efficient method calls 9

10. Distributed Storage Files split across storage Files replicated Nearest node responds Abstracted Administration Hadoop in the Cloud Extensible APIs to extend functionality Add new capabilities Allow for inclusion in custom environments Automated Failover Unmonitored failover to replicated data Built for resiliency Metadata stored for later retrieval Hyper-Scale Add resources as desired Built to include commodity configs Direct correlation of performance and resources Distributed Compute Distributed processing Resource Utilization Cost-Efficient method calls 10

11. Hadoop in the Cloud 11 Agenda Why Benefits of running Hadoop in the cloud What Options to run Hadoop in the Cloud Hadoop Clusters in the cloud Cluster Customizations How Architecture of a Cloud deployment

12. Hadoop in the Cloud - Options Cloud Hadoop in IaaS Hadoop in PaaS Big Data as a Service Pros Complete Control On-Demand Cluster Sizing Storage - Local or Cloud Cons Only VMs managed for HA Administration required Clusters need to stay active Pros Fully managed – SLA bound Flexible resizing Customization Options Deployed in minutes Cons Forgo some control Pros Abstracted from clusters Automated resource alignment Easy to use interface and APIs Familiar languages Cons Forgo complete control Limited choice to tools

13. On-premises Hadoop Software Scenarios for deploying Hadoop as hybrid Cloud Cloud Specialized Workloads HDInsight Cloud Bursting HDInsight Cloud Backup/archive HDInsight

14. Traditional Hadoop Clusters – On Prem 14 Hadoop Cluster Worker Node HDFS HDFS HDFS Tasks Tasks Tasks Tasks Tasks Tasks Task Tracker Master Node Client Job (jar) file Job (jar) file

15. Hadoop Clusters in the Cloud

16. 16 Azure HDInsight Hadoop and Spark as a Service on Azure Fully managed Hadoop and Spark for the cloud 100% Open Source Hortonworks Data Platform Clusters up and running in minutes Managed, monitored and supported by Microsoft with the industry’s best enterprise SLA Use familiar BI tools for analysis , or open source notebooks for interactive data science 63% lower total cost of ownership than deploy your own Hadoop on-premises* *IDC study “The Business Value and TCO Advantage of Apache Hadoop in the Cloud with Microsoft Azure HDInsight”

17. HDInsight Cluster Architecture Azure Virtual Network HTTPS traffic ODBC/JDBC WebHCatalog Oozie Ambari Secure gateway AuthN HTTP Proxy Highly available Head nodes Worker nodes Azure Data Lake Store

18. Decoupling Compute from Storage Latency? Consistency? Bandwidth? Network

19. Decoupling Compute from Storage Network HDD-like latency 50 Tb+ aggregate bandwidth [1] Strong consistency [1] Azure Flat Network Architecture

20. Decoupling - Benefits Cloud NoSQL Workload Pros Smaller clusters can achieve the same level of performance as large clusters No need to add nodes just for storage capacity Depending on workloads, you see any where from 6x – 20x cost benefits Query + ML Workload Pros Clusters required only while processing data Data Persists for tools to connect and use Data can be replicated on other geo Delete clusters when not processing of data Streaming Workload Pros No need for large clusters to hold historical streams data Directly align throughput to cluster size as per SLA Cluster up only when streams are active

21. 21 Azure Data Lake Store A hyper scale repository for big data analytics workloads Hadoop File System (HDFS) for the cloud No limits to scale Store any data in its native format Enterprise grade access control and encryption Optimized for analytic workload performance

22. Customize cluster? HDInsight cluster provisioning states RDP to cluster, update config files (non-durable) Ad hoc Cluster customization options Hive/ Oozie Metastore Storage accounts & VNET’s ScriptAction Via Azure portal Ready for deployment Accepted Cluster storage provisioned AzureVM configuration Running Timed Out Error Cluster operational Configuring HDInsight Cluster customization (custom script running) Config values JAR file placement in cluster Via scripting / SDK No Yes

23. Cluster integration options Each cluster surfaces a REST endpoint for integration, secured via basic authN over SSL /thrift – ODBC & JDBC /Templeton – Job Submission, Metadata management / ambari – Cluster health, monitoring / oozie – Job orchestration, scheduling

24. Hadoop in the Cloud 24 Agenda Why Benefits of running Hadoop in the cloud What Options to run Hadoop in the Cloud Hadoop Clusters in the cloud Cluster Customizations How Architecture of a Cloud deployment

25. Cloud Deployments for Big Data 25 Data Factory Data Lake Store HDInsight Machine Learning SQL Database Power BI DocumentDB Event Hubs Stream Analytics

26. Introducing Cortana Intelligence Suite Action People Automated Systems Apps Web Mobile Bots Intelligence Dashboards & Visualizations Cortana Bot Framework Cognitive Services Power BI Information Management Event Hubs Data Catalog Data Factory Machine Learning and Analytics HDInsight (Hadoop and Spark) Stream Analytics Intelligence Data Lake Analytics Machine Learning Big Data Stores SQL Data Warehouse Data Lake Store Data Sources Apps Sensors and devices Data

27. Where Big Data is a cornerstone Action People Automated Systems Apps Web Mobile Bots Intelligence Dashboards & Visualizations Cortana Bot Framework Cognitive Services Power BI Information Management Event Hubs Data Catalog Data Factory Machine Learning and Analytics HDInsight (Hadoop and Spark) Stream Analytics Intelligence Data Lake Analytics Machine Learning Big Data Stores SQL Data Warehouse Data Lake Store Data Sources Apps Sensors and devices Data

28. Excel BI Power BI Mahout HiveQL HIVE Sqoop Pig Azure Data Lake Analytics HBase on Azure HDInsight Big Data Sources (Raw Unstructured) Log files Storm for Azure HDInsight Azure Stream Analytics Spark Streaming for Azure HDInsight Spark SQL Spark MLib Azure Data Lake Store U-SQL Data Orchestration/ Workflow Azure Data Factory Oozie for Azure HDInsight Kafka for Azure HDInsight (future) SQL Server Integration Services Azure Machine Learning R Server SQL Server R Services SSRS SharePoint BI Transactional systems Azure SQL DW SQL Server APS ETL Azure Event Hubs Data Generation Streaming Consumption Processing Storage Operational Analytical / Exploratory Data Warehouse Azure Website SSAS Spark MLLib

29. Why Benefits of running Hadoop in the cloud – Far outrun tradeoffs What Options to run Hadoop in the Cloud – IaaS, PaaS, Hybrid Hadoop Clusters in the cloud – Fully Managed Cluster Customizations – Immensely well leveraged How Architecture of a Cloud deployment – Simplify deployment Summary 29

30. Get started today! For more information on HDInsight visit: http://azure.com/hdinsight For more information on Data Lake visit: http://azure.com/datalake

31. http://microsoft-events.jp/mstechsummit/

32. Q&A [email protected] @ satonaoki