Securing your Big Data Environments in the Cloud

Big Data tools are becoming a critical part of enterprise architectures and as such securing the data, at rest, and in motion is a necessity. More so, when you’re implementing these solutions in the cloud and the data doesn't reside within the confines of your trusted data center. Also, there is a fine balance between implementing enterprise-grade security and negotiating utmost performance given the overheads of encryption and/or identity management.

This session is designed to tackle these challenges head on and explain the various options available in the cloud. The focal points are the implementation of tools like Ranger and Knox for cloud deployments, but we also pay attention to the security features offered in the cloud that complement this process and secure the data in unprecedented ways.

Cloud Security + OSS Security tools are a deadly combination, when it comes to securing your Data Lake.

1. Securing your Big Data environments in the Cloud Nishant Thacker Technical Product Manager Microsoft

2. What is so different about the cloud?

3. Traditional Hadoop Clusters – On Prem 4 Hadoop Cluster Worker Node HDFS HDFS HDFS Tasks Tasks Tasks Tasks Tasks Tasks Task Tracker Master Node Client Job (jar) file Job (jar) file

4. Hadoop Clusters in the Cloud

5. Decoupling Compute from Storage Latency? Consistency? Bandwidth? Network

6. Decoupling Compute from Storage Network HDD-like latency 50 Tb+ aggregate bandwidth [1] Strong consistency [1] Azure Flat Network Architecture

7. Decoupling - Benefits Cloud NoSQL Workload Pros Smaller clusters can achieve the same level of performance as large clusters No need to add nodes just for storage capacity Depending on workloads, you see any where from 6x – 20x cost benefits Query + ML Workload P ros Clusters required only while processing data Data Persists for tools to connect and use Data can be replicated on other geo Delete clusters when not processing of data Streaming Workload Pros No need for large clusters to hold historical streams data Directly align throughput to cluster size as per SLA Cluster up only when streams are active

8. Customize cluster? HDInsight cluster provisioning states Cluster customization options Ready for deployment Accepted Cluster storage provisioned AzureVM configuration Running Timed Out Error Cluster operational Configuring HDInsight Cluster customization (custom script running No Yes

9. OK, I understand the cloud has this optimized model for deployment, now help me SECURE IT Step 1 – Secure your data

10. Azure Data Lake (the cloud HDFS) Security Model Data Protection Store Authorization Authentication 2 Authenticate user and get group information 1 Check source IP address against Firewall rules 3 Authorize against POSIX ACLs on the file/folder 4 Encrypt/Decrypt data using the Data & Block Encryption Keys (Master Key is always in AKV) 5 Perform and Audit operation Auditing IP Firewall 1 2 3 4 5 OAuth Token Graph API

11. Secure Analytics on Azure Data Lake Data Protection Store Authorization Authentication Auditing OAuth Token Graph API SPI Kerb Analytic Engines have two options : Represent the caller (carry the user’s OAuth token) Represent itself (“engine”) using Service Principals Azure Data Lake Analytics: Represent the caller (carry the user’s OAuth token) HDInsight: Clusters can be configured with Service Principal Identities Can also pass through end user’s OAuth token SPI User OAuth Token

12. Contoso Big Data Pipeline People Automated Systems Apps Web Mobile Bots Action Intelligence Data Telemetry Feedback Sales Identity Management Azure Active Directory (contosodatalake.onmicrosoft.com) Power BI Dashboards & Visualizations Big Data Stores Data Lake Store Azure DW Azure Storage Landing Zone Data Prep, Analytics & ML HDInsight HDInsight

13. Getting Started

14. Connect to Azure Active Directory People Automated Systems Apps Web Mobile Bots Action Intelligence Data Telemetry Feedback Sales Identity Management Azure Active Directory (contosodatalake.onmicrosoft.com) Power BI Dashboards & Visualizations Landing Zone Data Prep, Analytics & ML HDInsight Big Data Stores Data Lake Store Azure DW Azure Storage HDInsight

15. I want to use AAD as identity provider for my big data architecture including Hadoop and Data Lake

16. Use Active Directory Domain Services and VNet Peering

17. Configure Active Directory Domain Services

18. Landing Zone - Data ingestion People Automated Systems Apps Web Mobile Bots Action Intelligence Data Telemetry Feedback Sales Identity Management Azure Active Directory (contosodatalake.onmicrosoft.com) Power BI Dashboards & Visualizations Landing Zone Data Prep, Analytics & ML HDInsight Big Data Stores Data Lake Store Azure DW Azure Storage HDInsight

19. I don’t want my cluster to be exposed to public internet

20. Securing Landing Zone Secure via Service principal & File and Folder ACLs Secure via access keys

21. Configure Network Security Group

22. Create a Kafka Cluster inside the VNet

23. Create a Spark Cluster on secure Data Lake Store

24. Configure access to Spark Cluster on Data Lake

25. Analytics using multi-user cluster People Automated Systems Apps Web Mobile Bots Action Intelligence Data Telemetry Feedback Sales Identity Management Azure Active Directory (contosodatalake.onmicrosoft.com) Information Management Azure Data Factory (data movement & job orchestration) Power BI Dashboards & Visualizations Landing Zone Data Prep, Analytics & ML HDInsight Big Data Stores Data Lake Store Azure DW Azure Storage HDInsight

26. I want sensitive information to be visible only to privileged users Example – I don’t want suppliers to see customer information

27. Behind the scenes

28. Apache Ranger Centralized Policy Management Portal. Open source industry standard for managing authorization policies. Extremely powerful auditing tool. Plugins available for various Hadoop components like Hive HBase Kafka Storm

29. ARM VNET Gateway Head Node HDInsight Cluster WASB ADLS Worker node (s) ThriftServer Zeppelin Kerberos AuthN Kerberos Ticket HTTPS Basic Auth OAuth Ticket Classic VNET Active Directory Domain Services AAD tenant Ranger DB LDAP Alice Bob Azure VNET to VNET peering Ranger

30. Apache Ranger in HDInsight

31. Why No Knox? Azure VNet HTTPS traffic ODBC/JDBC WebHCatalog Oozie Ambari Secure gateway AuthN HTTP Proxy Highly available Head nodes Worker nodes ADLS/ WASB

32. Sneak preview of Future releases

33. Ranger plugin for WASB No RBAC for file and folder level permissions Microsoft invested in a Ranger plugin for WASB. Similar to Ranger plugin for HDFS

34. Configuring policies in Ranger

35. WASB: File and Folder ACLs (Ranger)

36. In Conclusion …

37. © 2016 Microsoft Corporation. All rights reserved.