Some Advanced OpenStack Overview Document

Advanced OpenStack

1. ADVANCED OPENSTACK ADMINISTRATION 1

2. AGENDA Getting to know OpenStack Horizon Dashboard & OpenStack Client CLI Keystone Identity Management Glance – Image Management Nova – Compute Service Cinder – Block Storage Neutron – Networking Troubleshooting 2

3. I. Getting to know OpenStack 3

4. OpenStack Overview & History OpenStack Infrastructure as a Service – Cloud User Perspective High Level Overview of OpenStack Architecture Architecting & Implementing OpenStack Deployment 4

5. Objective What is OpenStack? OpenStack Community & Foundation Who is using OpenStack and how? Getting the Software 5 OpenStack Overview & History

6. Cloud Operating System Abstraction Layer for Infrastructure as a Service Collection of Projects implementing IaaS Services Well-defined set of Common APIs Self-service & Real-time Automation of Cloud Resources Programmable Infrastructure for Infrastructure as Code 6 What is OpenStack?

7. Open Source No, Enterprise Editions Apache License 2.0 Open Design Community controlled Design Process Project Teams Gatherings open to Everyone Open Development Everyone can participate at every stage Open Community Everyone has a voice 7 OpenStack – Open Sources Project

8. Board of Directors Technical Committee Active Technical Contributors (ATCs) User Committee Active User Contributors (AUCs) Project Teams: Project Team Leaders (PTLs) Active Project Contributors (APCs) Core Team Members 8 OpenStack Foundation

9. 9 OpenStack Projects

10. 10 OpenStack Roadmap

11. 11 OpenStack Release

12. 12 OpenStack Community Individual Members Corporate Members: Platinum, Gold, Sponsors Community Open Communication: IRC Channels Wiki Pages Etherpads Mailing Lists Community Events: Summit Project Team Gathering OpenStack Day & MeetUp User Groups

13. 13 OpenStack Publications Docs Guides Superuser OpenStack Foundation YouTube Channel Ask OpenStack The OpenStack Blog & News Analyst Reports

14. 14 Contributing to OpenStack Code and Documentation Events – Help to organize User Groups – Join & make them Alive Users – Share your Experience & Advocate New Features

15. 15 Typical OpenStack Deployments Private Cloud: Big Data, DBaaS, High Performance Computing, Enterprise Apps, Container Optimized, DevOps Public Cloud: Web Applications & Hosting, eCommerce, HPC Telco Cloud: NFV & SDN, Video Processing & Content Delivery

16. 16 OpenStack Foundation User Survey

17. 17 OpenStack Foundation User Survey

18. 18 OpenStack Users Academic/Research & Government – CERN, Postal Savings Bank of China Cloud Providers – Rackspace, T-Systems Open Telecom Cloud Telecom Carriers – AT&T, Verizon, China Mobile Finance – Insurance Australia Group, BBVA, China UnionPay Information Technology – Tencent, City Network, Liveperson Manufacturing – Volkswagen AG, JFE Steel Corporation, SGCC Retail/eCommerce – Snapdeal, Nike, eBay, GAP Inc, Bloomberg

19. 19 Getting OpenStack Software Community Version: Installation Tutorials for Ubuntu, Red Hat an SUSE Linux 3-5 Standard Servers at minimum Start with Core Projects, extend with Optional Projects Community Support Requires Skills & Experience Commercial Releases: Red Hat, Cannonical , SUSE, Huawei, Mirantis , IBM, Oracle, VMWare Integrated OpenStack Bug fixing and Security Hardening for Enterprise Grade Only a handful of Supported Modules Vendor Support, Architecture & Deployment Services Vendor System Management Services

20. Objective High Level Overview of OpenStack Services Consumption Models Detailed Review of Services 20 OpenStack Infrastructure as a Service

21. 21 Using OpenStack Services Manual, On-demand: Horizon Dashboard – GUI available via Web Browser OpenStack Client – Command Line Interface available on Linux, Windows and MacOS Cloud Management Platforms – ManageIQ / CloudForms , InContinuum Cloud Controller, Scalr Automated, Infrastructure-as-Code: Heat Orchestration Templates Ansible Playbooks Puppet Modules Chef Cookbooks Shell Scripts with OpenStack Client

22. 22 Compute Services Major Compute Abstractions available in OpenStack: Virtual Machine Bare Metal Server Container Orchestration Engine Container (Docker) Other Abstractions in Compute Services: Image, Flavor, Key Pair, Server Group, Availability Zone, Host Aggregate, Cluster, Cluster Policy Common User Interface regardless of: Hypervisor type (KVM, VMware or Xen) managing virtual machine Power and Management (IPMI or Redfish), Boot Interface (PXE or Virtual Media) Particular COE (Kubernetes, Docker Swarm or Apache Mesos)

23. 23 Compute Projects Nova – COA Focus Project, delivery and management of Compute Instances Glance - COA Focus Project, management of Instance Images Ironic – delivery and management of Bare Metal Compute Instances Magnum – provisioning and management of Compute Orchestration Engines Zun – delivery and management of Container Instance Senlin – standard OpenStack Clustering Service Storlets – implements Functions as a Service in context of Object Storage

24. 24 Security, Identity & Compliance Services Major abstractions available in this Security Group: Region, Domain, Project, Group, User, Role Service, Endpoint, Catalog of Services Secret, Secret Store Policy Workload, Task, Action, Cron-trigger, Workflow Workbook Common User Interface regardless of: Identity Database (LDAP, AD, SQL, … ) Secret Store implementation: Database, HSM, …

25. 25 Security, Identity & Compliance Projects Keystone – core Identity Project of OpenStack Barbican – secure provisioning, management and storage of secrets (password, X.509 certificates, encryptions keys) Congress – Governance Service Project Mistral – Workflow Service Project

26. 26 Storage, Backup & Recovery Services Major abstractions available in this Security Group: Volume, Volume Snapshot, Consistency Group, Backup Object, Object Container Share, Filesystem Snapshot Common User Interface regardless of: Storage Backend implementation: Disk Array, Storage Appliance, Software Define Storage Backup Technology

27. 27 Storage, Backup & Recovery Projects Cinder – core Block Storage Service Project Swift – core Object Storage Service Project Manila – File Share Service Project Karbor – deployed Application Data and Meta-Data backup & restore Freezer – file and filesystem Backup & Recovery Service

28. 28 Networking & Content Delivery Services Major abstractions available in this Security Group: Network, Subnet, Port Floating IP, Router Security Group, Firewall, VPN Load Balancer Port Chain, Port Group, Port Pair, Flow Classifier Zones, PTR Records Virtual Network Function Common User Interface regardless of: Underlying Physical Network technology Usage of SDN

29. 29 Networking & Content Delivery Projects Neutron – core OpenStack Networking Project Designate – multi-tenant DNSaaS Dragonflow – distributed SDN controller for Neutron Kuryr – Docker network plugin Octavia – LBaaS Project Tacker – Virtual Network Function Manager Project Tricircle – Extending Network abstractions across multiple OpenStack instances

30. 30 Data & Analytics Services Major abstractions available in this Security Group: Datastore Type, Database Instance, Database Configuration Group, Database Backup, Database Replication Data Analytics Cluster, Cluster Template Common User Interface regardless of: Database Engine (MySQL, Cassandra, CouchBase , CouchDB, DB2, MariaDB, MongoDB, Percona , PostgresSQL , Redis, Vertica) Data Analytics Engine (Hadoop, Spark, Storm)

31. 31 Data & Analytics Projects Trove – DBaaS Project Sahara – Data Analytics Project Searchlight – provides indexing and search capabilities across OpenStack resources

32. 32 Application Services Major abstractions available in this Security Group: Orchestration Template, Orchestration Stack Application Package, Application Environment, Application Catalogue Message, Message Queue, Data Processing Node Group, Data Processing Cluster

33. 33 Application Projects Heat – core Orchestration Project Zaqar – Message Queue as a Service Murano – Application Catalog Service Solum – aPaaS ( Appliction Platform as a Service) Project

34. 34 Monitoring & Metering Services Major abstractions available in this Security Group: Alarm Meter Event Publisher Transformer Pipeline

35. 35 Management Projects Horizon – Dashboard (GUI) Service OpenStack Client – primary Command Line utility Rally – OpenStack deployment Benchmarking & Profiling Service Vitrage – Root Cause Analysis Service Watcher – Infrastructure Optimization for Projects

36. 36 Deployment Tools Kolla – Container-based Deployment for OpenStack Cloud TripleO – OpenStack-Over-OpenStack OpenStack-Ansible – OpenStack Cloud installation with Ansible Chef OpenStack – Chef Cookbooks for OpenStack deployment Puppet OpenStack – Puppet Modules for OpenStack deployment OpenStack Charms – Juju Charms for OpenStack deployment

37. 37 Monitoring & Metering Projects Ceilometer – primary Telemetry Service Aodh – Alarming Service Gnocchi – Time Based Series Database Service Panko – Event Storage Service Monasca – Highly-scalable Monitoring as a Service

38. Objective General OpenStack Architecture Anatomy of OpenStack Service Mapping Services to Physical Servers Control Plane and User Plane Components 38 High Level Overview of OpenStack Architecture

39. 39 General OpenStack Architecture

40. 40 Anatomy of OpenStack Service

41. 41 Launching a New Instance #source user_x-openrc #openstack server create --image a --flavor z -- nic net-id=b new_vm

42. 42 Launching a New Instance CLI Client Keystone 1. Authenticate user “x”

43. 43 Launching a New Instance CLI Client Keystone 2. Token for user “x”

44. 44 Launching a New Instance CLI Client Keystone 3. Give me endpoint for Compute Service Nova

45. 45 Launching a New Instance CLI Client Keystone 4. Here is the endpoint Nova

46. 46 Launching a New Instance CLI Client Keystone 5. Launch new instance with image “a”, connect to network “b”, here is the token for user “x” Nova

47. 47 Launching a New Instance CLI Client Keystone 6. Validate the token and access permissions for user “x” Nova

48. 48 Launching a New Instance CLI Client Keystone 7. Validate, OK Nova

49. 49 Launching a New Instance CLI Client Keystone 8. Give me image “a” for user “x” Nova Glance

50. 50 Launching a New Instance CLI Client Keystone 9. Validate token for user “x” Nova Glance

51. 51 Launching a New Instance CLI Client Keystone 10. Validate, OK Nova Glance

52. 52 Launching a New Instance CLI Client Keystone 11. Here is the image “a” Nova Glance

53. 53 Launching a New Instance CLI Client Keystone 12. Give me a port on network “b” for user “x” Nova Glance Neutron

54. 54 Launching a New Instance CLI Client Keystone 13. Validate token for user “x” Nova Glance Neutron

55. 55 Launching a New Instance CLI Client Keystone 14. Validate OK Nova Glance Neutron

56. 56 Launching a New Instance CLI Client Keystone 15. Here is the port Nova Glance Neutron

57. 57 Launching a New Instance CLI Client Keystone 16. Give me a volume for user “x” Nova Glance Neutron Cinder

58. 58 Launching a New Instance CLI Client Keystone 17. Validate token for user “x” Nova Glance Neutron Cinder

59. 59 Launching a New Instance CLI Client Keystone 18. Validate OK Nova Glance Neutron Cinder

60. 60 Launching a New Instance CLI Client Keystone 19. Here is the volume Nova Glance Neutron Cinder

61. 61 Launching a New Instance CLI Client Keystone 20. Instance spawned Nova Glance Neutron Cinder

62. 62 Mapping Service Components to Hosts Control Nodes: 1 – 3+: Linux Hosts Infrastructure Components (Database, Message Queue, Memory Object Caching, HTTP Server) OpenStack Services app Compute Nodes: 2+: OS plus Hypervisor, Compute Service Agents, Network Agents, Telemetry Agents Storage Nodes: 2+: Linux, Block Storage Agents, Object Storage Agents, Fileshare Storage Agents, Telemetry Agents Network Nodes: 1+: Linux, L3 Agent, DHCP Agent, LBaaS , VPNaaS , FWaaS , Telemetry Agents

63. Objective OpenStack Infrastructure Components Implementing Control Plane Architecting Compute Resource Pool Implementing OpenStack Underlay Network Architecting and Implementing Storage Backend Considerations for other Services 63 Architecting & Implementing OpenStack Deployment

64. 64 OpenStack Infrastructure Components HTTP Server – Apache2 + WSGI, Nginx SQL Database – MySQL, MariaDB, PostgreSQL Advanced Message Queue Protocol (AMQP) Server – RabbitMQ, Qpid , Zeromq Memory Object Caching Server - Memcached

65. 65 Control Plane Implementation Infrastructure Components API Endpoints Conductor Services Scheduling Services User Dashboard Authentication and Authorization for Identity Management Image Management Service

66. 66 Control Plane High Availability Galera Cluster for Database HAProxy as Load Balancer for API Endpoints HA Systemd for local Host HA Pacemaker – Linux HA Technology Collapsed – every Component run on every node Segregated – every Component runs in own 3+ node cluster Mixed – select best strategy for each Component

67. 67 Compute Resource Pool Virtual Machine Instances – Hypervisors: Linux KVM Xen VMware ESXi /vCenter Hyper-V Bare Metal Server Pools Container Instances – Docker Hosts Availability Zones Host Aggregates

68. 68 Compute Hardware Considerations Instance Density & Workload Patterns: Popular Instance Flavors Server Configuration vs Cost COTS Rack Server vs Blade/HCI Power & Cooling Density OpenStack Overcommit Ratios: CPU – 16:1 (default) RAM – 1.5:1 (default) Hardware Acceleration: GPUs SSD or Flash PCI cards

69. 69 Physical Network of OpenStack Instance Traffic Segregation: management, external (provider network), storage, internal user Hardware-based Network Security Redundancy – multiple NICs in Compute Nodes, switching layer redundancy Network Performance: Throughput Latency QoS IPv6 Support Network Acceleration Intelligent NICs SRIOV Hardware VTEPs – VXLAN Termination in Switches

70. 70 SDN in OpenStack Implementation User Workload Networking: Direct mapping to underlying VLANs Virtual L2 Subnets and Tunneling: VXLAN, GRE Linux Networking – LinuxBridge Open Virtual Switch OVS Open Source or Commercial SDN Controller: Scalability Orchestration of existing DC Networking OpenContrail , OpenDaylight , VMware NSX, Cisco

71. 71 Storage Backend Implementation Hardware Storage Backends: Disk Arrays Performance Redundancy & Availability Dell EMC, NetApp, HPE Software Defined Storage Backends Scalability & Flexibility Reliability & Availability LVM, Ceph , GlusterFS Data Path between Compute Nodes and Storage Nodes: Local Storage on Compute Nodes Fiber Channel Ethernet – iSCSI, FCoE

72. 72 Other Services Telemetry Services (ceilometer, aodh , gnocchi, panco , monasca ) Database Service Data Analytics Service Container Orchestration Engine Service

73. II. OpenStack Dashboard and Command Line Clients 73

74. OpenStack Dashboard & Command Line Client Dashboard & CLI Summary and Review 74

75. Objective Introduction to OpenStack Dashboard OpenStack Command Line Client Command Line Clients Configuration Options OpenStack command Structure and Arguments 75 OpenStack Dashboard & Command Line Clients

76. 76 Horizon Dashboard Introduction Available through Web Browser Typical URL: http://management_vip/horizon Login with Username and Password, optional Domain name User Menu: Settings Themes Project Menu: List of User’s projects to set Project Scope Main Menu: Project Menu Admin Menu Identity Menu Horizon Plugins

77. 77 OpenStack Command Line Client Primary CLI command: openstack Project specific command: cinder glance keystone neutron nova swift

78. 78 CLI Clients Configuration Options Authentication Parameters: Domain Project Username Password Authentication URL Identity API Version Environment Variables Command execution options Clouds: yaml File Token/URL Authentication Type

79. 79 CLI Clients Environment Variables $ export OS_USERNAME=demo $ export OS_PASSWORD= openstack $ export OS_PROJECT_NAME=demo $ export OS_USER_DOMAIN=Default $ export OS_PROJECT_DOMAIN_NAME= Default $ export OS_AUTH_URL=http://10.0.0.11:35357/v3 $ export OS_IDENTITY_API_VERSION=3 $ openstack [command] [options] openrc scripts are used set Environment Variables before command execution

80. 80 CLI Clients Execution Options $ openstack -- os -username demo \ -- os -password openstack \ -- os -project-name demo \ -- os -user-domain-name Default \ -- os -project-domain-name Default \ -- os -auth- url http://10.0.0.11:35357/v3 \ -- os -identity- api -version 3 \ [command] [options]

81. 81 clouds.yaml File ~/.config/ openstack / clouds.yaml clouds: demo: auth: auth_url : http://10.0.0.11:35357/v3 project_name : demo username: demo password: openstack region_name : RegionOne $ openstack -- os -cloud demo [command] [options] Or $ export OS_CLOUD=demo $ openstack [command] [options]

82. 82 openstack Command structure and arguments General structure: $ openstack [global-options] [object] [action] [command arguments] where: [global-options] are the -- os -…options, for example setting authorization parameters [object] – specifies what type of Service is in scope of the command, examples on next page [action] – specifies what we want to do with [object], examples on next page [command arguments] – additional parameters, necessary to perform the command

83. 83 Command [object] and [action] Example [object] endpoint flavor keypair image network router server user volume Example [action] add create delete issue list migrate remove show start Example commands: $ openstack token issue $ openstack keypair delete my-keypair $ openstack server start web- vm

84. 84 openstack Command Interactive Mode $ export OS_CLOUD=admin $ openstack ( openstack ) image list ( openstack ) token issue ( openstack ) exit

85. III. Keystone Identity Management 85

86. Understanding OpenStack Identity Management 86

87. Objective Keystone Overview and Architecture Keystone Services Resource Management Hierarchy Detailed Overview of Identity Objects Service Catalogue & Endpoints 87 Understanding OpenStack Identity Management

88. 88 Keystone Overview What it is? OpenStack Users – who is required to authenticate? What services it delivers? How this services are delivered?

89. 89 Keystone Architecture Serving REST API calls on ports 5000 and 35357 Universal Unique Identifiers - UUIDs Keystone backends Identity – SQL, LDAP, SAML v2, Oauth v1.0a, OpenID Connect Other – SQL, Memcache , Files Regions – each may have own Keystone Instance Federated Keystone – multiple Regions sharing a common Keystone Database

90. 90 Keystone Services Identity – Users and Groups Resource – Projects (Tenants) and Domains Assignment - Roles Token Catalog and Endpoint Policy

91. 91 Resource Management Hierarchy Domain A Domain B Domain C Group 1 Group N Role: user Role: admin Role: admin X Y Z VM1 VM2 VM3 Project Y Project X

92. Domain 92 Contains Users, Groups & Projects Properties Actions list description create show name set delete enabled

93. Project 93 Contains Resources (Networks, Instances, etc ) Properties Actions list description create show name set delete enabled is_domain domain_id

94. User 94 Properties Actions list description create show name set delete enabled domain_id project_id password email

95. Group 95 Properties Actions list description create show name set delete domain_id add user remove user contain user Contains Users

96. 96 Roles Implement Role Based Access Control (RBAC) User is assigned a Role in Project within Domain Role name is arbitrary, most often: admin, user/_member_ Role names are lookup by a Service in policy.json file to determine user’s authorization for Service actions: “ create_subnet ”: “ rule:admin_or_network_owner ”

97. 97 Service Catalog & Endpoints Service Catalog enables standardized discovery of Service API Endpoint –URLs targets for REST calls. Service Properties: name type description enabled Service Name & Type examples: Keystone – identity Nova – compute Cinder – volume Neutron - network

98. 98 Service Catalog & Endpoints Endpoint Types: Public – for Cloud Users, URL on Public network Internal – for Cloud Users, URL on Internal network Admin – for Cloud Admins & Operators, URL on secure Internal or Admin network Endpoint Properties: Region Service_name Service_type Interface url Enabled Endpoint Examples: http://10.0.0.11:9292 http://controller:35357/v3 http://controller:8080/v1/AUTH_%(tenant_id)s

99. 99 Service Catalog & Endpoints Service Operations create delete set list show Endpoint Operations create delete set list show

100. 100 Operations available CLI only Domain Operations: create, delete, list, set, show Service Operations: create, delete, set Endpoint Operations: create, delete, set All operations on Other Domains (non default)

101. IV. Glance – Image Management 101

102. Inside of Glance 102

103. Objective Introduction to OpenStack Images Glance Overview and Architecture Image Types, Properties and Actions Downloading ready-made Cloud Images Creating Images Modifying and Converting Images 103 Inside of Glance

104. 104 OpenStack Images Server Image is a file containing ready to run, bootable virtual disk Virtual Server in OpenStack is a Running Instance of an Image Image must include some sort of an Operating System OpenStack supports Linux and Windows images by default OpenStack Images include other mandatory software components, like cloud- init Image may contain middleware or application software, like Apache, MySQL and PHP – full LAMP Stack Glance can store any Binary Blob as an Image

105. 105 Glance Overview & Architecture Image Service manages and stores Server Images for Virtual Server Instances, Bare Metal Server Instances and Zun Containers Image Service API @ http://controller:9292 Glance Registry File system Swift HTTP SQL DB Glance Backend

106. 106 From Image Store to running Instance Nova Compute Agent KVM + Libvirt Volume API Volume Backend Image Service API Image Storage Backend Compute Node Volume Service Image Service

107. 107 From Image Store to running Instance Nova Compute Agent KVM + Libvirt Volume API Volume Backend Image Service API Image Storage Backend Compute Node Volume Service Image Service server 1 boot disk local disk Persistent disk /dev/ sda /dev/ sdb /dev/ sdc Step 1: Create new Server Instance

108. 108 From Image Store to running Instance Nova Compute Agent KVM + Libvirt Volume API Volume Backend Image Service API Image Storage Backend Compute Node Volume Service Image Service server 1 /dev/ sda /dev/ sdb /dev/ sdc Step 2: Get the Image

109. 109 From Image Store to running Instance Nova Compute Agent KVM + Libvirt Volume API Volume Backend Image Service API Image Storage Backend Compute Node Volume Service Image Service server 1 /dev/ sda /dev/ sdb /dev/ sdc Step 2: Get the Image base Instance disk

110. 110 From Image Store to running Instance Nova Compute Agent KVM + Libvirt Volume API Volume Backend Image Service API Image Storage Backend Compute Node Volume Service Image Service server 1 /dev/ sda /dev/ sdb /dev/ sdc Step 2: Get the Image base Instance disk

111. 111 From Image Store to running Instance Nova Compute Agent KVM + Libvirt Volume API Volume Backend Image Service API Image Storage Backend Compute Node Volume Service Image Service server 1 /dev/ sda /dev/ sdb /dev/ sdc Step 3: Create local disk base Instance disk

112. 112 From Image Store to running Instance Nova Compute Agent KVM + Libvirt Volume API Volume Backend Image Service API Image Storage Backend Compute Node Volume Service Image Service server 1 /dev/ sda /dev/ sdb /dev/ sdc Step 4: Attack persistent disk base Instance disk

113. 113 Image Type & Formats OpenStack Image Service supports multiple Compute Abstractions: Virtual Machines, Bare Metal Servers and Containers OpenStack Image Service also stores Images for: Magnum – Container Orchestration Engine Service Trove – Database Service Sahara – Data Processing Service Tacker – NFV Orchestration Service

114. 114 Image Type & Formats OpenStack Image Service supports following Disk Formats aki , ami , ari iso qcow2 raw vdi vhd , vhdx vmdk , ova

115. 115 Image Actions Create Delete List Show Set Unset Save Add project – make Image available in given Project Remove project – remove the Project from image

116. 116 Image requirements - Linux Disk partitions and filesystems allow resizing on boot MAC addresses removed SSH server installed and running Firewall disabled Cloud- init installed and running Accept Public Key for SSH and User Data for startup script Paravirtualized Xen support in Linux kernel (if you plan to use Xen Hypervisor)

117. 117 Downloading ready-made Cloud Images Preconfigured Cloud Images available from Major Linux Distributions: CentOS, CirrOS , Debian, Fedora, Ubuntu, SUSE, RHEL Windows Cloud Images for OpenStack maintained by Cloudbase Solutions

118. 118 Creating custom Images Tools to create Cloud Images: Diskimage -builder Oz VeeWee Packer Image-bootstrap Imagefactory KIWI SUSE Studio virt -builder

119. 119 Creating custom Images Creation process: Launch an Instance from a standard Image SSH to the Instance Install additional software Customize System Configuration Shutdown the Instance Snapshot the Instance Create a Volume from the Snapshot Create an Image from the Volume

120. 120 Modifying and Converting Images Guestfish and virt -builder support extensive Image customization. Qemu-img converts Images to formats: Qcow2 (KVM, Xen, QEMU) Qed (KVM) Raw Vdi (VirtualBox) Vpc (VHD – HyperV ) Vmdk

121. 121 Operation available in CLI only Download Images Share Images with other Projects

122. V. Nova – Compute Service 122

123. Compute Service Introduction Virtual Server Instances – Deep Dive 123

124. Objectives Compute Service Architecture Implementation of Compute Hosts in User Plane Managing Flavors Managing Keypairs Security Groups Resource Quotas Manage Compute Hosts Compute Service Monitoring 124 Compute Service Introduction

125. 125 Compute Service Architecture Keystone Neutron Glance Cinder NovaAPI Nova Conductor Nova Scheduler Compute Host Compute Host SQL DB

126. 126 Compute Hosts in User Plane Keystone Neutron Glance Cinder NovaAPI Nova Conductor Nova Scheduler nova-compute KVM Host Compute Host nova-compute Nova ESX Host Bare Metal Host zun -compute Docker Host ESX Host1 ESX Host1 ESX Host1 vCenter Ironic API Ironic Conductor Zun API Zun wsproxy $ openstack server … $ openstack appcontainer …

127. 127 Managing Keypair Keypair – Public and Private Keys Public Key injected to an Instance at boot time Private Key stored in secured file on local system Private Key used by ssh command to access an Instance Public Key owned by User, not Project Key actions: Create Delete List Show

128. 128 Managing Flavors Flavor is a Virtual Server Architecture Template Flavor Properties: Name vCPU RAM Disk Ephemeral Swap Rxtx Is_public Extra specs

129. 129 Managing Flavors Flavor actions: Create Delete List Show Set Unset Flavor Name vCPUs RAM Disk(GB) m1.tiny 1 512 1 m1.small 1 2048 20 m1.medium 2 4096 40 m1.large 4 8192 80 m1.xlarge 8 16384 160

130. 130 Security Groups Security Groups control incoming and outgoing network traffic of the Instance Each Instance has at least one Security Group assigned Security Group has 1 or more Rules, controlling particular traffic pattern Each Project has got a default Security Group, which blocks all incoming traffic and allows all outgoing traffic Instances on the same Subnet communicate without limits, by default Set allow_same_net_traffic to false in / etc /nova/ nova.conf to enforce Security Groups for all traffic.

131. 131 Security Groups Properties: Name Description Project_id Revision_number Created_at Updated_at Rules Actions: Create, detelet , list, show, set

132. 132 Security Group Rules Rule define allowed traffic pattern Rules are either ingress or egress Rule Definition specifies Protocol – TCP/UDP, ICMP or any other IP based Protocol, defined as Interger 0:255 For ICMP protocol ICMP Code and ICMP Type can be specified, otherwise all ICMP traffic is regulated. For TCP and UDP destination port or port range (123:125) can specified Remote IP defines address range in from of CIDR (10.0.1.0/24) Remote Group – if present, rule allows traffic with all instances having specified Security Group assigned.

133. 133 Security Group Rules Rules Properties: Created_at Updated_at Description Project_id Security_group_id Revision_number Direction – ingress or egress Ethertype – IPv4 or IPv6 Port_range_min Port_range_max Protocol – TCP, UDP, ICMP Remote_group_id Remote_ip_prefix – for example 0.0.0.0/0 for all IP addresses

134. 134 Resource Quotas OpenStack maintains basic control mechanism of workload size – Resource Quotas Quotas are set by Project, can also be set by User There are default values assigned at Project creation Cloud Admin can alter specific values at any time

135. 135 Resource Quotas List of Available Quotas Compute Block Storage Network Cores Volumes Networks Instances Snapshots Subnets Key-pairs Backup Routers Ram Gigabytes Subnetpools … … …

136. 136 Resource Quotas Quota Operations: openstack quota set openstack quota show nova quota-show --user < projectUser > --tenant < project_id > Show current Quota usage for the Project nova limits --tenant < project_id >

137. 137 Manage Compute Hosts Compute Host must be disabled for Maintenance openstack compute service set --disable --disable-reason “maintenance” coa -lab nova-compute Enable Compute Host after Maintenance openstack compute service set --enable coa -lab nova-compute Instances running on Compute Host can be: Live Migrated: opentack server migrate <instance> --live TargetHost Evacuated: nova evacuate < instance_id > TargetHost

138. 138 Monitor Compute Service List all Hosts and their Services $ openstack host list Show Host statistics $ openstack host show coa -lib $ openstack hypervisor show coa -lab Show Instance statistics $ openstack server list $ openstack diagnostics myInstance Summary Usage Statistics for Projects $ openstack usage list

139. Objectives Instance Overview Instance Properties Launching a New Instance Star, Stop, Delete and other Actions Manage NICs & IP Addresses of an Instance Accessing an Instance with Nova host consoles Manage Instance Snapshots 139 Virtual Server Instance

140. 140 Instance Overview OpenStack Instance is based on Linux Virtual Machine Model, extended to cover Bare Metal Servers. Virtual Server Instance – “hardware” configuration: Virtual CPUs (Cores) Virtual RAM Virtual I/O Devices: Disks (Block Volumes), Network Cards ( vNICs ) Nova Scheduler selects Compute Node to run an Instance, based on: Flavor – “hardware” configuration Properties (architecture,…) and Metadata (GPU required, …)

141. 141 Instance Properties Name– OpenStack Instance Name OS-EXT-SRV-ATTR:instance_name – Instance name in hosting Hypervior Image – name of the image used to create the Instance ….

142. 142 Launching a New Instance Minimum parameters: Name Flavor Image Network Optional parameters: Security Groups Keypair User Data Files to inject at boot time Server Groups Availability Zone Scheduler Hints Metadata

143. 143 Cloud- init & User Data Check Cloud Init Community Page User Data can be a shell script, include file or list of HTTP URLs to download and interpret To run a shell script at rc.local boot sequence, put a first line: #! /bin/ sh Place an arbitrary file in the new instance with: $ openstack server create … --file destination-file=source-file

144. 144 Instance Actions Basic Instance Management Actions: Create Delete List Show Set Unset Block Instance Access to Admin Users: Lock Unlock

145. 145 Instance Actions Virtual Machine State Management: Start, stop, reboot (hard, soft) Pause, unpause Suspend, resume Shelve, unshelve

146. 146 Instance Actions Virtual Machine Configuration Management Actions: Resize Rescue, unrescue Rebuild Migrate

147. 147 Instance Actions Instance Data Management Actions: Add volume, remove volume Dump create Image create

148. 148 Manage Instance NICs & IP Addresses Instance must have at least one Virtual Network Interface Card ( vNIC ), can have many Additional vNIC can be attached during Launch or later Each vNIC must be connected to a Network or Port Attached vNICs must be configured in Instance’s Operating System Each vNIC has got unique MAC and IP addresses IP address types: DHCP Fixed IP Floating IP

149. 149 Instance Actions Instance Network Management Actions: Add fixed ip , remove fixed ip Add floating ip , remove floating ip Add security group, remove security group

150. 150 Accessing Instances Use Nova Console: Horizon Dashboard CLI: $ openstack console url show myFirstInstance -- novnc View console log of instance $ openstack console log show myFirstInstance Security Groups (allow SSH), Floating IP

151. 151 Manage Instance Snapshots Instance Snapshot preserves its boot disk state only Snapshot stopped Instance, or make sure to flush buffers and synchronies disk Create Snapshot in Horizon Dashboard or CLI: $ openstack server image create --name < snap_name > <instance> Instance Snapshot creates actual Boot Volume Snapshot and a New Image based on this Snapshot Cloud User can immediately launch a new Instance with Instance Snapshot

152. VI. Cinder – Block Storage 152

153. Understanding Block Storage 153

154. Objectives Block Storage Service Architecture Block Storage Backends Managing Volume Quotas Creating and Managing Volumes Attaching Volume to Instance Volume Snapshots Volume Backups Encrypted Volumes Block Storage Monitoring 154 Understanding Block Storage

155. 155 Architecture of Block Storage Service CLI Client Cinder-API Cinder Scheduler Cinder Volume Cinder Backup VM Instance Driver Driver Swift Container SQL Database Message Queue Data Paths Storage Backend (SAN)

156. 156 Block Storage Backends Cinder support multiple Backends Drivers: Ceph RADOS LVM Dell EqualLogic EMC VNX, VMAX, ScaleIO and XtremIO Fujitsu Hitachi HPE IBM NetApp Pure Storage …

157. 157 Block Storage Backends Cinder-Volume exposes Storage Pools to enable Volume Scheduling to specific Storage Devices and/or Locations Storage Pools are defined by Cloud Operator in cinder.conf file: enabled_backends = lvm , lvm-2 [ lvm ] volume_group =cinder-volumes volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver volume_backend_name =LVM [lvm-2] volume_group =cinder-volumes-2 volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver volume_backend_name =LVM-2

158. 158 Managing Volume Quotas To show Block Storage related Quota: $ cinder quota-show <project> $ cinder quota-usage <project> To set Quota for a Project $ openstack quota set --snapshots 50 <project>

159. 159 Creating and Managing Volumes Volume Properties Name Description Availability_zone Os-vol-host-attr:host Bootable Encrypted Properties Size Status Type Volume Actions Create Delete List Show Set Unset

160. 160 Creating and Managing Volumes Volume can be created as Empty Volume: $ openstack volume create … <name> Volume can be created by cloning Another Volume: - - source <volume> A Volume Snapshot: - - snapshot <snapshot> An Image: - - image <image> Volume can be migrated $ openstack volume migrate - - host host@backend#pool <volume> Volume resize (only increase): $ openstack volume set - - size 15 <volume>

161. 161 Creating and Managing Volumes Attach a Volume to an Instance: $ openstack server add volume - - device /dev/ vdb <instance> <volume> Login to an Instance, check if device available: # fdisk -l Format the device # mkfs.ext3 /dev/ vdb Mount the device # mount /dev/ vdb / mnt Unmount the device # umount /dev/ vdb Detach the volume from an Instance $ openstack server remove volume <instance> <volume>

162. 162 Creating and Managing Volumes Volume can be transferred from one owner to another with Transfer Request Initiate the transfer: $ cinder transfer-create - - name <name> <volume> Accept the transfer: $ cinder transfer-accept <transfer> < auth_key >

163. 163 Volume Snapshots Volume Snapshot Properties: Description Name Properties Size Status Volume_id Volume Snapshot Actions: Create Delete List Show Set, unset

164. 164 Volume Snapshots To create a Volume Snapshot: $ openstack snapshot create - - force - - volume <volume> < snap_name > There is NO Snapshot Restore “in place” Snapshot can be cloned into a New Volume: $ openstack volume create … - - snapshot <snapshot> … <name>

165. 165 Volume Backups Volume Backup is a copy of Volume Data, stored in other location Cinder Volume Backup drivers: Ceph NFS Filesystem Swift Google Cloud Storage IBM Tivoli

166. 166 Volume Backups Backup can be created from Inactive/Detached Volume, or - - force option is required Backup can be created from specific Snapshot of The Volume, - - snapshot option Restore the Backup: $ openstack volume backup restore <backup> <volume> <volume> can be any volume with matching size

167. 167 Encrypted Volumes Block Volume Service supports Volume encryption on per-tenant (project) basis Volume Encryption encodes data stored in Backend, but also encodes data transmitted between Backend and an Instance Volume Encryption can be integrated Key Management Service (Barbican)

168. 168 Encrypted Volumes To encrypt Volumes, Cloud Admin has to create Encrypted Volume Type: $ openstack volume type create LUKS And add encryption details to the Volume Type: $ cinder encryption-type-create - - cipher aes-xts-plain64 - - key_size 512 - - control_location front-end LUKS nova.volume.encryptors.luks.LuksEncryptor Create encrypted Volume $ openstack volume create - - size 1 - - type LUKS <volume>

169. 169 Block Storage Monitoring Status of Block Volume Service: $ openstack volume service list Display Backend Host Capabilities: $ cinder get-capabilities host@backend Display Pools details: $ cinder get-pools - - detail

170. VII. Neutron – Networking 170

171. Network Service Architecture Virtual Network Resources 171

172. Objectives Network Service Architecture Neutron Implementation with LinuxBridge Network flows 172 Network Service Architecture

173. 173 OpenStack Virtual Network Overview Subnet 1 192.168.1.0/24 Subnet 3 192.168.3.0/24 Network 1 Subnet 2 192.168.2.0/24 Network 2 Subnet 4 192.168.4.0/24 Provider Network VM1 VM2 SG1 SG2 VM3 SG3 Internet vRouter 1 vRouter 2 Layer 2 Connectivity

174. 174 OpenStack Virtual Network Overview Subnet 1 192.168.1.0/24 Subnet 3 192.168.3.0/24 Network 1 Subnet 2 192.168.2.0/24 Network 2 Subnet 4 192.168.4.0/24 Provider Network VM1 VM2 SG1 SG2 VM3 SG3 Internet vRouter 1 vRouter 2 Layer 3 Addressing Floating IP

175. 175 Network Service Architecture Internet Network Node Control Node Compute Node Neutron-server Cinder Keystone Nova Glance Neutron-plugin-agent L3 Agent Neutron-metadata-agent DHCP Agent LBaaS , FWaaS Nova-compute Neutron-plugin-agent Management Network API Network Tenant Network External Network

176. 176 Implementation with LinuxBridge Internet Network Node Compute Node 2 DHCP Namespace qdhcp1 External Bridge Router Namespace qrouter2 Router Namespace qrouter1 Management Network Tenant Network External Network Compute Node 1 eth1 eth1 eth1 br -int br -int br -int eth2 DHCP Namespace qdhcp2 qg qr qg qr Tunnel Bridge Tunnel Bridge VXLAN Iface VXLAN Iface Port tap VM1 VM2 iptables2 Tunnel Bridge VXLAN Iface eth0 eth0 tap0 iptables1 Tunnel Bridge VXLAN Iface tap0 VM4 iptables3 iptables4 Tunnel Bridge VXLAN Iface eth0 tap0 VM3 eth0 tap1 eth0 eth0 eth0

177. Objectives Network & Subnets Routers Floating Ips Network Service Quotas Network Interfaces in Instance Monitoring Network Service 177 Virtual Network Resource

178. 178 Network & Subnets Self-service Networks – Project Networks – enable regular Cloud Users to provision basic network connectivity on demand, in real time, without Network Admin invention Provider Networks – External Networks – map to existing physical networks in the Data Center, provide L2 and L3 connectivity between Self-service Network and Internet Neutron (Virtual) Network provides L2 connectivity for Instance, is a L2 Broadcast Domain, can be configured with DHCP Neutron Subnet is IP Address Segment, providing IPv4 and/or IPv6 address space to the Network Neutron Port has a MAC & IP addresses, plus Security Groups, can be attached to an Instance, detached to another.

179. 179 Network & Subnets Network Properties: Name Description Project_id Status Mtu … Network Actions: Create Delete List Show Set …

180. 180 Network & Subnets Subnet Properties: Name Description Project_id Network_id CIDR … Subnet Actions: Create Delete List Show Set Unset …

181. 181 Network & Subnets Create a Project/Tenant Network $ openstack network create - - dns -namespace 8.8.8.8 project-private-net1 Create an External Network $ openstack network create - - provider-physical-network provider – provider-network-type flat – external - - share public-net Create a Subnet $ openstack subnet create - - network project-private-net1 – subnet-range 192.168.100.0/24 private-net1-subnet

182. 182 Routers Router provides L3 connectivity between Networks Two major types: East-West Router – connects 2 or more Project Networks Nort-South Router – connects 1 or more Project Networks to an External Network Nort-South Router has an External Gateway Interface, connected to External Network Nort-South Router perform SNAT on internal addresses Instance can become an IP destination with Floating IP only, which is One-to-One NAT

183. 183 Routers Router Properties: Name Description Project_id Routes … CLI command to show Router Interface $ openstack port list - - router <router> Router Actions: Create Delete List Show Set Unset Add port Add subnet …

184. 184 Floating IPs Floating IPs are required, if we want an Instance to be accessible from External Network Floating IP is simply a Public IP address Cloud Operator, creating External Provider Networks defines Public Address pools in their Subnets. Floating IP must be first allocated to a Project: $ openstack floating ip create < external_net > Cloud User can attach allocated Floating IP to an Instance $ openstack server add floating ip <instance> < floating_ip >

185. 185 Network Service Quotas Network related Quotas FloatingIP Network RBAC_policy Router Security_group Security_group_rule Subnet Subnetpool

186. 186 Network Interfaces in Instances New instance must have at least one Default Operating System configuration of first NIC depends on the Image Assumes DHCP configuration for Network Interface

187. 187 Network Interfaces in Instances Add a new vNIC to the Instance in Horizon Dashboard Add a new vNIC to the Instance on Command Line: $ nova interface-attach --net-id … Login to the Instance, edit / etc /network/interfaces to add eth1: auto eth1 iface eth1 inet dhcp

188. 188 Monitoring Network Service Basic Sanity Check of Network Service: # openstack network agent list Networks and IP Namespaces: # openstack network list # ip netns | grep dhcp Routers and IP Namespaces: # openstack router list # ip netns | grep router IP Availability: # openstack ip availability list

189. 189 Monitoring Network Service Neutron Log files located in /var/log/neutron neutron-server.log neutron-metadata-agent.log neutron-linuxbridge-agent.log neutron-linuxbridge-cleanup.log neutron-dhcp-agent.log neutron-l3-agent.log

190. IX. OpenStack Troubleshooting 190

191. Troubleshooting overview 191

192. Objectives Configuration Files OpenStack Services Log Files Central Database OpenStack Message Queues Quotas Verification Troubleshooting Server Instance Troubleshooting Network Connections 192 Troubleshooting Overview

193. 193 OpenStack Configuration Files OpenStack Infrastructure Configuration Files: NTP Time Server: / etc / chrony / chrony.conf MariaDB: / etc / mysql / mariadb.conf.d /99-openstack.cnf Memcached: / ect / memcached.conf Apache: / etc /apache2/apache2.conf

194. 194 OpenStack Configuration Files OpenStack Services Configuration Files: Keystone : / etc /keystone/ keystone.conf Glance : / etc /glance/glance- api.conf ; / etc /glance/glance- registry.conf Nova : / ect /nova/ nova.conf ; / ect /nova/nova- compute.conf Neutron : / etc /neutron/ neutron.conf Cinder : / ect /cinder/ cinder.conf Swift : / ect /swift/ swift.conf Horizon : /etc/openstack-dashboard/local-settings.py

195. 195 OpenStack System Service Most of OpenStack Services Server Software run as System Services To check Service Status: # systemctl status < service_name > To start System Service: # systemctl start < service_name >

196. 196 OpenStack System Service OpenStack Infrastructure NTP Time Server: chrony MariaDB: mysql RabbitMQ: rabbitmq -server Memcached: memcached Apache2: apache2

197. 197 OpenStack System Service OpenStack Services: Keystone: run as mod_wsgi in Apache2, no System Service Glance: glance- api , glance-registry Nova: nova- api , nova- consoleauth , nova-scheduler, nova-conductor, nova- novncproxy , nova-compute Neutron: neutron-server, neutron- linuxbridge -agent, neutron- dhcp -agent, neutron-metadata-agent, neutron-l3-agent Horizon: run as mod_wsgi in Apache2, no System Service Cinder: cinder-scheduler, cinder- api , cinder-volume Swift: rsync , swift-proxy, swift-account, swift-container

198. 198 OpenStack System Service Quick check of OpenStack Services: # openstack service list –long # openstack compute service list # neutron agent-list # cinder service-list

199. 199 OpenStack Log Files Most of OpenStack Infrastructure Services write Log files to /var/log: /var/log/apache2/access.log /var/log/apache2/error.log /var/log/ libvirt / qemu /instance-*.log /var/log/ mysql /error.log /var/log/ rabbitmq /rabbit.log

200. 200 OpenStack Log Files OpenStack Services write Log files to: Keystone: /var/log/apache2/keystone*.log Cinder: /var/log/cinder/cinder*.log Glance: /var/log/glance/glance*.log Neutron: /var/log/neutron/neutron*.log Nova: /var/log/nova/nova*.log Swift: /var/log/syslog

201. 201 The end!