OpenNebulaConf2019 - How We Use GOCA to Manage our OpenNebula Cloud - Jean-Philippe Fourès - Iguane Solutions

At Iguane Solutions, a lot of our "DevOps" tools are developed in Golang, and we have a good amount of experience in contributing to the Goca. I'll review just what contributions we make, as well as how we use Goca with different tools, on a daily basis, to manage and monitor our OpenNebula cloud.

I will delve into the concept of Infrastructure as Code - deployment of VM instances on cloud, as well as, also address the metrics collection of deployed VMs. Finally, I will present how we can abstract VM management with automation tools thanks to GOCA.

1. Iguane Solutions | Public
Iguane Solutions ©2019

2. Iguane Solutions
Iguane Solutions ©2019 - all rights reserved
2
Our contribution to “Goca”

Jean-Philippe Fourès

3. Iguane Solutions
Agenda
3
Iguane Solutions ©2019 - all rights reserved
●Iguane Solutions small introduction

●Goca contribution
○Goca introduction
○Details of our contribution
○Roadmap

●Example of applications using Goca

4. 1. Company Introduction
4
Iguane Solutions
Iguane Solutions ©2019 - all rights reserved

5. Iguane Solutions
5
•Network & CDN
•>200 racks in 3 DCs
•OpenNebula clusters
•Infrastructure Expert (Network, Hardware, Storage)
•“DevOps” culture: everything is automated
•Our tools are Open Source based:
CI/CD, Infra as code, metrology, Cloud
•1 OpenNebula cluster (2 very soon ☺ ) in Paris:
○10 KVM nodes (580 vCPU, 3.6 TB RAM)
○Storpool for performance Storage
•Cloud Hybridation specialist:
Public / Private or Physical / Virtual
•Network & CDN
•>5 racks
•Network & CDN
•>10 racks
•Network & CDN
•>5 racks
•Network & CDN
•>10 racks
1.1 Who are we ?
Founded in 2000, HQ in Paris, France
Iguane Solutions ©2019 - all rights reserved

6. 6
6
Iguane Solutions
●For 7 years, IG1 cloud has been using VMware but:
○Licence are expensive
○Troubleshooting VSAN is a nightmare (and it is a black box)
○VMware has lots of features => complexity



1.2 Iguane and One
Iguane Solutions ©2019 - all rights reserved
}
●In 2018, development of a new cloud service
○Simpler than VMware
○API friendly
○Easy to deploy, to extend, to manage, to maintain and to hack
around
○Able to manage: VMware clusters and Public clouds



=> End of 2018: deployment of our first cluster based on OpenNebula

In parallel to the deployment: Contribution to Goca and development tools using it

7. 7
Iguane Solutions
2. Goca
Iguane Solutions ©2019 - all rights reserved

8. 8
8
Iguane Solutions
2.1 Introduction to Goca
Iguane Solutions ©2019 - all rights reserved
OCA (OpenNebula Cloud API)
(ruby, java, python, golang)

9. 9
9
Iguane Solutions
2.1 Introduction to Goca
Iguane Solutions ©2019 - all rights reserved
What is Goca ?

Why Goca instead of Ruby or
Python API ?
Golang
Wrapper on top of XML RPC
API
Written in Golang
Go + OCA = GOCA
Add helpers to ease XML RPC
instructions
Python API not maintained
Not Ruby friendly:
Sorry Guys
Golang: growing language in
area of backend development
https://talks.golang.org/2014/gocon-tokyo.slide#16

10. 10
10
Iguane Solutions
2.2 Our contribution
Iguane Solutions ©2019 - all rights reserved
Rewrite XML mapping to parse and build
entities data

●Change the XML handling package and add
structures to map XML content

●Add dynamic management of template parts
Rework Architecture: adding a controller layer

●Better split of concerns, client and controller

●Modify the API to access the resources in a
controller centric way
Rework of the XML-RPC client

●Give the control of the client to the user:
○Client externalization,
○Possible to extend it

●Leverage the use of the initial XML-RPC:
○Manage only XML-RPC in XML-RPC package
○Manage HTTP in the client directly
Rewrite the error management

●Enable fine-grained management of errors with
types and codes

●Better error handling:
Don’t kill everyone on a package error

11. 11
11
Iguane Solutions
2.3 Goca Organization
Iguane Solutions ©2019 - all rights reserved
Goca subpackages
OpenNebula cluster
“errors”:
Error management package
“schema”:

Package containing all XML
structures defined in
different packages. Each
package matches an
OpenNebula object
“goca”:

It contains a file per
object family api call.
client.go manages
HTTP connection
controller.go
manages resources

12. 12
12
Iguane Solutions
2.4 Goca workflow
Iguane Solutions ©2019 - all rights reserved
How it works ?
OpenNebula
/RPC2
client
Model
VM
Controller
group
...
ask client to call
one.vm.info
with value ‘42’

Controller eases objects manipulation
and parsing.
An application only needs to “play”
with the controller to perform queries
to OpenNebula
Model, here is the client which
performs HTTP requests to
OpenNebula.
It builds XML requests (session, XML
formatting) and HTTP layer.
# application code to get
# info for VM 42

cli = goca.NewClient(...)
c =
goca.NewController(cli)

vm = c.VM(42).Info()


HTTP request to
OpenNebula XML
RPC endpoint
Goca

13. 13
13
Iguane Solutions
2.5 Roadmap Ideas
Iguane Solutions ©2019 - all rights reserved
Improve Tests - CI/CD

●Add more test cases to improve reliability and code
coverage on missing entities

●Rewrite existing tests according to Go coding style, and
consistently across entities

●Add a complete CI to test some scenarios with HTTP
request to a miniOne in a container
Increase the coverage of the XML-RPC API

●Highlights non-covered methods (missing tests case)

●Highlights unimplemented methods (dead code)

●Detect breaking changes, implement fallback methods

●Bug fixes
Increase the user friendliness of the code

●Make the code more readable, improve code comments

●Try to minimize boilerplate code on developer side.
For instance: type conversions

●Minimize the user need to read the documentation.
For instance: remove variable parameters handling
New functionalities

●Allow to filter entities on various criterias as template
elements for instance

●Contribute to the development of OpenNebula new
functionalities
Suggestions are welcomed

14. 3. Applications
Iguane Solutions
14
Iguane Solutions ©2019 - all rights reserved

15. 15
15
Iguane Solutions
3.1 Terraform Provider
Iguane Solutions ©2019 - all rights reserved
Important note:
Runtastic and Blackberry developed an OpenNebula provider using XML RPC calls directly before
goca merge into One repository.

●Software developed by Hashicorp:
Commonly used to deploy
Infrastructure on Public clouds thanks
to a large ecosystem of providers:
AWS, GCP, Azure, …

●Written in Golang, it offers APIs to
write custom providers easily


Thanks to the Goca, we initiated the
development of the OpenNebula Provider.
Tested with

●Terraform version: 0.12+

●OpenNebula 5.8
+
Official OpenNebula Add-on since June 2019!

16. 16
16
Iguane Solutions
3.1 Terraform Provider
Iguane Solutions ©2019 - all rights reserved
Group Image Virtual
Network
Security
Group
VDC VM
Template
VM
For more details, issues, roadmap, contribution are welcomed:
https://github.com/OpenNebula/terraform-provider-opennebula
October 2018 June 2019 September 2019
Major steps history
Internal beta release
Added as
OpenNebula Add-on
Submission to
Terraform
Contribution to Goca
Validation of first resources
Resources currently supported:
Official in Terraform
Hopefully:
December 2019

17. 17
17
Iguane Solutions
3.1 Terraform Provider
Iguane Solutions ©2019 - all rights reserved
Goca used subpackages
OpenNebula cluster
schema:

Terraform provider uses schema
packages linked to goca files and
permissions.go to manage object
permissions
quota.go to manage group quotas
lock.go to manage lock on images
“goca”:

Each terraform resources
matches with an
OpenNebula object

18. 18
18
Iguane Solutions
Sismograf:

●Metrology agent, developed
internally, deployed on each
monitored host or VM

●Metrics are then collected by our
Metrology tool: Sismology

●Several plugins available to
monitor system and common
application metrics:

Sismograf plugin for OpenNebula
has been developed to get usage
metrics of our OpenNebula
clusters


3.2 Sismograf Plugin
Iguane Solutions ©2019 - all rights reserved
OpenNebula Metrics into Iguane Solutions Metrology
OpenNebula Fronts
Sismograf agents to
collect OpenNebula
metrics
VMs
on OpenNebula
Sismograf agents to
collect VM metrics
Iguane Sismology:
Metrology system

19. 19
19
Iguane Solutions
3.2 Sismograf Plugin
Iguane Solutions ©2019 - all rights reserved
Goca used subpackages
OpenNebula cluster
“goca”:

It is used in the Sismograf
plugin dedicated to
OpenNebula.
It gets all metrics of our
cloud.
It mainly use Info API calls.
schema:

Sismograf OpenNebula plugin uses
schema packages linked to goca to get
metrics on the various objects
it also uses quota.go and snapshot.go
because we also have metrics on quotas
and size of a snapshot.

20. Questions ?
Iguane Solutions
20
Iguane Solutions ©2019 - all rights reserved

21. Iguane Solutions ©2019