ScyllaDB Kubernetes Operator Goes Global

ScyllaDB Operator is a Kubernetes Operator for managing and automating tasks related to managing ScyllaDB clusters. In this talk, you will learn the basics about ScyllaDB Operator and its features, including the new manual MultiDC support.

1. What’s new in Scylla Operator Tomáš Nožička Principal Software Engineer at ScyllaDB

2. Biography Tomáš Nožička Tomáš is a principal software engineer leading the Scylla Operator team. Prior to joining ScyllaDB, he worked at Red Hat on a self-hosted, auto-upgrading Kubernetes control plane for OpenShift. He’s an emeritus Kubernetes Sig-Apps approver.

3. Scylla Operator Scylla Operator is a Kubernetes Operator for managing and automating tasks related to managing ScyllaDB clusters. Kubernetes API extensions + controllers that reconcile the desired state Open source ( https://github.com/scylladb/scylla-operator ) https://operator.docs.scylladb.com

4. Releases Aiming for ~6 weeks cadence (modulo PTOs, holidays, ..) Supporting 2 latest releases N-1 compatibility 1.10 -> 1.11 -> 1.12 -> … Release GA EOL 1.11 2023-11-09 Release of 1.13 1.10 2023-08-25 Release of 1.12 1.9 2023-07-04 2023-11-09 1.8 2023-01-25 2023-08-25

5. Differences VMs Separate VM per tenant ScyllaDB runs directly on host 1 ScyllaDB per VM Dedicated resources Direct access to kernel, devices, other processes running on host User need to manually execute maintenance and upgrade tasks Kubernetes Multi-tenancy Everything runs in containers Multiple ScyllaDB’s on a Kubernetes node Shared pool of resources (QoS classes) Virtualized devices, separate namespaces Management and upgrade task are handled by the operator seamlessly Declarative API Operator reports status

6. How it works r ead /write apiVersion: scylla.scylladb.com/v1 kind : ScyllaCluster spec : version : 5.4.0 agentVersion : 3.2.5 datacenter : name : us-east- 1 racks : - name : us-east-1a members : 3 storage : capacity : 200Gi resources : limits : cpu : 8 memory : 128Gi status : racks : us-east-1 : readyMembers : 3 Controller / Operator Observe Calculate Reconcile

7. Storage Primarily oriented on local storage (NVMes) Best performance / latency Complex setup (gradularily automating the setup to reduce the complexity) Works with network storage as well

8. Local Storage Options github.com/scylladb/k8s-local-volume-provisioner Kubernetes CSI plugin Dynamic PersistentVolumes Directory based, uses xfs prjquota Throughput is limited with ScyllaDB io properties (limited multitenancy) github.com/kubernetes-sigs/sig-storage-local-static-provisioner Static (fixed PVs count, each pre-created by administrator) Block devices or directories without any quota

9. Available APIs ScyllaCluster.scylla.scylladb.com/v1 Creates ScyllaDB clusters ScyllaOperatorConfig.scylla.scylladb.com/v1alpha1 Configures the Scylla Operator NodeConfig.scylla.scylladb.com/v1alpha1 Configures tuning and node options ScyllaDBMonitoring.scylla.scylladb.com/v1alpha1 Sets up a monitoring stack and connects it to ScyllaClusters

10. Discovering Available APIs API Reference (operator.docs.scylladb.com) $ kubectl api-resources --api-group='scylla.scylladb.com' $ kubectl explain --api-version='scylla.scylladb.com/v1alpha1' NodeConfig.spec

11. Features

12. Multitenancy

13. Scaling

14. Multi-AZ Support

15. Rolling Config Changes

16. Rolling Upgrades

17. Auto Healing

18. Available since v1.11.0 IP source can be configured using `scyllacluster.spec.exposeOptions` Supported types ServiceClusterIP (default, existing clusters - ClusterIPs are virtual / reachable only in the same k8s cluster) ServiceLoadBalancerIngress PodIP Given ScyllaDB publishes `broadcast_rpc_address` to clients, this is essential for exposing ScyllaClusters on “external” IPs. (Load balancer on its own wouldn’t work.) Configuring IP Address Source for ScyllaClusters

19. apiVersion : scylla.scylladb.com/v1 kind : ScyllaCluster spec : exposeOptions : nodeService : type : ClusterIP broadcastOptions : clients : type : ServiceClusterIP nodes : type : ServiceClusterIP Broadcast Options - In-cluster Only

20. apiVersion : scylla.scylladb.com/v1 kind : ScyllaCluster spec : exposeOptions : nodeService : type : ClusterIP broadcastOptions : clients : type : PodIP nodes : type : ServiceClusterIP Broadcast Options - Clients over VPC

21. apiVersion : scylla.scylladb.com/v1 kind : ScyllaCluster spec : exposeOptions : nodeService : type : LoadBalancer broadcastOptions : clients : type : ServiceLoadBalancerIngress nodes : type : ClusterIP Broadcast Options - Internet

22. apiVersion : scylla.scylladb.com/v1 kind : ScyllaCluster spec : exposeOptions : nodeService : type : Headless broadcastOptions : clients : type : PodIP nodes : type : PodIP Broadcast Options - Multi VPC / Region

23. Available since v1.11.0 ScyllaCluster has new API field `. spec.externalSeeds ` that allows configuring seeds from other datacenters In most cases, Kubernetes cluster represents a region / datacenter Common MultiDC setup runs on multiple k8s clusters with Independent Scylla Operators in each k8s cluster Independent ScyllaClusters in different k8s clusters Connected through external (public or private) IPs and/or DNS using `.spec.externalSeeds` Manual MultiDC Support

24. Manual MultiDC - Networking Example

25. Global ScyllaDB Manager deployment Scylla Operator runs a small manager agent alongside each ScyllaDB node configures global ScyllaDB Manager with access to all ScyllaClusters configures ScyllaDB Manager with repair and backup tasks defined in ScyllaCluster CR ScyllaDB Manager Integration

26. ScyllaDBMonitoring CustomResource manages monitoring stack deployment and life cycle management Prometheus instance + ScyllaDB scraping rules Grafana instance + ScyllaDB dashboards Service discovery and authentication Managed ScyllaDB Monitoring

27. apiVersion : scylla.scylladb.com/v1alpha1 kind : ScyllaDBMonitoring metadata : name : example spec : type : Platform endpointsSelector : matchLabels : app.kubernetes.io/name : scylla scylla-operator.scylladb.com/scylla-service-type : member scylla/cluster: replace-with-your-scyllacluster-name components : prometheus : storage : … grafana : exposeOptions : webInterface : ingress : dnsDomains : [ test-grafana.test.svc.cluster.local ] ScyllaDBMonitoring Example

28. Creating RAID arrays Formatting disks with a filesystem (xfs) Mounting devices (disks / RAID arrays) Device tuning NodeConfig

29. apiVersion : scylla.scylladb.com/v1alpha1 kind : NodeConfig metadata : name : cluster spec : localDiskSetup : raids : - name : nvmes type : RAID0 RAID0 : devices : nameRegex : ^/dev/nvme\d+n\d+$ filesystems : - device : /dev/md/nvmes type : xfs mounts : - device : /dev/md/nvmes mountPoint : /mnt/persistent-volumes unsupportedOptions : - prjquota NodeConfig example placement: nodeSelector: scylla.scylladb.com/node-type: scylla tolerations: - effect: NoSchedule key: role operator: Equal value: scylla-clusters

30. Operator automatically creates and manages TLS certificates to setup CQL over TLS Serving certificates secret/<sc-name>- local-serving-ca secret/<sc-name>-local-serving-certs configmap/<sc-name>-local-serving-ca Client CA secret/<sc-name>-local-client-ca configmap/<sc-name>-local-client-ca Admin user credentials (signed by client CA) secret/<sc-name>-local-user-admin Automatic TLS certificates for CQL

31. We needs dozens of YAML manifest + logs to figure out most of the issues This used to take a lot of back and forth to figure out an issue Must-gather is a new tool that collects all necessary manifests and logs that relate to our APIs into one folder and redacts Secrets Users can always review the data before uploading them. Must-gather is now mandatory for reporting issues https://operator.docs.scylladb.com/master/support/must-gather.html Must-gather

32. Stay in Touch Tomáš Nožička [email protected] https://twitter.com/TomasNozicka https://github.com/tnozicka https://www.linkedin.com/in/tnozicka/