stackconf 2024 | Insights into Managed Service Provision A STACKIT Retrospective by Patrick Koss.pdf

Embark on the innovative journey of STACKIT, a premier European cloud provider, as we showcase our expertise in Kubernetes-based managed services. This talk will delve into the dynamic processes of deploying and managing robust services like PostgreSQL, InfluxDB and MongoDB on Kubernetes clusters, a testament to our technological prowess. We’ll unravel our multifaceted strategies for processing customer requests, from the initial API call to the final deployment stage. Our discussion will highlight diverse methodologies, including the integration of databases with message queues and the direct creation of Kubernetes resources, offering insights into their unique efficiencies and challenges. Join us to deeply understand the trade-offs of each approach. We’ll address vital issues such as scaling capabilities, backup strategies and effective resource management.

1. INSIGHTS INTO MANAGED
SERVICE PROVISION: A
STACKITRETROSPECTIVE

2. WHO AM I?
-Senior Software Engineer / Architect
-Building PaaS in the Cloud
-Created an Observability Service, DNS and now an AI Platform
-Giving Lectures about Databases
-Like to write blog posts about programming, system design and
architecture

3. SOCIALS
https://www.linkedin.com/in/patrick-koss-a129071a1/
https://medium.com/@patrickkoss

4. PROVIDERS EVERYWHERE

5. WHAT IS A MANAGED SERVICE?
Updates & Upgrades Scaling Up & Down High Available
Backups & Restore Monitoring & Status & OnCall Operational Knowledge

6. MANAGED SERVICE EXAMPLES

7. WHAT FUNCTIONALITY NEED WE PROVIDE?
Instance provision, update, deprovision
Backup, Backupschedule CRUD
Restore
ACL CRUD
Config Management

8. THE CHALLENGE
Client (TF, UI) Restful API
Instance (DB,
LLM)
Compute
Service

9. COMPUTE
Virtual Machines Kubernetes
Comparison Criteria:
- Scaling
- Management
- Updates/Upgrades
- Zero Downtime -
Deployments
- Spin Up Time
- Complexity
- Team experience
- Resource Management
of Instances(GB/CPU)
- Environment

10. THE CHALLENGE REVISED
Client (TF, UI) Restful API
Instance (DB,
LLM)
Kubernetes Cluster
Service
Instance (DB,
LLM)
Kubernetes Cluster
…

11. INITIALIZE A KUBERNETES CLUSTER
Cert-Manager
External-DNS
External-Secrets
Prometheus
OTEL
Git Repository
watch
apply

12. KUBERNETES OPERATORS
apiVersion: acid.zalan.do/v1
Kind: Postgresql
Kube API
Postgres
Operator
Watch
postgresql
StatefulSet Service
ownsowns
- Single Leader System (cached client, in-memory
adjustable queue)
- Owns: Get notified if changes occur
- Bundle operative knowledge
- Bundle functionality -> testing
- Idempotency, state transition built into the philosophy

13. DEPLOY AN INSTANCE (POSTGRES, GRAFANA, INFLUX)
apiVersion: v1
Kind: Ingress
---
apiVersion: v1
Kind: Service
---
apiVersion: acid.zalan.do/v1
Kind: Postgresql
yamls
N Kubernetes Cluster
???
Challenges:
-Parameterize yamls
-Get yamls in n Kubernetes Cluster
-Propagate the status of the
instance to the customer

14. DEPLOY AN INSTANCE (POSTGRES, GRAFANA, INFLUX)
Client (TF, UI) Restful API
Datastore
Name, Description,
Version, ACL
Store metadata
Message Queue
Publish Message
Consumer
Consume
Kubernetes
Cluster
Generate yamls and apply

15. DEPLOY AN INSTANCE (POSTGRES, GRAFANA, INFLUX)
Client (TF, UI) Restful API
Datastore
Name, Description,
Version, ACL
Store metadata
Outbox Worker
Message Queue
Take out
outbox
Publish
Message
Consumer
Consume
Kubernetes
Cluster
Generate yamls and apply
Cluster
N Cluster

16. DEPLOY AN INSTANCE (POSTGRES, GRAFANA, INFLUX)
Client (TF, UI) Restful API
Datastore
Name, Description,
Version, ACL
Store metadata
Outbox Worker
Message Queue
Take out
outbox
Publish
Agent (Message
Consumer)
Consume
Kubernetes
Cluster
Generate yamls and apply
Cluster
N Cluster

17. DEPLOY AN INSTANCE (POSTGRES, GRAFANA, INFLUX)
Client (TF, UI) Restful API
Datastore
Name, Description,
Version, ACL
Store metadata
Outbox Worker
Message Queue
Take out
outbox
Publish
Message
Consumer
Consume
Git Repo
Generate yamls and push
Kubernetes
Cluster
pull
apply
Cluster
N Cluster

18. DEPLOY AN INSTANCE (POSTGRES, GRAFANA, INFLUX)
Client (TF, UI) Restful API
Name, Description,
Version, ACL
Agent(let)
(Operator)
Kubernetes
Cluster
Generate yamls and apply
Cluster N Cluster
Instance CRD
Store
Metadata
Watch

19. STATUS PROPAGATION
Client (TF, UI) Restful API
Datastore
Is my instance
Ready / updated?
Desired State = Current State -> healthy
Desired State != Current State -> reconciling
Error in State -> Error
Desired ACL 1.2.3.4 != Current ACL 0.0.0.0/0 ->
reconciling
Desired healthy pods 1 > Current healthy pods 3 ->
healthy

20. STATUS PROPAGATION
Client (TF, UI) Restful API
Datastore
Is my instance
Ready / updated?
Cluster
Status Check
Worker
Get last check timestamp
Check Status Field
Update status

21. STATUS PROPAGATION
Client (TF, UI) Restful API
Datastore
Is my instance
Ready / updated?
Message Queue
Status Check
Worker
Consumer status message
Update status
Agent(let)
(Operator)
Cluster
Query / Watch
resources
Publish status
change message

22. STATUS PROPAGATION
Client (TF, UI) Restful API
Is my instance
Ready / updated?
Update status
Agent(let)
(Operator)
Cluster
Watch
resources
Instance CRD
Get CRD and calculate status

23. OTHER MANAGEMENT RESOURCES (BACKUPS, ACLS)
Client (TF, UI) Restful API
1. Create Backup
3. Reconcile
Agent
Instance
(Postgres)
4. Pg_dump
Backup CRD
2. Create Backup CRD
Object Storage
5. Upload Backup
6. Update Status
CreatedAt
8. Get Status7. Get Backups
(Status)

24. TAKE AWAYS
-All approaches can be valid depending on the use case
-Aiven uses a DB+Queue and host on VMs
-Measure / Monitor potential bottlenecks
-Kubernetes is a powerful platform and operators are great to manage components
-Be careful to not overload the kube api
-Commit to one approach and use it in the entire system

25. THAT´S IT FOLKS!