Postgres on Kubernetes - Dos and Donts.pdf

Running databases in containers has been the biggest anti-pattern of the last decade. The world, however, moves on and stateful container workloads become more common, and so do databases in Kubernetes. People love the additional convenience when it comes to deployment, scalability, and operation.

With PostgreSQL on its way to become the world’s most beloved database, there certainly are quite some things to keep in mind when running it on k8s. Let us evaluate the important Dos and especially the Don’ts.

Presentation by Chris Engelbert of simplyblock (https://www.simplyblock.io)

1. https://www.marketingdonut.co.uk/pr-and-promotion/exhibitions/dos-and-don-ts-when-exhibiting

2. PostgreSQL Kubernetes❤

3. Chris Engelbert
Devrel @ simplyblock
Previous fun companies:
-Ubisoft / Blue Byte
-Hazelcast
-Instana
-clevabit
-Timescale
Interests:
-Developer Relations
-Anything Performance Engineering
-Backend Technologies
-Fairy Tales (AMD, Intel, Nvidia)
@noctarius2k
@[email protected]
@noctarius.com

6. Question 01

7. Question 01
Why you shouldn't run a
database in Kubernetes?

8. Why not to run a database in Kubernetes?

9. Why not to run a database in Kubernetes?

10. Why not to run a database in Kubernetes?
K8s is not designed with Databases in mind!

11. Why not to run a database in Kubernetes?
K8s is not designed with Databases in mind!
Never run Stateful Workloads in k8s!

12. Why not to run a database in Kubernetes?
K8s is not designed with Databases in mind!
Never run Stateful Workloads in k8s!
Persistent Data will kill you! Too slow!

13. Why not to run a database in Kubernetes?
K8s is not designed with Databases in mind!
Never run Stateful Workloads in k8s!
Persistent Data will kill you! Too slow!
Nobody understands Kubernetes!

14. Why not to run a database in Kubernetes?
K8s is not designed with Databases in mind!
Never run Stateful Workloads in k8s!
Persistent Data will kill you! Too slow!
Nobody understands Kubernetes!
What’s the benefit; databases don’t need autoscaling!

15. Why not to run a database in Kubernetes?
K8s is not designed with Databases in mind!
Never run Stateful Workloads in k8s!
Persistent Data will kill you! Too slow!
Nobody understands Kubernetes!
What’s the benefit; databases don’t need autoscaling!
Databases and applications should be separated!

16. Why not to run a database in Kubernetes?
K8s is not designed with Databases in mind!
Never run Stateful Workloads in k8s!
Persistent Data will kill you! Too slow!
Nobody understands Kubernetes!
What’s the benefit; databases don’t need autoscaling!
Databases and applications should be separated!
Not another layer of indirection / abstraction!

17. Why not to run a database in Kubernetes?
K8s is not designed with Databases in mind!
Never run Stateful Workloads in k8s!
Persistent Data will kill you! Too slow!
Nobody understands Kubernetes!
What’s the benefit; databases don’t need autoscaling!
Databases and applications should be separated!
Not another layer of indirection / abstraction!

18. Why not to run a database in Kubernetes?
BURN IN HELL!

19. The Happy Place

20. Where are my gamers at?
So we need to cheat!?

21. The Happy Place

22. Why?

23. No Cloud-Vendor Lock-In
Why?

24. No Cloud-Vendor Lock-In
Faster Time To Market
Why?

25. No Cloud-Vendor Lock-In
Faster Time To Market
Decreasing cost
Why?

26. No Cloud-Vendor Lock-In
Faster Time To Market
Decreasing cost
Automation
Why?

27. No Cloud-Vendor Lock-In
Faster Time To Market
Decreasing cost
Automation
Unified deployment architecture
Why?

28. No Cloud-Vendor Lock-In
Faster Time To Market
Decreasing cost
Automation
Unified deployment architecture
Need read-only replicas
Why?

29. Let’s get something
out of the way first!

30. Call the Police!

31. Enable TLS
Call the Police!

32. Enable TLS
Use Kubernetes Secrets
Call the Police!

33. Enable TLS
Use Kubernetes Secrets
Use Cert-Manager
Call the Police!

34. Enable TLS
Use Kubernetes Secrets
Use Cert-Manager
Encrypt Data-At-Rest
Call the Police!

36. Enable TLS
Use Kubernetes Secrets
Use Cert-Manager
Encrypt Data-At-Rest
Call the Police!

37. Backup and Recovery
https://www.ovhcloud.com/de/bare-metal/backup-storage/

38. You want Continuous Backup and PITR
Backup and Recovery
https://www.ovhcloud.com/de/bare-metal/backup-storage/

39. You want Continuous Backup and PITR
Roll your own pg_basebackup or pg_dump (don’t!)
Backup and Recovery
https://www.ovhcloud.com/de/bare-metal/backup-storage/

40. You want Continuous Backup and PITR
Roll your own pg_basebackup or pg_dump (don’t!)
Use tools like pgbackrest, barman, PGHoard, …
Backup and Recovery
https://www.ovhcloud.com/de/bare-metal/backup-storage/

41. You want Continuous Backup and PITR
Roll your own pg_basebackup or pg_dump (don’t!)
Use tools like pgbackrest, barman, PGHoard, …
Upload backups to S3? Cost!
Backup and Recovery
https://www.ovhcloud.com/de/bare-metal/backup-storage/

42. You want Continuous Backup and PITR
Roll your own pg_basebackup or pg_dump (don’t!)
Use tools like pgbackrest, barman, PGHoard, …
Upload backups to S3? Cost!
Backup and Recovery
https://www.ovhcloud.com/de/bare-metal/backup-storage/
" Test Your Backups "

43. PostgreSQL Configuration

44. PostgreSQL Configuration
The PostgreSQL Configuration isn’t too much influenced

45. shared_buffers
(maintenance_)work_mem
effective_cache_size
PostgreSQL Configuration
The PostgreSQL Configuration isn’t too much influenced

46. shared_buffers
(maintenance_)work_mem
effective_cache_size
PostgreSQL Configuration
The PostgreSQL Configuration isn’t too much influenced
Use Huge Pages!

47. PostgreSQL Configuration
https://www.youtube.com/watch?v=S0LEDGbAnn8
https://www.crunchydata.com/blog/optimize-postgresql-server-performance
https://www.percona.com/blog/using-huge-pages-with-postgresql-running-inside-kubernetes/

48. Do you need more? Extensions!

49. Do you need PG Extensions?
Do you need more? Extensions!

50. Do you need PG Extensions?
Do you need more? Extensions!
Is the extension part of the container image?

51. Do you need PG Extensions?
Do you need more? Extensions!
Is the extension part of the container image?
If not, you need to build your own layer…

52. Do you need PG Extensions?
Do you need more? Extensions!
Is the extension part of the container image?
If not, you need to build your own layer…
or use some magic (more on this later).

53. Keep an Eye on PG and Kubernetes Versions
Versions and Updates

54. So What is important or different?

55. Storage

56. Storage

57. Use Persistent Volumes
Storage

58. Use Persistent Volumes
Storage
(local volumes are a bad idea)

59. Use Persistent Volumes
Storage
Should be dynamically provisioned
(local volumes are a bad idea)

60. Use Persistent Volumes
Storage
Should be dynamically provisioned
CSI provider enables encryption at rest
(local volumes are a bad idea)

61. Use Persistent Volumes
Storage
Should be dynamically provisioned
CSI provider enables encryption at rest
High IOPS (SSD or NVMe)
(local volumes are a bad idea)

62. Use Persistent Volumes
Storage
Should be dynamically provisioned
CSI provider enables encryption at rest
High IOPS (SSD or NVMe)
Low Latency
(local volumes are a bad idea)

63. Use Persistent Volumes
Storage
Should be dynamically provisioned
CSI provider enables encryption at rest
High IOPS (SSD or NVMe)
Low Latency
Database performance is as fast as your storage
(local volumes are a bad idea)

64. Use Persistent Volumes
Storage
Should be dynamically provisioned
CSI provider enables encryption at rest
High IOPS (SSD or NVMe)
Low Latency
Database performance is as fast as your storage
(local volumes are a bad idea)
I’d recommend a disaggregated storage!

65. Storage
www.storageclass.info/csidrivers

66. Requests, Limits, and Quotas
Capacity
Limits
Requests
Used

67. Requests, Limits, and Quotas
Capacity
Limits
Requests
Used
Use Resource Requests, Limits, Quotas

68. Requests, Limits, and Quotas
CPU and memory requests need to be accurate

to prevent contention and ensure predictable performance
Capacity
Limits
Requests
Used
Use Resource Requests, Limits, Quotas

69. Requests, Limits, and Quotas
CPU and memory requests need to be accurate

to prevent contention and ensure predictable performance
Capacity
Limits
Requests
Used
https://codimite.ai/blog/kubernetes-resources-and-scaling-a-beginners-guide/
Use Resource Requests, Limits, Quotas

70. Make it big!
Enable Huge Pages!

71. Make it big!
Enable Huge Pages!
In your OS and the Resource Descriptor.

72. Make it big!
Enable Huge Pages!
In your OS and the Resource Descriptor.
https://www.percona.com/blog/using-huge-pages-with-postgresql-running-inside-kubernetes/

73. Resiliency and Overhead

74. Resiliency and Overhead
High Availability

75. Patroni, repmgr, pg_auto_failover, …
Resiliency and Overhead
High Availability

76. Patroni, repmgr, pg_auto_failover, …
Resiliency and Overhead
High Availability
https://medium.com/@kristi.anderson/whats-the-best-postgresql-high-availability-framework...

77. Resiliency and Overhead

78. Resiliency and Overhead
Connection Pooling

79. Never use PostgreSQL without Connection Pooling!
Resiliency and Overhead
Connection Pooling

80. Never use PostgreSQL without Connection Pooling!
Optimizes Overhead and Resource Utilization
Resiliency and Overhead
Connection Pooling

81. Never use PostgreSQL without Connection Pooling!
Optimizes Overhead and Resource Utilization
Handles failovers, central switching of Primary
Resiliency and Overhead
Connection Pooling

82. Never use PostgreSQL without Connection Pooling!
Optimizes Overhead and Resource Utilization
Handles failovers, central switching of Primary
Enables easy use of Read-Replicas
Resiliency and Overhead
Connection Pooling

83. Never use PostgreSQL without Connection Pooling!
Optimizes Overhead and Resource Utilization
Handles failovers, central switching of Primary
Enables easy use of Read-Replicas
Resiliency and Overhead
Connection Pooling
PgBouncer, PgPool-II, pgagroal, PgCat, Odyssey, …

84. Never use PostgreSQL without Connection Pooling!
Optimizes Overhead and Resource Utilization
Handles failovers, central switching of Primary
Enables easy use of Read-Replicas
Resiliency and Overhead
Connection Pooling
PgBouncer, PgPool-II, pgagroal, PgCat, Odyssey, …
https://tembo.io/blog/postgres-connection-poolers

85. Where’s my Replicant?

86. Where’s my Replicant?
Use available Kubernetes features

87. Where’s my Replicant?
Use available Kubernetes features
StatefulSet

88. Where’s my Replicant?
Use available Kubernetes features
StatefulSet

89. Networking and Access Control
https://timeclock365.com/tc22-door-access-controller/

90. Use Network Policies
Networking and Access Control
https://timeclock365.com/tc22-door-access-controller/

91. Use Network Policies
Enable TLS (you remember?!)
Networking and Access Control
https://timeclock365.com/tc22-door-access-controller/

92. Use Network Policies
Enable TLS (you remember?!)
Setup Security Policies
Networking and Access Control
https://timeclock365.com/tc22-door-access-controller/

93. Use Network Policies
Enable TLS (you remember?!)
Setup Security Policies
Configure RBAC (Role-Based Access Control)
Networking and Access Control
https://timeclock365.com/tc22-door-access-controller/

94. Use Network Policies
Enable TLS (you remember?!)
Setup Security Policies
Configure RBAC (Role-Based Access Control)
Networking and Access Control
Think about a policy manager such as OPA or kyverno
https://timeclock365.com/tc22-door-access-controller/

95. Observability and Alerting

96. Observability and Alerting
Like anything cloud, make sure you have
monitoring (meaning observability) and alerting!

97. Prometheus Exporter, Log Collector, Aggregation, Analysis, Traceability, …
Observability and Alerting
Like anything cloud, make sure you have
monitoring (meaning observability) and alerting!

98. Prometheus Exporter, Log Collector, Aggregation, Analysis, Traceability, …
Observability and Alerting
Like anything cloud, make sure you have
monitoring (meaning observability) and alerting!
Datadog, Instana, DynaTrace, Grafana, …

99. Operator

100. Operator
Use a Postgres Kubernetes Operator

101. Operator
Use a Postgres Kubernetes Operator
Handles or configures many of the typical tasks (HA, backup, …)

102. Operator
Use a Postgres Kubernetes Operator
Handles or configures many of the typical tasks (HA, backup, …)
Brings cloud-nativeness to PG

103. Operator
Use a Postgres Kubernetes Operator
Handles or configures many of the typical tasks (HA, backup, …)
Brings cloud-nativeness to PG
Integrates PG into k8s

104. Operator
If not, use Helm Charts
Use a Postgres Kubernetes Operator
Handles or configures many of the typical tasks (HA, backup, …)
Brings cloud-nativeness to PG
Integrates PG into k8s

105. Operator
CloudNativePG
Crunchy Postgres
for Kubernetes
OnGres
StackGres
KubeDB
Zalando Postgres
Operator
Supported
versions
12, 13, 14, 15, 1611, 12, 13, 14, 15, 1612, 13, 14, 15, 169.6, 10, 11, 12, 13, 1411, 12, 13, 14, 15, 16
Postgres
Clusters
✔ ✔ ✔ ✔ ✔
Streaming
replication
✔ ✔ ✔ ✔ ✔
Supports
Extensions
✔ ✔ ✔ ✔ ✔

106. Operator
CloudNativePG
Crunchy Postgres
for Kubernetes
OnGres
StackGres
KubeDB
Zalando Postgres
Operator
Hot Standby ✔ ✔ ✔ ✔ ✔
Warm Standby ✔ ✔ ✔ ✔ ✔
Automatic Failover ✔ ✔ ✔ ✔ ✔
Continuous
Archiving
✔ ✔ ✔ ✔ ✔
Restore from

WAL archive
✔ ✔ ✔ ✔ ✔
Supports PITR ✔ ✔ ✔ ✔ ✔
Manual backups ✔ ✔ ✔ ✔ ✔
Scheduled backups ✔ ✔ ✔ ✔ ✔

107. Operator
CloudNativePG
Crunchy Postgres
for Kubernetes
OnGres
StackGres
KubeDB
Zalando Postgres
Operator
Backups via
Kubernetes
✔ ✘ ✔ ✔ ✘
Custom
resources
✔ ✔ ✔ ✔ ✔
Uses default
PG images
✘ ✔ ✔ ✘ ✘
CLI access ✔ ✔ ✔ ✔ ✘
WebUI ✘ ✘ ✔ ✔ ✘
Tolerations ✔ ✔ ✔ ✔ ✔
Node affinity ✔ ✔ ✔ ✔ ✔

108. Operator
https://www.simplyblock.io/post/choosing-a-postgres-kubernetes-operator
https://operatorhub.io/?keyword=postgres

109. Pinning and Tainting

110. Always use specific, dedicated machines for your database.
Pinning and Tainting

111. Always use specific, dedicated machines for your database.
Pinning and Tainting
(except you’re running super small databases)

112. Always use specific, dedicated machines for your database.
Pin your database containers to those hosts.
Pinning and Tainting
(except you’re running super small databases)

113. Always use specific, dedicated machines for your database.
Pin your database containers to those hosts.
Taint the hosts to prevent anything else from running on it.
Pinning and Tainting
(except you’re running super small databases)

114. Always use specific, dedicated machines for your database.
Pin your database containers to those hosts.
Taint the hosts to prevent anything else from running on it.
Pinning and Tainting
(except you’re running super small databases)
(except the minimum necessary Kubernetes services, like KubeProxy)

115. Trust me, I’m Kelsey!
https://x.com/kelseyhightower/status/1624081136073994240

116. Trust me, I’m Kelsey!
https://x.com/kelseyhightower/status/1624081136073994240

117. More Resources
Data on Kubernetes Community: https://dok.community
Data on Kubernetes Whitepaper

118. Thank you very much!
Questions?
@noctarius2k
@[email protected]
@noctarius.com