Enforcing API Standards for Kubernetes CRDs at Scale using OpenAPI, Arazzo and Overlay

In modern engineering organizations, APIs evolve faster than the processes around them. Specs get copied instead of reused, workflows are buried in client logic, and documentation rarely reflects reality. This session tackles a common but growing challenge: how to manage the multiple Kubernetes operators with various Custom Resources (CRs),  the solution would be "Enforcing API Standards for Kubernetes CRDs at Scale" 

1. Mitul Sharma Enforcing API Standards for Kubernetes CRDs at Scale Using OpenAPI, Overlay and Arrazo Senior Software Engineer @ Red Hat India Bangalore Oct 8-9

2. Our Kubernetes Challenge ? Multiple Operators, Multiple Problems Memcached CacheCluster WebApp Team C Team B Team A

3. Our Kubernetes Challenge ? Multiple Operators, Multiple Problems Platform runs multiple Kubernetes operators Different teams author CRDs Frequent drift in structure and semantics

4. When Things Go Wrong Consequences: ❌ Schema mismatches ❌ Deployment failures ❌ Production downtime ❌ Developer frustration

5. Solution Unified, Automated API Governance T hree pillars of our solution: OpenAPI - Consistent validation standards Arazzo  - Declarative workflow orchestration Overlay - Deterministic compliance enforcement

6. OpenAPI Specification v3.1.1: Foundation Standard specification for describing RESTful APIs in a structured and machine-readable format (usually YAML or JSON) Defines API endpoints, request/response structures, authentication methods in a consistent document Usecase for automatic documentation, client generation and API validation

7. OpenAPI 3.2 : What’s new ? New Features: I ntroduces the new ‘query’ method and ‘additionalOperations’ for defining custom, non-standard HTTP verbs, increasing flexibility in API design. Supports OAuth 2.0 Device Flow and adds ‘oauth2metadataurl’ for easier integration with identity providers Enriched ‘tag’ objects with hierarchy and categorization fields, plus improved discriminator defaults for better model management

8. CRDs to OpenAPI controller-gen crd2openapi openapi-gen

9. Validation spec.versions[].schema.openAPIV3Schema Single source of truth

10. OpenAPI Documentation Generated schemas are always in sync with code Consistent across all teams and all CRDs

11. From Go Types to OpenAPI Schemas =>

12. Key benefits of crd to openAPI spec Detect the changes when size int32 changes, conversion reflects in spec Generated OpenAPI will show different field definitions Basic type information and validation rules are maintained

13. Arazzo 1.0.1 Arazzo is a specification for describing sequences of API calls It describes an approach to document use-case oriented workflows in a programmatically readable format Designed to work alongside with the openAPI Specification Additional human readable nature improves the ability of API specification to tell the story of API(s) in a manner to improve the dev experience. 

14. Arazzo Specification Architecture Caption

15. Arazzo Specification Architecture Caption Info contains the metadata about the defined arazzo document

16. Arazzo Specification Architecture Caption Info contains the metadata about the defined arazzo document

17. Arazzo Specification Architecture Caption Info contains the metadata about the defined arazzo document Lists source descriptions that can be referenced by one or more Workflows(eg. openAPI description)

18. Arazzo Specification Architecture Caption Info contains the metadata about the defined arazzo document Lists source descriptions that can be referenced by one or more Workflows

19. Arazzo Specification Architecture Caption Info contains the metadata about the defined arazzo document Lists source descriptions that can be referenced by one or more Workflows Describes the workflow across one or more APIs

20. Arazzo Specification Architecture Caption Input is a json schema objects respresents input used by workflow

21. Arazzo Specification Architecture Caption Step represents an call to an API operation

22. Arazzo Specification Architecture Caption Output is a map between a friendly name and a dynamic output value for a workflow

23. Arazzo Specification Architecture Reusable objects

24. Arazzo Specification Architecture Success/Failure evaluation rules

25. Arazzo Specification Architecture

26. Arazzo Driven CI Pipeline + flow diagram Caption

27. Arazzo 1.0.1: Declarative API Workflows Multi-step API workflow specification Validation and linting Declarative, not imperative Enables consistent integration patterns Perfect for CI/CD orchestration *Spectral won't lint the document if no ruleset is specified and no default ruleset file is found

28. Arazzo 1.0.1: Declarative API Workflows Key Benefits: Reproducible workflows Version-controlled processes Cross-team consistency Automated validation pipelines Code generation and document tool for communication

29. Why Arazzo Works for CRDs

30. Overlay 1.0 Overlays are the modification specifications that transforms base OpenAPI specification for different environments, use cases, or compliance. Instead of original openAPI yaml, it creates changes in separate files Fix schema inconsistencies across environments Enhance security (authentication, rate limiting) Customize APIs (SDKs) for different environments (dev, stage, prod)

31. Architecture Base OpenAPI Spec(openapi.yaml) Overlay files(prod,dev,stage) Speakeasy for modify/extend/sdk generation BumpCli version management Generate SDK for different environments

32. Overlay 1.0 Use Cases: Enforce minimum validation rules Add security requirements Standardize error responses Apply organizational policies

33. Enforcing Size Limits Across All CRDs / Env specific

34. Enforcing Size Limits Across All CRDs

35. Complex Compliance Scenarios with CI/CD integration

36. Advantages and Feedback Reduction in validation errors with faster deployment cycles Teams can focus on business logic, not validation boilerplate Onboarding new developers is much smoother 100% compliance with security policies Consistent schemas across 25+ CRDs

37. OpenAPI + Arazzo + Overlay = Success Dev commit CRD changes Arazzo workflow triggers Generate OpenAPI Schemas Deploy or reject Validation against cluster Apply Overlay patches

38. Implementation Foundation Rollout Scal scale

39. Phase 1: Foundation Audit existing CRDs Define baseline Overlay rules Create Arazzo validation workflow Phase 2: Rollout Implement in one repository Train first team Refine based on feedback Phase 3: Scale Roll out to all teams Add advanced Overlay rules Integrate with existing CI/CD

40. What We Wish We Knew Earlier Technical implementation is the easy part Getting teams to adopt new processes is harder Focus on making their lives easier, not harder Show value quickly and iterate based on feedback "The technology is only 30% of the solution. The other 70% is change management and team adoption."

41. Q&A 

42. Github LinkedIn