[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf

Kief Morris rethinks the infrastructure code delivery lifecycle, advocating for a shift towards composable infrastructure systems. We should shift to designing around deployable components rather than code modules, use more useful levels of abstraction, and drive design and deployment from applications rather than bottom-up, monolithic architecture and delivery.

1. Rethinking the infrastructure
delivery lifecycle
Moving beyond
spaghetti infrastructure Kief Morris
Global Infrastructure Practice Lead
July, 2024

2. My journey
CFEngine
FAI
2000
Puppet
Chef
VMWare
DevOps
CD
Cloud
1st
Edition
2nd
Edition
2010 2020
Various digital startups and scaleups
Learning and
adopting
Thoughtworks consultant
Helping enterprises
with adoption
Thoughtworks global role
3rd
Edition
Exploring and
sharing what's next

3. Our
infrastructure
codebases are
spaghetti

4. Problems with
infrastructure
●Too much custom work
●Blocker rather than enabler
●Costs not well-aligned
●Lack of confidence
●Legacy accumulates

5. Goals
●Share more infrastructure code
●Empower teams
●Build governance in
●Lower the cost of scaling
●Continuously reduce legacy

6. How?

7. Rethink infrastructure code architecture
Rethink infrastructure code delivery
Rethink infrastructure code deployment

8. Rethinking
infrastructure code
architecture

9. Beyond monolithic
infrastructure
deployments

10. What is an infrastructure monolith?
Deploy
Infrastructure
project
and is deployed
as a single unit
that defines
infrastructure for
different workloads

11. What is an infrastructure monolith?
Deploy
Infrastructure
project
and is deployed
as a single unit
that defines
infrastructure for
different workloads
Low cohesion
High coupling

12. What is an infrastructure monolith?
Deploy
Infrastructure
project code
and is deployed
as a single unit
that defines
infrastructure for
different workloads
Low cohesion
High coupling
Barriers to change
Barriers to quality

13. Are code libraries the solution?
Deploy
Infrastructure
project code
Infrastructure
code libraries

14. Are code libraries the solution?
Deploy
Infrastructure
project code
Infrastructure
code libraries
Libraries are useful
for organizing and
sharing code
But they don't
reduce the size or
risk of deployments

15. From monolith to composable deployments


Deploy

16. From monolith to composable deployments
Separately
deployable
infrastructure
components
Separately
developed,
tested, and
delivered

17. We need more
helpful
abstractions

18. About
abstractions
●Abstractions that hide stuff are
unhelpful
●Abstractions that disempower
people are unhelpful
●Abstractions that support
cognitive focus are helpful

19. Current
infrastructure
abstractions are
not helpful enough
●Infrastructure code is super low
level
●Great for systems administrators,
who think at this level
●Not great for application owners,
who don't think at this level

20. Infrastructure
code abstractions
Workload
IaaS Resources

21. Stacks:
Deployable
components
Workload
IaaS Resources
Infrastructure stacks

22. Infrastructure
Products:
Consumable
components
Workload
IaaS Resources
Infrastructure stacks
Infrastructure products

23. Rethinking
infrastructure code
delivery

24. Common issues
●Environments are difficult and
expensive to create and change
●Environments are inconsistent
and outdated
●Not enough environments
available
●Environments are
over-provisioned and
under-utilized

25. ENVIRONMENT ENVIRONMENT ENVIRONMENT
Infrastructure delivery lifecycle
Develop Deploy
Use

26. Recoupling the
infrastructure
delivery lifecycle

27. ENVIRONMENT ENVIRONMENT ENVIRONMENT
Bespoke environments as code
Develop Deploy
Use
Infrastructure code
developed and deployed
for a specific workload
and its environments
Creates coupling across
the infrastructure
delivery lifecycle

28. ENVIRONMENT ENVIRONMENT ENVIRONMENT
Bespoke environments
Develop Deploy
Use
Infrastructure code
developed and deployed
for a specific workload
and its environments
Creates coupling across
the infrastructure
delivery lifecycle
This creates
issues at scale
Silos, bottlenecks

29. ENVIRONMENT
Development and deployment are coupled
Develop Deploy
Use
Responsibility for
developing, configuring,
and deploying
infrastructure code
Responsibility
for making
everything work

30. ENVIRONMENT
Decoupling development and deployment
Develop Deploy
Use
Responsibility for
developing
infrastructure code
Responsibility for
configuring,
deploying, and using
the infrastructure

31. Repository makes
components
available to deploy
ENVIRONMENT
Decoupling development and deployment
Develop Deploy
Use
Pipelines deliver
deployable infrastructure
components
Self-service
configuration and
deployment of
infrastructure
components

32. ENVIRONMENT
Decoupling development and deployment
Develop Deploy
Use
Multiple pipelines to
provide composable
infrastructure components
Infrastructure
components are
independently
deployed

33. Rethinking
infrastructure code
deployment

34. Application-
driven
infrastructure
deployment

35. Bottom-up infrastructure deployment
Deploy
Use
Configure and deploy
infrastructure in isolation
from deploying the
software that runs on it

36. Application-driven infrastructure deployment
Deploy
Use
Infrastructure deployment
is triggered when the
workload that uses it is
deployed

37. Application-driven infrastructure deployment
We can use this to deploy
different composable
infrastructure components
for multiple workloads
Infrastructure deployment
is triggered when the
workload that uses it is
deployed

38. Application-driven infrastructure deployment
But, how do we handle
deploying shared
infrastructure?
We can use this to deploy
different composable
infrastructure components
for multiple workloads
Infrastructure deployment
is triggered when the
workload that uses it is
deployed

39. Composable
deployment
contexts

40. Composable deployment contexts
Workload-specific
infrastructure
A context can include
multiple deployable
infrastructure
components

41. Composable deployment contexts
Workload-specific
infrastructure
Shared
infrastructure

42. Composable deployment contexts
Workload-specific
infrastructure
Shared
infrastructure
Workload groups

43. Composable deployment contexts
Workload-specific
infrastructure
Shared
infrastructure
Workload groups

44. Composable deployment contexts
Workload-specific
infrastructure
Shared
infrastructure
Platform services
Workload groups

45. Composable deployment contexts
Infrastructure should
be defined as close to
the workload as
possible
Lower contexts should
be ignorant of their
consumers

46. Deploying
infrastructure by
context

47. Options for
deploying
infrastructure by
context
●Application deployment trigger
●Pipeline stage
●Developer portal
●Platform framework

48. How can you use
these ideas?

49. Atlantis
Control Monkey
Crossplane
Digger
env0
Garden
Gruntwork DevOps
HCP Terraform
Harness
Pulumi Cloud
Scalr
Spacelift
Terrakube
Terramate
Terrateam
Infrablocks
Terragrunt
Terraspace
Cluster.dev
Gruntwork DevOps
Infrablocks
Resourcely
Tools and things that may or may not be useful
Codebase and builds
Deployment and
orchestration
Infrastructure products
and catalogs
http://infrastructure-as-code.com

50. Rethinking the infrastructure
delivery lifecycle
Moving beyond
spaghetti infrastructure
Global Infrastructure Practice Lead
July, 2024
http://infrastructure-as-code.com
Kief Morris
Thank
you!