Terraform: Tales from the Trenches

Terraform is an infrastructure-as-code software solution that enables businesses to safely and predictably create, change, and improve infrastructure. It is all the rage right now in the DevOps community. In this presentation, experts from HG Data share some hands-on experiences and introduce some practical applications of terraform beyond the traditional “Hello World” example. This presentation covers evolving Terraform projects, implementing behavioral testing, and tips to achieve blue/green deployment zen.

1. Terraform: Tales from the Trenches Santa Barbara DevOps Copyright © HG Data 2019

2. Presenters Rob Fox CTO Sam Chapin Chief Architect Brendan Keane DevOps Engineer Chris Deutsch DevOps Architect

3. The Story Thus Far... Rob Fox

4. Background Product: https://discovery.hgdata.com ( SaaS Platform and related services) Issues Includes: Previous iteration of infrastructure as code fell way short Rigid configuration Ansible+bash hell No automation No ability to validate No visibility as to what was actually running and where Holes around secrets management Had to run on a specific version of Ubuntu Copyright © HG Data 2019

5. Why Terraform @ HG Data? Infrastructure as Code Immutable Infrastructure Declarative Client Only Solves a lot of our current issues Copyright © HG Data 2019

6. Success so far... Customer facing platform completely refactored using Terraform, Kubernetes, Vault Blue/Green Deployment tied into CI/CD Many shared modules and libraries in github created and distributed to rest of engineering Platform rock solid (easy to deploy, easy to manage and monitor) Big Data team now adopting Terraform and related tooling to manage large-scaled Spark clusters and pipeline automation/orchestration Engineers much happier and way more productive!! Copyright © HG Data 2019

7. What we learned along the way... ...A lot!! What follows are three mini-presentations based on our learnings and experiences to help you be successful Enjoy! Copyright © HG Data 2019

8. Evolution of a Terraform project Sam Chapin

9. me.tf :

10. Step 1. Experimentation (Someone allowed me to play with cool new toys) The starting state of the tf project: Single developer Toy deployment No "environment" No modularization No shared state Copyright © HG Data 2019

11. Step 1. Experimentation (The halcyon days) main.tf: Hierarchy: terraform apply -auto-approve Example usage: Copyright © HG Data 2019

12. Step 1. Experimentation - What Hurts? no sandbox isn't parameterizable hard to test non-DRY can't execute anywhere Copyright © HG Data 2019

13. Step 2. Reality sets in (My boss tells me I need to actually deploy it) The OLD State of the tf project: Single developer Toy deployment No "environment" No modularization No shared state The NEW State of the tf project: Single developer Toy deployment Prod/Stage/Dev deployment No "environment" Environment management No modularization No shared state Copyright © HG Data 2019

14. Step 2. Reality sets in - Environment via configuration prod.tfvars: Hierarchy: terraform apply \ -var-file=prod.tfvars Example usage: Copyright © HG Data 2019

15. Step 2. Reality sets in - Environment via symlinks conf.auto.tfvars: Hierarchy: cd prod terraform apply Example usage: Copyright © HG Data 2019

16. Step 2. Reality sets in - Environment via workspace conf.auto.tfvars: Hierarchy: terraform workspace new prod terraform apply Example usage: Copyright © HG Data 2019

17. Step 2. Reality sets in - Environment via workspace & consul If you’re lucky enough to already have consul... Copyright © HG Data 2019

18. Step 2. Reality sets in - What Hurts? no sandbox isn't parameterizable hard to test non-DRY can't execute anywhere Copyright © HG Data 2019

19. Step 3. Refactor (I panic at how much I've been copy-pasting) The OLD State of the tf project: Single developer Prod/Stage/Dev deployment Environment management (confs, copies, or workspaces) No modularization No shared state The NEW State of the tf project: Single developer Prod/Stage/Dev deployment Environment management (confs, copies, or workspaces) No modularization Modularization ( inner modules / module library ) No shared state Copyright © HG Data 2019

20. Step 3. Refactor - Modules Hierarchy: modules/instance/main.tf:

21. Step 3. Refactor - Modules Hierarchy: ./main.tf: Copyright © HG Data 2019

22. Step 3. Refactor - Modules Hierarchy: More on this from Brendan! Copyright © HG Data 2019

23. Step 3. Refactor - External Modules What if we want them to be external libs? Copyright © HG Data 2019

24. Step 3. Refactor - What Hurts? no sandbox isn't parameterizable hard to test non-DRY can't execute anywhere Copyright © HG Data 2019

25. Step 4. Make it easy (Someone is going to see the mess I made) The OLD State of the tf project: Single developer Prod/Stage/Dev deployment Environment management (confs, copies, or workspaces) Modularization (inner modules or reusable module library) No shared state The NEW State of the tf project: Single developer Multiple developers Prod/Stage/Dev deployment Environment management (confs, copies, or workspaces) Modularization (inner modules or reusable module library) No shared state Shared state Copyright © HG Data 2019

26. Step 4. Make it easy - Shared state across environments Hierarchy: ./main.tf: Modularize this!

27. Step 4. Make it easy - State locking Lock your damn state. All the standard backends support state locking. Copyright © HG Data 2019

28. Backend Step 4. Make it easy - State locking - A tale of two terraformers .tfstate Apply Apply Copyright © HG Data 2019

29. Step 4. Make it easy - What Hurts? no sandbox isn't parameterizable hard to test non-DRY can't execute anywhere Collaboration with others (PR plan workflow) CI/CD locking your state all the time (Build queue) Check out https://www.runatlantis.io/ Okay, okay... What Still Hurts? Copyright © HG Data 2019

30. In summary... Experiment and get familiar with your new declarative friend Solve your environment problem with convention & opinions Make your codebase small and testable with modules Ease collaboration & FUD via state sharing & locking Copyright © HG Data 2019

31. Terraform, Behave! Brendan Keane

32. What we needed Shared modules Certainty on shared code. Ability to build on top of shared code, we need a high level of certainty Copyright © HG Data 2019

33. What would make us more certain? Tests! But what kind of tests in this scenario? Copyright © HG Data 2019

34. Do what I say not what I mean Terraform does this for you with every action. resource “aws_instance” “my_server” { ami = “ami-0c32356aac847d7e8” } resource “aws_rds_instance” “my_database” { ami = “ami-0c223918e1” } Copyright © HG Data 2019

35. Do what I mean not what I say Terraform does not test for what you mean . subject { “my_server” } it “should be able to reach the database” do # db connection test end Copyright © HG Data 2019

36. Do what I say not what I mean Great for auditing known good configuration Do what I mean not what I say Great for iterating towards a good configuration Copyright © HG Data 2019

37. Base Case: Assert: A can ping B B A Tests more than at first sight: security groups, subnets, routing, dns… Terraform tells me everything is as I declared it. Terraform does not tell me if it does what I want it to do. Copyright © HG Data 2019

38. Building Blocks

39. Building Blocks - Bats! https://github.com/bats-core/bats-core Rspec-ish.No ‘shelling out’, can run where infra is stood up Good helpers for stdout, stderr, and exit status. Before and after hooks if needed. Copyright © HG Data 2019

40. Building Blocks - Templating https://www.terraform.io/docs/providers/template/d/file.html ssh driven: this is a plus and minus. Many approaches: scripts, inline, powerful when combined with templating. Copyright © HG Data 2019

41. Building Blocks - Remote Exec https://www.terraform.io/docs/provisioners/null_resource.html Copyright © HG Data 2019

42. Building Blocks - Structure Copyright © HG Data 2019

43. Spec Run Host A Install bats Run bats Dev Machine Terraform apply 1. Spin up 3. Runs tests in cloud context 2. Remote-exe Bats Host B Copyright © HG Data 2019

44. Proof Pudding (Demo Time) Copyright © HG Data 2019

45. Concluding Thoughts Keep the tests simple, too many null_providers == bad Positive assertions are usually functionality: A can ping B Negative assertions are usually security: B cannot ping A Future: dockerize, dashboard of health tests orthogonal from application health checks. Copyright © HG Data 2019

46. Terraforming Blue/Green Deployments Chris Deutsch

47. The problem... “One of the challenges with automating deployment is the cut-over itself, taking software from the final stage of testing to live production.” https://www.martinfowler.com/bliki/BlueGreenDeployment.html Copyright © HG Data 2019

48. Example basic setup Copyright © HG Data 2019

49. Blue and green deployments Copyright © HG Data 2019

50. Users are routed to green (live) Copyright © HG Data 2019

51. Dev/QA are routed to blue (next) Copyright © HG Data 2019

52. Bringing a new release live Copyright © HG Data 2019

53. Oh no! Copyright © HG Data 2019

54. Recovery: flip live back to green Copyright © HG Data 2019

55. Considerations While the idea is relatively simple, here are three important things to keep in mind. Copyright © HG Data 2019

56. 1. Infrastructure We have two clusters: blue and green Each cluster should be configured in the same way Each cluster should operate independently and be easy to maintain In addition, the more we split things up, the more value we can get out of blue/green deploys for our infrastructure . But we also increase cost and complexity. Copyright © HG Data 2019

57. 2. Configuration Management Applications should have the same configuration regardless of whether they are blue or green. Applications should have the same configuration regardless whether they are the live version or the next version. (Well… as much as is reasonable.) Copyright © HG Data 2019

58. 3. Application State “Twelve-factor processes are stateless and share-nothing. Any data that needs to persist must be stored in a stateful backing service, typically a database.” https://12factor.net/processes Copyright © HG Data 2019

59. Terraform + ELB + Kube + RDS + ElastiCache Copyright © HG Data 2019

60. Module 1: Terraforming a Load Balancer Copyright © HG Data 2019

61. Starting the app_load_balancer module Copyright © HG Data 2019

62. Add an aws_lb_target_group... Copyright © HG Data 2019

63. And an aws_lb_listener_rule... Copyright © HG Data 2019

64. Okay! Time to use it Copyright © HG Data 2019

65. Module 2: the blue deployment

66. Starting the app_cluster module Copyright © HG Data 2019

67. Attaching the Instance to a Target Group my_arns = { next = “some_next_arn” live = “some_live_arn” } lookup(var. my_arns, “next”) => “some_next_arn”

68. Excellent! Time to use it Copyright © HG Data 2019

69. Module 3: the green deployment Copyright © HG Data 2019

70. (It’s actually almost the same as blue) Copyright © HG Data 2019

71. (It’s actually almost the same as blue) Color flipping , Terraform style Copyright © HG Data 2019

72. 1. Change Green From Next to Live Copyright © HG Data 2019

73. Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: yes Copyright © HG Data 2019

74. Now both are live! Copyright © HG Data 2019

75. Verify With Curl $ curl http://live.example.com/ I'm an app running in the production environment on green $ curl http://live.example.com/ I'm an app running in the production environment on blue Copyright © HG Data 2019

76. 2. Move Blue From Live To Next Copyright © HG Data 2019

77. Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: yes Copyright © HG Data 2019

78. Now green is live, but blue is next Copyright © HG Data 2019

79. Verify With Curl $ curl http://live.example.com/ I'm an app running in the production environment on green $ curl http://next.example.com/ 503 Service Temporarily Unavailable # Wait a couple of minutes... $ curl http://next.example.com/ I'm an app running in the production environment on blue Copyright © HG Data 2019

80. Deploy Complete! Copyright © HG Data 2019

81. Q&A Thank you! Follow us online @ https://www.hgdata.com