Secure second days operations with Boundary and Vault.pdf

We now live in the age of cloud native which also means that we now have an incredibly dynamic fleet of nodes to maintain. Instead of injecting a well known SSH key into our entire infrastructure this talk will show how we can utilise Boundary and Vault to build a zero trust system. This than allows us to hook boundary up to a config management tool like ansible to manage our nodes

1. Secure second day
operations with
Boundary and
Vault

Bram Vogelaar
@attachmentgenie

2. “I bet I know the most
powerful user in your
platform”

3. “I am 99.9% sure it is not
root”

4. “It probably is your own ssh key”

5. “I am also pretty sure it’s
as old as your platform”

6. ~ ❯ whoami
Previously a Molecular Biologist
Then became a Dev, now an Ops

Amsterdam HUG organizer
HashiCorp Ambassador 2024
HashiCorp Core Contributor 2024

7. On today’s agenda

8. “Let's rotate the secrets …right now”

9. Connect GHA to Vault




CODE EDITOR
steps:
- name: Import Vault Secrets
id: vault_secrets
uses: hashicorp/vault-action@v3
with:
url: "https://nonofyourbusiness.hashicorp.cloud:8200"
namespace: "admin/engineering"
githubToken: ${{ secrets.CI_RUNNER_ACCESS_TOKEN }}
method: approle
roleId: ${{ secrets.CI_VAULT_ROLE_ID }}
secretId: ${{ secrets.CI_VAULT_SECRET_ID }}
secrets: |
shared/secrets/data/not_a_secret foo ;
- name: Print foo secret length
run: echo -n $FOO | wc -l

10. Create a Github application

11. Create a Github application

12. Connect your github application

13. Connect your github application




CODE EDITOR
data "github_repository" "atc" {
full_name = "attachmentgenie/atc"
}
resource "github_app_installation_repository" "atc" {
installation_id = "1234567"
repository = data,github_repository.atc.name
}

14. Link the Github application



CODE EDITOR
vault write sys/sync/github-apps/hashidays_gha_secrets \
app_id=<APP_ID> \
private_key=<PATH_TO_PRIVATE_KEY>

15. Create the sync location


CODE EDITOR
vault write sys/sync/destinations/gh/hashidays-gha-secrets \
installation_id=<INSTALLATION_ID> \
repository_owner=<GITHUB_USER> \
repository_name=<MY_REPO_NAME> \
app_name=<APP_NAME>

16. Create the secret and sync instructions



CODE EDITOR
vault secrets enable -path=hashidays-gha-secrets kv-v2
vault kv put -mount='hashidays-gha-secrets' my-secret foo='bar'
vault write sys/sync/destinations/gh/hashidays-gha-secrets/associations/set \
mount=hashidays-gha-secrets \
secret_name=my-secret

17. “O11y exposes a wealth of
information”

18. Securing Node Exporter



CODE EDITOR
tls_server_config:
cert_file: /etc/node_exporter/ssl/node.pki.vagrant.cert
key_file: /etc/node_exporter/ssl/node.pki.vagrant.key
client_auth_type: RequireAndVerifyClientCert
client_ca_file: /etc/node_exporter/ssl/ca.cert

19. Securing Node Exporter



CODE EDITOR
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: "tls"
static_configs: - targets: [”node.pki.vagrant:9100"]
scheme: https
tls_config:
cert_file: /etc/prometheus/ssl/prometheus.pki.vagrant.cert
key_file: /etc/prometheus/ssl/prometheus.pki.vagrant.key
ca_file: /etc/prometheus/ssl/ca.cert

20. Vault Agent Config





CODE EDITOR
vault {
address = "http://prometheus.pki.vagrant:8200"
}
auto_auth {
method "approle" {
config = {
role_id_file_path = "/vagrant_data/tmp/role_id"
secret_id_file_path = "/vagrant_data/tmp/secret_id"
remove_secret_id_file_after_reading = false
}
}
}


* Wildly insecure config, for demo purpose only

21. Vault Agent Config Cont.






CODE EDITOR
template {
contents="{{ with secret \"pki_int/issue/pki-dot-vagrant\" \"ttl=3m\"
\"common_name=prometheus.pki.vagrant\" }}{{ .Data.certificate }}{{ end }}"
destination="/etc/prometheus/ssl/prometheus.pki.vagrant.cert"
command = "killall -HUP prometheus"
}
…
{{ .Data.issuing_ca }} => "/etc/prometheus/ssl/ca.cert”
{{.Data.private_key }} => "/etc/prometheus/ssl/prometheus.pki.vagrant.key"

22. Build In certificate expiration

23. “Ben the BI guy, needs
database access for a bit”

24. Single Use Database Credentials






CODE EDITOR
vault secrets enable database

vault write database/config/postgresql \
plugin_name=postgresql-database-plugin \
connection_url="postgresql://{{username}}:{{password}}@$POSTGRES_URL/postgres" \
allowed_roles=readonly \
username="root" \
password="nolongercommonknowledge"

25. Create “create user” instructions






CODE EDITOR
tee readonly.sql <<EOF
CREATE ROLE "{{name}}" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'
INHERIT;
GRANT ro TO "{{name}}";
EOF

vault write database/roles/readonly \
db_name=postgresql \
[email protected] \
default_ttl=1h \
max_ttl=24h

26. Create “create user” instructions






CODE EDITOR
vault read database/creds/readonly

Key Value
--- -----
lease_id database/creds/readonly/fyF5xDomnKeCHNZNQgStwBKD
lease_duration 1h
lease_renewable true
password A1a-ckirtymYaXACpIHn
username v-token-readonly-6iRIcGv8tLpu816oblPY-1556567086

27. “Can you just put on the
sftp server anyway?”

28. Conway’s Law OU group

30. Boundary as Source of truth

31. Model your organization
CODE EDITOR
resource "boundary_scope" "org" {
name = "hashdays inc"
description = "top level"
scope_id = "global"
auto_create_admin_role = true
auto_create_default_role = true
}

resource "boundary_scope" "bi_team" {
name = "platform team"
description = "project for the BI team"
scope_id = boundary_scope.org.id
auto_create_admin_role = true
}

32. Infrastructure Mesh as Code


CODE EDITOR
resource "boundary_target" "bi_pgsql" {
name = "bi_pgsql"
description = "BI Postgresql"
address = aws_instance.hashidays.public_ip:5432
type = "tcp"
default_port = "5432"
scope_id = boundary_scope.bi_team.id
}


boundary connect postgres -target-id ttcp_eTcZMueUYv -username admin -dbname postgres

33. “Can you update that
webserver, you know the
one i am talking about”

34. CODE EDITOR
resource "boundary_host_catalog_static" "webservers" {
name = "webservers"
scope_id = boundary_scope.project.id
}

resource "boundary_host_static" "webserver" {
name = "webserver"
address = aws_instance.hashidays.public_ip
host_catalog_id = boundary_host_catalog_static.webservers.id
}
Model your fleet

35. CODE EDITOR
resource "boundary_credential_store_vault" "platform" {
name = "platform”
address = var.vault_url
token = vault_token.hashidays.client_token
scope_id = boundary_scope.project_team.id
}

resource "boundary_credential_library_vault" "platform" {
name = "platform"
credential_store_id = boundary_credential_store_vault.platform.id
path = "ssh/connect/hashidays"
http_method = "GET"
credential_type = "username_password"
}
Define what credentials to use

36. CODE EDITOR
resource "boundary_target" "webserver" {
name = "webserver"
description = "public website"
type = "ssh"
default_port = "22"
scope_id = boundary_scope.platform_team.id
host_source_ids = [
boundary_host_set.webserver.id
]
injected_application_credential_source_ids = [
boundary_credential_library_vault.platform.id
]
}
Credential Injection from vault

37. Boundary as ansible transport

CODE EDITOR
hashidays_inventory:
hosts:
webserver:
ansible_host: 18.196.188.109
ansible_ssh_user: ubuntu
ansible_ssh_private_key_file: /home/attachmentgenie/.ssh/id_rsa_aws

hashidays_inventory:
hosts:
webserver:
ansible_host: 127.0.0.1
ansible_port: 42921

38. Inventory + Acls + API =
Break glass as Code
https://www.hashicorp.com/blog/event-driven-access-controls-with-hashicorp-boundary-and-vault

39. Thank you!




[email protected]
@attachmentgenie
https://www.slideshare.net/attachmentgenie