KCDC- Keeping Secrets Out of Your Pipeline

Commit your code, push a button, and everything is deployed – the Holy Grail of DevOps. Nobody has to type any commands or passwords to push your app live, everything lives in source control and it all just works.
Wait a second! My passwords are in source code? How did they get in there? How can I stop it from happening again? And how can I keep credentials out of source code and still make them available to my DevOps pipeline?
We’ll talk about using the open-source TruffleHog tool to find sensitive information in our source code repositories. And how to catch credentials before they are exposed. Finally, we’ll look at HashiCorp Vault, another open-source tool designed specifically to securely store and retrieve secrets from the pipeline without making them available to everyone.

1. Keeping Secrets
Out of Your Pipeline
Gene Gotimer
Principal DevOps Engineer at Praeses, LLC
@OtherDevOpsGene
June 28, 2024 #KCDC2024

2. The State of Secrets Sprawl Report
@OtherDevOpsGene #KCDC2024 2
https://www.gitguardian.com/state -of-secrets-sprawl-report-2024

3. Secrets
@OtherDevOpsGene #KCDC2024 3

4. What are secrets?
•Passwords
•Credentials
•API keys
•Signing keys
•SSH keys
•Access tokens
@OtherDevOpsGene #KCDC2024 4

5. The scenario
•Working towards continuous delivery and DevOps
•Writing automated deployment code
•And writing automated tests
•So, you have credentials locally in your environment
•Pipeline needs those credentials as well
@OtherDevOpsGene #KCDC2024 5

6. Oops!
•File with the credentials gets included with source control
•“But I thought that repo was private”
•Keys get hard-coded, “just while we are testing”
•Permissions wide open so we don’t need credentials in dev
•Written to an issue ticket
•Packaged with a release
•Included in a Docker image
@OtherDevOpsGene #KCDC2024 6

7. Detecting committed secrets
@OtherDevOpsGene #KCDC2024 7

8. Some tools to try
TruffleHog OWASP WrongSecrets
@OtherDevOpsGene #KCDC2024 8
https://github.com/trufflesecurity/trufflehog https://github.com/OWASP/wrongsecrets

9. Using TruffleHog
$ trufflehog github --repo https://github.com/OWASP/wrongsecrets \
--issue-comments --pr-comments
$ trufflehog github --org OtherDevOpsGene \
--issue-comments --pr-comments --gist-comments
$ trufflehog docker --image webgoat/webgoat-8.0:latest
$ trufflehog --help
@OtherDevOpsGene #KCDC2024 9

10. Found some. So now what?
1.Revoke the credentials
•Do not bother finding out first if anyone has seen them
•Assume they are compromised,
even when you are 100% sure it couldn’t happen
2.Plug the leak
3.Rotate the credentials- https://howtorotate.com/
@OtherDevOpsGene #KCDC2024 10

11. What about just fixing the repo?
•Don’t
@OtherDevOpsGene #KCDC2024 11

12. If you insist
$ git reset HEAD~ --soft # or HEAD@2 if it was two commits back
# Make your changes and commit the right stuff
$ git push origin --force
@OtherDevOpsGene #KCDC2024 12

13. Catch secrets before commit
@OtherDevOpsGene #KCDC2024 13

14. pre-commit hooks
@OtherDevOpsGene #KCDC2024 14
https://pre-commit.com/

15. .pre-commit-config.yaml
repos:
- repo: https://github.com/trufflesecurity/trufflehog
rev: v3.78.2
hooks:
- id: trufflehog
stages: ["commit", "push"]
@OtherDevOpsGene #KCDC2024 15

16. Set up pre-commit hooks
$ pip install pre-commit # add -U to upgrade
$ pre-commit install
@OtherDevOpsGene #KCDC2024 16

17. Storing secrets secretly
@OtherDevOpsGene #KCDC2024 17

18. HashiCorp Vault
open-source hosted service
@OtherDevOpsGene #KCDC2024 18
https://vaultproject.io/ https://hashicorp.com/products/vault

19. AWS Secrets Manager
@OtherDevOpsGene #KCDC2024 19
https://aws.amazon.com/secrets -manager/
Similar to
Azure Key Vault
GCP Secret Manager

20. Avoiding secrets entirely
@OtherDevOpsGene #KCDC2024 20

21. 3 factors of authentication
•Something you know
•password
•Something you have
•authenticator app on smartphone
•smart card
•physical key
•Something you are
•fingerprint
•retinal scan
•other biometric method
@OtherDevOpsGene #KCDC2024 21

22. AWS IAM Roles
•Scenario
•EC2 instance needs to access an RDS database
•Traditional solution
•Create username and password and store on EC2 instance
•Better solution
•Create username and password and store in AWS Secrets Manager
•Even better solution
•Assign IAM role to EC2 instance that has access to the RDS database
@OtherDevOpsGene #KCDC2024 22

23. Honeytokens
@OtherDevOpsGene #KCDC2024 23

24. Honeypots
•Unused systems that look like valuable targets
•Aren’t referenced or linked to anywhere
•Deliberately vulnerable
•If someone accesses them, they are likely an attacker
•They can be tracked and/or blocked elsewhere
@OtherDevOpsGene #KCDC2024 24

25. Honeytokens
•Similar to honeypots
•Unused, but valid-looking credentials, URLs, files, API keys
•Stored somewhere that should be secure
•private Git repo
•S3 bucket
•CI environment variable
•company Slack
•If someone tries to use them, they are likely an attacker
@OtherDevOpsGene #KCDC2024 25

26. Honeytoken providers
@OtherDevOpsGene #KCDC2024 26
https://canarytokens.org/
Alternatives:
•GitGuardian ggcanary
•SpaceSiren
https://github.com/GitGuardian/ggcanary
https://github.com/spacesiren/spacesiren

27. Someone found one. So now what?
•You know that your secure location has been compromised
•Anything you thought was secure in there is not
•Check permissions
•Rotate credentials
•Replace the honeytoken
@OtherDevOpsGene #KCDC2024 27

28. Wrap up
@OtherDevOpsGene #KCDC2024 28

29. Key takeaways
•Scan your code for secrets.
•Don’t let secrets into your code.
•Assume exposed secrets are compromised and rotate them.
•Immediately!
•Use a secrets-as-a-service solution.
•Use honeytokens to alert you to private areas being exposed.
@OtherDevOpsGene #KCDC2024 29

30. Tools
•TruffleHog- https://github.com/trufflesecurity/trufflehog
•How To Rotate- https://howtorotate.com/
•pre-commit- https://pre-commit.com/
•HCP Vault Secrets- https://hashicorp.com/products/vault
•AWS Secrets Manager- https://aws.amazon.com/secrets -manager/
•Canary Tokens- https://canarytokens.org/
•GitGuardian ggcanary- https://github.com/GitGuardian/ggcanary
•SpaceSiren- https://github.com/spacesiren/spacesiren
@OtherDevOpsGene #KCDC2024 30

31. Sample apps
•OWASP WrongSecrets- https://github.com/OWASP/wrongsecrets
•lotr- https://github.com/OtherDevOpsGene/lotr
@OtherDevOpsGene #KCDC2024 31

32. The State of Secrets Sprawl Report
@OtherDevOpsGene #KCDC2024 32
https://www.gitguardian.com/state -of-secrets-sprawl-report-2024

33. Questions?
Gene Gotimer
Principal DevOps Engineer at Praeses, LLC
@OtherDevOpsGene
@OtherDevOpsGene #KCDC2024 33