Concourse - CI for the cloud

Continuous Deployment presents new challenges that traditional CI systems can rarely meet. Concourse is designed from the ground up for the Continuous Delivery of Cloud native Software.

1. Concourse
CI for the Cloud
June 27th 2017

2. Agenda
-Why Concourse?
-Concourse Core Concepts
-Demo
-Scaling to Complex Pipelines

3. Continuous Integration

Integrate code in a shared repository
several times a day

Automate building and testing

Run builds in a controlled
environment
Continuous Delivery

Deliver functionality to customers several times a day
Short feedback cycles reduce risk, cost and time
Reduce “software inventory”

Continuous Deployment

Deploy several times a day

Automate the deployment of
software

Infrastructure and Configuration as
Code reduce risk

4. Challenges of “Traditional” CI Systems
●Builds use an ever increasing amount of resources
○git repos, docker images, credentials, …

●Cloud Native: Build Agents as Pets vs. Cattle
○Repeatable builds?
○Clear build step inputs and outputs?

●(Ab?)using CI to orchestrate deployment
○Modelling “deployment destinations”

●Configuration as Code vs. manually built Pipelines

5. Concourse: Solutions
●Builds use an ever increasing amount of resources
○git repos, docker images, credentials, …

●Cloud Native: Build Agents as Pets vs. Cattle
○Repeatable builds?
○Clear build step inputs and outputs?

●(Ab?)using CI to orchestrate deployment
○Modelling “deployment destinations”

●Configuration as Code vs. manually built Pipelines

Resources
Jobs and
Pipelines
Tasks and
Workers

6. The Three Core Concepts of Concourse
●Resources
○any entity that can be checked for new versions
○Pull at a specific version
○Push up to idempotently create new versions

●Tasks
○execution of a script in a container
○well-defined inputs and outputs via volume mounts

●Jobs
○connect tasks and resources and can depend on other Jobs
○Jobs define the shape of your delivery pipeline.

7. Demo
Based on https://github.com/starkandwayne/concourse-tutorial

8. Scaling to Complex Pipelines

9. Scaling to complex Pipelines
Use an Artifacts Store:
-e.g. S3, Swift, HTTP
-FTP, rsync, …
Use custom Caches:
-e.g. for npm, yarn, mvn, bundler...
-with custom versioning (e.g. last git ref
that changed a yarn.lock file)
-https://www.meshcloud.io/en/2017/05/25/caching-dir
ectories-in-concourse-ci-pipelines/

10. Scaling to complex Pipelines
Extend your Pipeline with Resources
There’s likely already a
Resource for almost anything
●SCM: git, hg, perforce, ...
●Notifications: Slack,
Twitter, Email, HipChat…
●Deploy: Cloud Foundry,
Kubernetes, BOSH

11. Scaling to complex Pipelines
Use Resources to update your Team about Build Status

12. Scaling to complex Pipelines
aggregate: execute steps in parallel

13. Scaling to complex Pipelines
do: eliminate repetition with YAML Anchors

14. Scaling to complex Pipelines
Grouping Jobs

15. Scaling to complex Pipelines
Implement custom Resources

●Sounds scary, but it’s easy. All you need is
○A docker image...
○which supports three executables that consume/return simple
JSON
■check: check if new versions of a resource are available
■in: fetch a specific version of a resource
■out: idempotently push a specific version
●Resources may “nop” operations

16. End
Johannes Rudolph <[email protected]>


www.meshcloud.io

@meshstack

@meshstack

17. Links
●Great interactive Tutorial with many Samples
○https://github.com/starkandwayne/concourse-tutorial
●Official Concourse Docs
○https://concourse.ci/
●Setting up Concourse with docker-compose
○https://concourse.ci/docker-repository.html (local install)
○https://www.meshcloud.io/en/2016/12/22/deploying-concourse-ci-on-openstack-usin
g-docker/ (semi production-ready setup)
●Concourse Tricks
○https://github.com/pivotalservices/concourse-pipeline-samples