How build and deploy shapes software architecture at thetrainline.com

By ensuing our software builds and deployments are Reliable, Repeatable, Rapid and Recurring, we can produce software which is lightweight, testable, monitorable, configurable and recoverable. These properties enable features such as load-balancing, scaling, elasticity and rapid recovery, all key for high-performing web systems. R-R-R-R builds and deployments also enable Continuous Delivery of features into Production.

1. #webperfdays
@matthewpskelton
How build and deployment
shapes software architecture
at thetrainline.com
Matthew Skelton | thetrainline.com
WebPerfDaysEU, London | #webperfdays
05 October 2012

2. #webperfdays
@matthewpskelton
architecture
= f (build&deploy)
(for some systems)

3. #webperfdays
@matthewpskelton
Systems engineering
(robotics, control theory, sensors, neuroscience)
Software
development
(finance, insurance, travel, pharma, media, medical imaging)
now
Build & Deployment
at thetrainline.com

4. #webperfdays
@matthewpskelton

5. #webperfdays
@matthewpskelton
9m visits per month to thetrainline.com
2nd most visited UK travel booking website
(behind Easyjet)*
Over 100,000 daily visits via our mobile apps
*source: IMRG hotshopslist Nov 2011

6. #webperfdays
@matthewpskelton
Booking engines for many UK train operators
Ticketing systems for Corporatesand TMCs
Several hundred individual websites
> £1bn ($1.5bn USD) per year in train ticket sales

7. #webperfdays
@matthewpskelton

8. #webperfdays
@matthewpskelton
•Early “distributed agile” success story
•http://www.thoughtworks.com/trainline

9. #webperfdays
@matthewpskelton
10+ dev teams in 2 locations

10. #webperfdays
@matthewpskelton
#webperfdays
@matthewpskelton
Server Farm
(Build, Deployment, Replication)
100+ build agents
20 dev environments x 20 VMs per
environment = 400 dev VMs
Git replication (Gitolite), …
MORE SERVERS THAN PRODUCTION

11. #webperfdays
@matthewpskelton

12. #webperfdays
@matthewpskelton

13. #webperfdays
@matthewpskelton
architecture
= f (build&deploy)
(for some systems)

14. #webperfdays
@matthewpskelton
“HERESY!”
#webperfdays
@matthewpskelton

15. #webperfdays
@matthewpskelton
#webperfdays
@matthewpskelton

16. #webperfdays
@matthewpskelton
#webperfdays
@matthewpskelton

17. #webperfdays
@matthewpskelton
#webperfdays
@matthewpskelton

18. #webperfdays
@matthewpskelton
Web-based
Frequently-
changing
Public-facing
High-volume

19. #webperfdays
@matthewpskelton

20. #webperfdays
@matthewpskelton
RELIABLE
REPEATABLE
RAPID
RECURRING
#webperfdays
@matthewpskelton

21. #webperfdays
@matthewpskelton
‘R-R-R-R’
BUILD AND DEPLOYMENT
Helps to avoid the Ball of Mud
Enables high-performance
architectures
#webperfdays
@matthewpskelton

22. #webperfdays
@matthewpskelton
#webperfdays
@matthewpskelton

23. #webperfdays
@matthewpskelton
LOAD BALANCING
HIGH AVAILABILITY
SCALING
ELASTIC
RAPID RECOVERY
#webperfdays
@matthewpskelton

24. #webperfdays
@matthewpskelton
BUILDABLE
Small pipelined builds on generic build machines
Seconds, not minutesor hours
Short feedback cycles
(Dan Worthington-Bodart, @danielbodart-http://bit.ly/M85wsX)
#webperfdays
@matthewpskelton

25. #webperfdays
@matthewpskelton
Build dependencies may mirror runtime deps
msbuild/mfor multi-threaded builds
(forces component isolation)
Gallio(MbUnit) –replacement engine for NUnit
NuGet+ Squid for super-fast build artefacts
RAM disk for some working copy operations

26. #webperfdays
@matthewpskelton
TESTABLE
Test (separation, harnesses, points)
IDENTIFIABLE
Meaningful versions, packages,
defined dependencies, artefact
management
(think component boundaries)
#webperfdays
@matthewpskelton

27. #webperfdays
@matthewpskelton
Enforced “onion” testing layers
(test separation, build machine firewalls, build agent types, …)
Semantic Versioning
(A.B.C.D + contract to ‘clients’)
30+ external services
(test harnesses)

28. #webperfdays
@matthewpskelton
DEPLOYABLE
Rapid, scriptable, simple failure modes
MONITORABLE
Logging, metrics, transaction tracing
CONFIGURABLE
Inject settings –no ‘black boxes’
LIGHTWEIGHT
Keep things small and easily comprehendible
#webperfdays
@matthewpskelton

29. #webperfdays
@matthewpskelton
GO, MSDeploy, hand-rolled, Octopus, MSDeploy
(“as simple as xcopy”)
Graphite + SqlToGraphite
(https://github.com/perryofpeek/SqlToGraphite)
LogStash+ log4net
(tune logging levels with config, transaction tracing)
SCOM –operations manager
(early monitoring in Dev)

30. #webperfdays
@matthewpskelton
INSTANTIABLE
No snowflakes or singletons
RECOVERABLE
No nasty zombies after failures
MTTR more important than MTBF*
* for most kinds of F
#webperfdays
@matthewpskelton

31. #webperfdays
@matthewpskelton
Rebuild Build Agents with Chef
Chef for environment rebuilds
Call Start()and Stop()on each
component

32. #webperfdays
@matthewpskelton
#webperfdays
@matthewpskelton

33. #webperfdays
@matthewpskelton
Well-known monolithic business rules engine
Costly to build, deploy, configure, test
Being replaced

34. #webperfdays
@matthewpskelton
RELIABLE
REPEATABLE
RAPID
RECURRING
#webperfdays
@matthewpskelton

35. #webperfdays
@matthewpskelton
LOAD BALANCING
HIGH AVAILABILITY
SCALING
ELASTIC
RAPID RECOVERY
#webperfdays
@matthewpskelton
LIGHTWEIGHT
TESTABLE
MONITORABLE
CONFIGURABLE
RECOVERABLE
IDENTIFIABLE

36. #webperfdays
@matthewpskelton
architecture
= f (build&deploy)
(for some systems)
thank you
webperfdays.org
matthewskelton.net| @matthewpskelton
Thanks to: Attila S, Jack R and OwainP for feedback.
Picture credits:
Label: seanlabel.com; Sheep: gallery.hd.org; Train:worldontrains.blogspot.co.uk; Gurner: mog.com; Petra:
Wikimedia/Berthold Werner; army engineers: US DoD; ball of mud: pwern.blogspot.co.uk; sports car:xarj.net;
zombie: bjj.org; feather: Wikipedia; punch: thelegalblitz.com; passport: coverpalace.com; dogs:
reluctantmemsahib.wordpress.com; Meccano: dalefield.com