Adopting PCF At An Automobile Manufacturer

Thomas Seibert and Gregor Zurowski demonstrate how Mercedes-Benz.io has achieved to go from idea to production in no time. Through the evolution of an effective and developer oriented application generation framework, utilization of a highly automated tool chain, organizational improvements and the infrastructure provided by PCF they will describe how their delivery performance has increased on many dimensions and how their ecosystem allows for scaling to a multitude of teams.

1. ADOPTING PIVOTAL CLOUD FOUNDRY
AT AN AUTOMOBILE MANUFACTURER
THOMAS SEIBERT | GREGOR ZUROWSKI

2. GREGOR ZUROWSKI
Software Architect, Independent Consultant
[email protected]
ABOUT US
THOMAS SEIBERT
Lead Architect, Mercedes-Benz.io GmbH
[email protected]

3. OUR JOURNEY WITH PIVOTAL CLOUD FOUNDRY

4. Customer
Mercedes mePortal
Product
eMB
Brand
mercedes-benz.com
INITIAL STATE (2014)
Java EE Portal Server CMS
CMS
Brokencustomerjourney

5. VISION
Create the Best Customer Experience for all Daimler customers and prospects
▪Deliver the most relevant content
▪Provide the most useful functionality
▪Constantly innovate and improve
Mercedes-Benz

6. TARGET STATE
CUSTOMER PRODUCT
BRAND
PIVOTAL CLOUD FOUNDRY CMS
IaaS
ANALYTICS
Coherent
customer
journey

7. OUR DIMENSIONS OF SCALING
Three Dimensions:Content, Functions, Markets

8. ONEWEB TODAY
▪Collaboration with more than 10 external partners
▪Canary rollout to UK in 2016 with C-Class only
▪Rollout to Austria in summer 2017
▪20 countries in less than 5 months
•4 countries per month
•Up to 3 countries per week
▪0 critical production incidents
▪0 management escalations
▪40+ countries planned for 2018

9. ARCHITECTURAL DECISIONS FOR THE ONEWEBPLATFORM

10. PROJECT INCEPTION
▪Build a lightweight and elastic application landscape
▪Organizational blueprint: Startup setting
•Deliver in small increments
•Constantly collect feedback from POs and customers
•Process feedback into next iteration
▪Hypothesis-driven development
•Define KPIs for each component
•Quick validation of our technical and business hypotheses
▪Create an efficient, scalable and extensible system for a global web presence

11. ARCHITECTURAL DECISIONS (1/2)
▪Use cloud infrastructure (IaaS)
•Faster provisioning of virtualized hardware
•Scale as you go, pay as you go
▪Use a cloud application platform (PaaS)
•Focus on developing business capabilities
•Deploy to a modern container environment
•Build on a ready-made outer architecture
•Automate configuration, building, deploying and operations

12. ARCHITECTURAL DECISIONS (2/2)
▪Adopt a Microservices-oriented architectural style
•For all benefits Microservices provide :)
•Be careful not to over-architect; experiment!
•Define common principles and guidelines
▪When scaling out with teams, decouple as much as possible
•Development efforts in parallel
•Isolated processes running in containers
•Independent releases and deployments
•Component intercommunication via HTTP or messaging

13. TECHNOLOGICAL LANDSCAPE
Daimler Data Center
API Gateway
CMS
API Gateway
App App
App
App
App
App

14. ENABLING OF TEAMS (1/2)
▪Minimize ramp up time of teams
•Automation of space creation, permission handling and service bindings
•Maven archetypes for OneWebcompliant applications
▪Allow self-servicing of teams as much as possible
•Service provisioning (RabbitMQ, Gemfire, various databases etc.)
•Usage of deployment templates
•Access to monitoring and logging

15. ENABLING OF TEAMS (2/2)
▪Common components for
•Default error handling
•Default actuator endpoints
•Other generic functionality
▪Ecosystem guidelines
•Give teams the safety to develop in a self-responsible manner
•Blueprints for documentation and specification
•Definition of good citizens

16. Setup ofspace, serviceskeletons, deploymentpipeline
andteampermissions
PRESENT
DEPLOYMENT &
INTEGRATION
REQUIREMENTS
ARCHITECTURE & DESIGN
IMPLEMENTATION & CONT. TESTING
INCEPTION
Ideation, scopingetc.
OPTIMIZING DEVELOPMENT FLOW
DEPLOYMENT &
INTEGRATION
IMPLEMENTATION
& CONTINUOUS
TESTING
PAST
REQUIREMENTS
Hardware provisioning&
hostingconfiguration
ARCHITECTURE &
DESIGN
INCEPTION
Ideation, scopingetc.
~30
DAYS
~2-3 DAYS
waitingtime
INCEPTION
Ideation, scopingetc.

17. INTEGRATING CLOUD FOUNDRY WITH OUR ARCHITECTURE

18. INCREASING EFFICIENCY WITH ISOLATION
▪Three levels of isolation
•Orgs and spaces provide isolation between applications and users
•Containers provide process-level isolation between applications
•A custom versioning concept allows to deploy different versions of applications in
parallel providing isolation of releases
BUT:
▪No isolation for updates of product tiles (e.g. Spring Cloud Services, Gemfire)
▪How do we update a product tile without affecting applications running in
production?

19. ISOLATION OF PCF FOUNDATIONS
▪Separate foundations for each deployment environment (e.g. development/test,
integration/pre-production and production)
▪Test product tile updates and gradually move changes to higher level environments
▪Uses a traditional workflow of promoting changes
▪Simple, stable and effective
Update Update
PROD
Foundation
<<Foundation>>
DEV/TEST
Spring Cloud Services
Version 1.4
Spring Cloud Services
Version 1.4
<<Foundation>>
PROD
Spring Cloud Services
Version 1.3
<<Foundation>>
INT / PRE-PROD
Spring Cloud Services
Version 1.4

20. ISOLATION WITH ORGS AND SPACES
▪PCF implements multitenancy with orgs and spaces
▪Each team gets its own space for deployment of
components
▪Provides maximum freedom for teams without the
risk of affecting other teams
▪Roles and permissions define what teams can see
and do
Teams work isolated from each other
<<Foundation>>
<<Organization>>
<<Space>>
Project 1
<<Space>>
Project 2
<<Space>>
Project 3
<<Space>>
Project 4
App
App App App
App App

21. SOLUTION FOR SHARED SERVICES
<<Foundation>>
<<Organization>>
<<Space>>
Product 1
<<Space>>
Product2
App App App
<<Space>>
SharedServices
Service
Registry
Config
Server
▪Shared services
•Generic functionality
•Maintained by platform team
•Example: Service Registry, Config Server
▪Product specific spaces
•Business functionality
•Dependent on shared services
•Developed by product teams
SharedserviceProductspecificservice

22. IMPLEMENTATION OF SHARED SERVICES
▪Disadvantages
•Requires SpaceDeveloperpermissions for viewing service dashboards
•Missing overview of currently active consumers (no delete protection)
•Tagging unavailable for user provided services which mandates static naming schemes
▪Advantages
•Reduces maintenance
•Simple, stable and effective
•Aligned with PCF future architecture (Pivotal is working on making shared services a first
class citizen)

23. DEVELOPMENT WITH CLOUD FOUNDRY

24. OUR APPROACH TO SERVICE VERSIONING
▪Enterprise systems must support versioning
▪Selection of SemVerfor our versioning concept
▪Every major version is a separate PCF app and deployed individually
•Apps only expose major versions through their APIs (e.g. v1, v2, etc.)
•Simplifies development and testing of new major versions
•Simplifies upgrading and transitioning to newer technologies
•Decouples changes in business logic

25. IMPLEMENTATION OF OUR VERSIONING CONCEPT
▪Apps register with major versions
▪Configuration is versioned in Git repositories
▪API Gateway detects version and forwards to appropriate service
Client
Service
Registry
API
Gateway
Config Server
service
config
service-v2
service-v2
Git Config
Repository
<<Foundation>>
properties
/service/v2
App
v1
App
v2

26. API GATEWAY
▪Edge service as a single entry point controlling external access
to our services
•Uniform endpoint behavior and same base URL for all services
•Centralized responsibility for
•Dynamic routing and service discovery
•Circuit breaking
•Client-side load balancing
•Timeouts
▪Netflix Zuul
•Light-weight and simple implementation
•Extensible with filters for fine-grained control
API Gateway
Comp 3 Comp 4
Comp 1 Comp 2
APIs
Service 1
Service 2
Service 3
•Retries
•Tracing
•Deliberate message transformation
•Security

27. API GATEWAY
Insights
▪Endpoint behavior can be managed centrally
▪No self-service for dev teams
▪No service catalog
▪Limited feature set compared to API management solutions

28. BLUE/GREEN DEPLOYMENTS
▪Precondition for continuous deployments and going into production
▪Also essential for non-production environments
▪Zero downtime deployments are no first-class citizens in PCF
▪Custom implementation following a simplified approach
▪Multiple deployments per day without disruption of other teams

29. DEVELOPMENT CHALLENGES
▪Insights based on our stack with Spring Boot and Spring Cloud
▪Various issues with Pivotal Cloud Foundry and Spring Cloud Services
•Garbage collection of dynamic proxies led to OOMs
•Spring Boot CF App Manager Actuator integration not compatible with custom
context paths
•Recommended blue/green deployment not aligned with SCS Service Registry
•SCS Config Server with reduced feature set compared to OSS version
▪Update of Spring Cloud Services necessitates update of every dependent
application

30. LESSONS LEARNED

31. LESSONS LEARNED
▪Automation provides obvious benefits, but not always easy to achieve
•Incrementally increase the level of automation
•To scale out, providing a toolbox to developers for automating project setup, build and
deployments is essential
▪A PaaS solution like PCF can tremendously ease repeatable tasks
•Creation of PCF routes
•Service binding
•Easy scaling out of instances

32. LESSONS LEARNED
▪PaaS is no panacea for everything
•Considerable efforts to integrate with underlying infrastructure
•PCF still misses some functionality
•Built-in monitoring and alerting
•OOTB blue/green deployments
•Multi-versioned tiles
•Simple C2C networking across orgs and spaces
▪Treat development teams as customers and collect as much feedback as possible
▪Time for onboarding new teams has initially been underestimated

33. LESSONS LEARNED
▪Provisioning of a centrally managed platform drastically improves velocity and
stability of components
▪Isolation and decoupling of components and teams is essential for parallel
development
▪A Microservices-oriented architecture significantly changes the way that our
teams develop and collaborate together –plan on an organizational level!

34. THANK YOU