Engineering Effectiveness - why the time is now - Max Griffiths

DORA’s 4 key metrics have been leading us astray. Much of my exploration into Engineering Effectiveness over the last several years has led me back to some of the original recommendations and practices surrounding the 4KM where companies are still struggling with engineering discipline. In this talk I’d like to walk you through some shocking results from a survey we commissioned through Forrester on Developer Effectiveness, and how—as developers--we could be thinking differently during our day. GenAI of course has a potential role to play, and I’ll walk through some of the practical applications and benefits we’re starting to see, avoiding the hype as we go.

1. 1
© 2024 Thoughtworks, Inc. Commercial in Confidence 11
Engineering
effectiveness
Max Griffiths - October 2024
Why the time is now

2. 2
© 2024 Thoughtworks, Inc. Commercial in Confidence
How
can I do more with the same?

3. 3
© 2024 Thoughtworks, Inc. Commercial in Confidence
How
do I know Iʼm doing the most with
what I have

4. 4
© 2024 Thoughtworks, Inc. Commercial in Confidence
“Weʼre investing in tech but not seeing
value for moneyˮ
Cost to run business is too high
leading to:
●lack of trust
●micromanaging
●unreasonable asks
○corner-cutting
What are we seeing today
●Tech leadership doesnʼt have clarity
on what initiatives are costing or
what value they are delivering
●business case for risk-based work
are not landing - can't just ask for
more money
●Are my software devs working
efficiently?
●“If I had €XM to improve the engine,
where would you apply thatˮ
4© 2024 Thoughtworks | Confidential
The Business Engineering Org

5. 5
© 2024 Thoughtworks, Inc. Commercial in Confidence
Max Griffiths
?????? 20yrs consulting
?????? Started as a Java dev
?????? DevOps, Cloud, Platform
Engineering, Digital Transformation
?????? Itʼs been 2 years since Iʼve
committed production code

6. 6
© 2024 Thoughtworks, Inc. Commercial in Confidence
Max Griffiths
Now: Head of Platform Engineering
for Thoughtworks Europe

Next: Global SME for Engineering
Effectiveness

7. 7
© 2024 Thoughtworks, Inc. Commercial in Confidence
© 2023 Thoughtworks, Inc. Commercial in Confidence
Welcome
Hear some industry stats
Learn a new
method (or three)
Disambiguate some terms
Challenge my thinking
Take away something you
can try

8. 8
© 2024 Thoughtworks, Inc. Commercial in Confidence
© 2023 Thoughtworks, Inc. Commercial in Confidence
Welcome
Hear some industry stats
Learn a new
method (or three)
Disambiguate some terms
Challenge my thinking
Take away something you
can try
Stay awake, stay alive

9. 9
© 2024 Thoughtworks, Inc. Commercial in Confidence
© 2023 Thoughtworks, Inc. Commercial in Confidence
Welcome
Hear some industry stats
Learn a new
method (or three)
Disambiguate some terms
Challenge my thinking
Take away something you
can try

10. 10
© 2024 Thoughtworks, Inc. Commercial in Confidence
4KM
They have led us astray)

11. 11
© 2024 Thoughtworks, Inc. Commercial in Confidence
Low Mid High
Deployment
frequency
Monthly or less often
Between daily and
weekly
On demand
(whenever we want)
Lead time
for changes
Between a week and 6
months
Less than a week Less than an hour
MTTR Less than a week Less than a day Less than an hour
Change
failure rate
Less than 15% Less than 15% Less than 5%
A quick reminder : Lagging Indicators
DORA4 metrics recognized as a predictor of a high
performance organization.

12. 12
© 2024 Thoughtworks, Inc. Commercial in Confidence
Beyond the 4 key metrics
How the “Elitesˮ became elite
By focusing on the factors that predict high delivery
performance—a goal-oriented generative culture, a
modular architecture, engineering practices that
enable continuous delivery, and effective
leadership — we can scale deployments per
developer per day linearly or better with the number
of developers. This allows our business to move
faster as we add more people, not slow down, as is
more typically the case.
Accelerate 2017
Platform engineering capabilities
Engineering productivity enablers
Architecture enablers
Communication, culture and learning
Investing in leadership
change lead time | deployment frequency
mean time to restore MTTR | change fail %
© 2024 Thoughtworks, Inc. Commercial in Confidence 12

13. 13
© 2024 Thoughtworks, Inc. Commercial in Confidence
So?
is this (still) a legitimate concern?

14. 14
© 2024 Thoughtworks, Inc. Commercial in Confidence
Thoughtworks + Forrester Research

15. 15
© 2024 Thoughtworks, Inc. Commercial in Confidence
In 2023, Forrester surveyed 400 software
engineering decision-makers across a
range of countries and industries to
understand what their organizationsʼ
practices look like today, the challenges
they face, and the benefits effective
engineering can deliver.




(link to the report coming at the end)
The challenges are still VERY
apparent

16. 16
© 2024 Thoughtworks, Inc. Commercial in Confidence
In 2023, Forrester surveyed 400 software
engineering decision-makers across a
range of countries and industries to
understand what their organizationsʼ
practices look like today, the challenges
they face, and the benefits effective
engineering can deliver.




(link to the report coming at the end)
The challenges are still VERY
apparent

17. 17
© 2024 Thoughtworks, Inc. Commercial in Confidence
Aiming to improve engineering effectiveness,
organizations have looked to make enhancements in
their engineering processes.

But less than half of organizations have adopted
fundamental engineering practices like DevOps,
test-driven development, and cloud-native technologies.

And less than a third have adopted basic processes like
formal code reviews and continuous delivery.
Adoption Of Core
Engineering Practices Is Low

18. 18
© 2024 Thoughtworks, Inc. Commercial in Confidence
82% of respondents said their organization
plans to implement a self-service internal
developer platform within the next three years
●and 78% plan to implement a developer
portal within that timeframe.

In addition
●their organizations are looking to automate
78%
●and add release orchestration 79%) to
their CI/CD pipelines
Focus on Developer
Effectiveness

19. 19
© 2024 Thoughtworks, Inc. Commercial in Confidence
A Variety Of Practices Is Key
To Improving Engineering
Effectiveness

20. 20
© 2024 Thoughtworks, Inc. Commercial in Confidence
So?
is this (still) a legitimate concern?

21. 21
© 2024 Thoughtworks, Inc. Commercial in Confidence
So?
is this (still) a legitimate concern?
YES

22. 22
© 2024 Thoughtworks, Inc. Commercial in Confidence
Assessment frameworks

23. 23
© 2024 Thoughtworks, Inc. Commercial in Confidence
©2023 Thoughtworks, Inc. Commercial in Confidence 23
https://queue.acm.org/detail.cfm?id=3454124
SPACE

24. 24
© 2024 Thoughtworks, Inc. Commercial in Confidence
©2023 Thoughtworks, Inc. Commercial in Confidence 24
https://dxcore4.com/
dxcore4

25. 25
© 2024 Thoughtworks, Inc. Commercial in Confidence
©2023 Thoughtworks, Inc. Commercial in Confidence 25
VSM

26. 26
© 2024 Thoughtworks, Inc. Commercial in Confidence
Common value stream metrics

27. 27
© 2024 Thoughtworks, Inc. Commercial in Confidence
https://flowengineering.org/

28. 28
© 2024 Thoughtworks, Inc. Commercial in Confidence
Common impediments
to flow & their cost
Examples of common friction that are draining
productivity and block flow

Finding information - The usually “hiddenˮ cost of not
quickly finding, or not finding at all, important knowledge about
APIs, integrations & systems.


Finding information
Slow feedback
Cognitive friction
DX friction
Operating model friction
Team effectiveness
Value-add
delivery
< 30%> 70%
Overhead /
Waste
Value-add
delivery
60%
Current
state
Future
state
28

29. 29
© 2024 Thoughtworks, Inc. Commercial in Confidence
Common impediments
to flow & their cost
Examples of common friction that are draining
productivity and block flow

Finding information - The usually “hiddenˮ cost of not
quickly finding, or not finding at all, important knowledge about
APIs, integrations & systems.

DevEX friction - The cost of productionizing software so that
it is highly available. Includes drag from deployment, testing,
observability, and resiliency. Also, the cost of having to use
clunky tools, poorly designed APIs, lack of self-service
capabilities.


Finding information
Slow feedback
Cognitive friction
DX friction
Operating model friction
Team effectiveness
Value-add
delivery
< 30%> 70%
Overhead /
Waste
Value-add
delivery
60%
Current
state
Future
state
29

30. 30
© 2024 Thoughtworks, Inc. Commercial in Confidence
Common impediments
to flow & their cost
Examples of common friction that are draining
productivity and block flow

Finding information - The usually “hiddenˮ cost of not
quickly finding, or not finding at all, important knowledge about
APIs, integrations & systems.

DevEX friction - The cost of productionizing software so that
it is highly available. Includes drag from deployment, testing,
observability, and resiliency. Also, the cost of having to use
clunky tools, poorly designed APIs, lack of self-service
capabilities.

Cognitive overload / task switching - The cost of
misunderstood integrations, abstractions, and data. Cognitive
taxes create quality issues, slowing delivery significantly.


Finding information
Slow feedback
Cognitive friction
DX friction
Operating model friction
Team effectiveness
Value-add
delivery
< 30%> 70%
Overhead /
Waste
Value-add
delivery
60%
Current
state
Future
state
30

31. 31
© 2024 Thoughtworks, Inc. Commercial in Confidence
Common impediments
to flow & their cost
Examples of common friction that are draining
productivity and block flow

Finding information - The usually “hiddenˮ cost of not
quickly finding, or not finding at all, important knowledge about
APIs, integrations & systems.

DevEX friction - The cost of productionizing software so that
it is highly available. Includes drag from deployment, testing,
observability, and resiliency. Also, the cost of having to use
clunky tools, poorly designed APIs, lack of self-service
capabilities.

Cognitive overload / task switching - The cost of
misunderstood integrations, abstractions, and data. Cognitive
taxes create quality issues, slowing delivery significantly.

Slow quality feedback loops - Defects caught in pre-prod or
prod have exponentially higher cost to remediate and disrupt
the flow of value added work.


Finding information
Slow feedback
Cognitive friction
DX friction
Operating model friction
Team effectiveness
Value-add
delivery
< 30%> 70%
Overhead /
Waste
Value-add
delivery
60%
Current
state
Future
state
31

32. 32
© 2024 Thoughtworks, Inc. Commercial in Confidence
Common impediments
to flow & their cost
Examples of common friction that are draining
productivity and block flow

Finding information - The usually “hiddenˮ cost of not
quickly finding, or not finding at all, important knowledge about
APIs, integrations & systems.

DevEX friction - The cost of productionizing software so that
it is highly available. Includes drag from deployment, testing,
observability, and resiliency. Also, the cost of having to use
clunky tools, poorly designed APIs, lack of self-service
capabilities.

Cognitive overload / task switching - The cost of
misunderstood integrations, abstractions, and data. Cognitive
taxes create quality issues, slowing delivery significantly.

Slow quality feedback loops - Defects caught in pre-prod or
prod have exponentially higher cost to remediate and disrupt
the flow of value added work.

Operating model friction - Lost time and rework due to the
flow of work between product and engineering, such as poorly
specified product specs and architecture requirements and
review.


Finding information
Slow feedback
Cognitive friction
DX friction
Operating model friction
Team effectiveness
Value-add
delivery
< 30%> 70%
Overhead /
Waste
Value-add
delivery
60%
Current
state
Future
state
32

33. 33
© 2024 Thoughtworks, Inc. Commercial in Confidence
Where
do you sit in the value stream?

34. 34
© 2024 Thoughtworks, Inc. Commercial in Confidence
Collect the pain
Connect the business

35. 35
© 2024 Thoughtworks, Inc. Commercial in Confidence
©2023 Thoughtworks, Inc. Commercial in Confidence 35
Pain
points

36. 36
© 2024 Thoughtworks, Inc. Commercial in Confidence
©2023 Thoughtworks, Inc. Commercial in Confidence 36
EEBO
Engineering Effectiveness
to Business Outcomes
https://www.eebo.org/

37. 37
© 2024 Thoughtworks, Inc. Commercial in Confidence
Think
Information over data

38. 38
© 2024 Thoughtworks, Inc. Commercial in Confidence
Customer / User metrics
●Feedback such as Net Promoter Score
●Time taken for customer journey
●No of new users and repeat users
●Conversion rate at steps in the value chain
●Drop off rate
Value Delivery
●Release frequency
●Time to value
●Value delivered
Avoiding metric overload
Operational
metrics
Value
Delivery
Engagement KPIs
Business
Value

●Employee engagement
●Increase in revenue
●Cost reduction
Delivery and Engineering Excellence
●Velocity trends
●Deployment frequency
●Lead time to change
●Change fail %

●Mean time to restore
●Defect rate
●Vulnerability age

Balanced scorecard
●Distribution %
●Lead time for onboarding
●People churn
●Client satisfaction
●User surveys
People
●Happiness index
●Agile Maturity
●Gamification



Information over data
Right information to the right flight-level of the business
Business Impact

39. 39
© 2024 Thoughtworks, Inc. Commercial in Confidence
A holistic dataset is a key
foundation for a
comprehensive view into
developer efficiency and
productivity
Data
Foundation
Qualitative
Survey
Lagging
Indicators
Quantitative
Survey
Leading
Indicators
Gathering a holistic view of data through
system data along with data directly from
developer through surveys
People
Process
Technology
39
This is the key toward
driving changes across
your organization

40. 40
© 2024 Thoughtworks, Inc. Commercial in Confidence
Hypothesis driven improvement cycle
Assess
Solutions
Survey and
Assessment Data
Value Stream
Map
Value DeliverySimulate ROIDefine ValueAssess
Calculate
Build Cost /
Savings
Define Value
Metrics
Articulate Business
Value
Leading and
Lagging
Metrics
Measure
Outcomes
Iterate and Identify
New Opportunities
Identify
Hypothesis

41. 41
© 2024 Thoughtworks, Inc. Commercial in Confidence
A real example

42. 42
© 2024 Thoughtworks, Inc. Commercial in Confidence
©2023 Thoughtworks, Inc. Commercial in Confidence 42
Pain
points

43. 43
© 2024 Thoughtworks, Inc. Commercial in Confidence
Lack of standard operating model causes waste
Lack of optimal utilization of workforce
Inefficient planning and forecasting results into
implementation issues
Delivery takes longer than expected
Quality concerns limits our ability to respond to
business needs
Escalating Engineering Expenditure
Fear/Risk of change because of legacy & fragile
systems and architecture
Reduced Cost of Engineering
Not able to keep up with evolving technology
landscape
Lack of collaboration between upstream and
downstream/ Friction between Business and
Engineering leading to misalignment.
Insufficient Client engagement throughout the
SDLC
Product complexity causes change risk
Better business, engineering, and client
alignment
Too many defects post production causing bad
customer experience & business loss
Improved Quality
Unbalanced capabilities causes inefficiencies
Optimal utilization of workforce
Knowledge retention and inconsistencies issues
Not able to keep up with evolving technology
landscape
Not able to attract or retain tech Talent
Improved Talent Retention
Clearly defined tech & product roadmap
for Engineering
Better Predictability
CompanyX's market
recognition/leadership as Tech company
Insufficient tools and access hamper engineering
productivity
Standard repeatable deliveries of a high
quality
Improved requirement completeness
resulting in reduced rework
Improved product reliability and reduction
in maintenance cost
Challenges & WasteDesired Outcomes Focus areas (in no particular order)
A multi-faceted, holistic approach to address the complex dynamic of the
challenges and desired outcomes. Thoughtworks recommends 6 focus areas

Operating model &
governance

Platform engineering
Developer Experience

Enterprise testing
strategy


Scaling knowledge &
leadership


Engineering
Excellence

44. 44
© 2024 Thoughtworks, Inc. Commercial in Confidence
Escalating Engineering Expenditure
Unbalanced capabilities causes
inefficiencies
Delivery takes longer than expected.
Lack of collaboration between upstream
and downstream/ Friction between
Business and Engineering leading to
misalignment
Lack of optimal utilization of workforce
Optimal utilization of
workforce
Desired Outcome
44
© 2024 Thoughtworks, Inc. Commercial in Confidence
Challenges

Operating model &
governance


Engineering
Excellence

Engineering Intelligence
Engineering org design using
team topologies
Better DevEx with Developer
Portal using Backstage
Scaling knowledge
SDLC Operating Model
Portfolio management &
outcome based governance
Strategic Workforce Planning
Tactics
Focus areas

Scaling knowledge &
leadership

Developer Experience

45. 45
© 2024 Thoughtworks, Inc. Commercial in Confidence
Three opportunities arise for realising the value unlocked - gaining cost
competitiveness, potential revenue enhancement and repurposing workforce
Repurposing
Workforce
Revenue Enhancement
Opportunity
Increased Cost
Competitiveness
●Same amount of features delivered by smaller workforce,
opening an opportunity to repurpose a portion of workforce
●Constants - Members/ team, # Features
●Impact Variables - # Teams
●Increase in number of features (potential revenue
enhancement) delivered by same workforce
●Constants - # Teams, Members/ team
●Impact Variables - # Feature, Revenue for portfolio
●Same amount of features delivered by smaller workforce,
thereby reducing the cost/ feature. The remaining could be
assigned to develop features in other portfolios
●Constants - Members/ team, # Features
●Impact Variables - Cost/ Feature, Cost/ Sprint, # Teams
Illustrative
Teams with current
Engineering
Practices
Teams using
EE
Repurposed
Employees
As-Is
Team 1Team ..Team ..Team 17
80 Features
delivered/ year
17 teams
To-Be
Team 1Team ..Team 12
Repurposed
Employees
80 Features
delivered/ year
12 teams
Illustrative
Teams with current
Engineering
Practices
Teams using
EE
As-Is
Team 1Team ..Team ..Team 17
80 Features
delivered/ year
AUD 18 Mn
Revenue/ year
17 teams
To-Be
Team 1Team ..Team ..Team 17
100 Features
delivered/ year
AUD 21 Mn
Revenue/ year
17 teams
Illustrative
Teams with current
Engineering
Practices
Teams using
EE
Assigned to
other
portfolios
As-Is
Team 1Team ..Team ..Team 17
80 Features
delivered/ year
AUD 697
cost/feature
17 teams
To-Be
Team 1Team ..Team 10Team 17
80 Features
delivered/ year
AUD 558
cost/feature
12 teams
+ 5 teams assigned
to other portfolios
45
© 2024 Thoughtworks, Inc. Commercial in Confidence

46. 46
© 2024 Thoughtworks, Inc. Commercial in Confidence
Execution of the initiatives to reduce cost of engineering can potentially generate
almost 8X returns within 5 years; with payback period of only 16 months
Disclaimer - These hypothesis is based on our experience of working with multiple clients, the actual range of improvement would depend on various systemic constraints
Illustrative
Key Metrics
Annual Cost Savings 25%
Payback Period 16 Months
Valued Unlocked as X times
Investments in 5 years
8X
Improve in Cycle Time 30%
Confidence Index of 60%
Illustrative benefits of reducing cost of engineering

47. 47
© 2024 Thoughtworks, Inc. Commercial in Confidence
Platform Engineering
Capabilities
DevEx and Productivity
Accelerators
Enterprise Testing Strategy
and Enablement
Engineering Excellence
Scaling Knowledge
and Leadership
Operating Model Alignment
and Governance
Product and Engineering Value Stream and Impact Analysis
Effective
Autonomy
Aligned
Engineering Effectiveness Interconnect
Looking into one area alone is insufficient
Organizational enablers optimized for engineering transformation
Communication, culture, strategy, OCM Upskilling & enablement program Talent management & recruitment
Generative AI

48. 48
© 2024 Thoughtworks, Inc. Commercial in Confidence
Our experience with GenAI

49. 49
© 2024 Thoughtworks, Inc. Commercial in Confidence
Maintenance DeploymentTesting DevelopmentDesignAnalysis PlanningResearch
●User research
●Market research
●Regulatory research
●Requirement analysis
●Security and compliance
●UI & process analysis
●Coding and implementation
●Unit and integration testing
●API development
●Data annotation & pipelines
●Project planning
●Data & migration planning
●Security & risk planning
●User centric design
●Policy and procedure design
●Solution Architecture
●Database design
●Functional test, E2E tests
●Load testing
●Compliance testing
●Deployment planning
●Monitoring and observability
●Automated deployment
●RCA and data
processing
●Bug fixing and
testing
●Technical debt
addressal
Focus now on quick wins & high leverage areas of the SDLC
Quick Win High Leverage

50. 50
© 2024 Thoughtworks, Inc. Commercial in Confidence
Understanding the risks
Security
vulnerabilities
Proliferation of
insecure code, and
exploitation of
generated
dependencies.
IP infringement
Unauthorized use of
intellectual property
rights from prompt
responses.
Data security
violations
Breaches of
corporate or
customer
information. Fear of job
displacement
AI replacing or
altering human
jobs.
False sense of
security
Usersʼ
complacency
increases once they
have had some
successes with AI
assistance.
Biased outputs
AI generating
biased inputs
during product
development.

51. 51
© 2024 Thoughtworks, Inc. Commercial in Confidence
You can code faster but can
you ship faster?
Example metrics to monitor:
●DORA & Related
○Cycle time
○Deployment frequency
○MTTD
○MTTR
○Deloy failure rates
○Reliability
●Test failure rates across stages
●Test environment queue depth
© 2023 Thoughtworks | Confidential
Capacity boost by GenAI
Current capacity
Approved features
Manual steps or shared environments
If you havenʼt invested in your engineering & cloud
platform, you wonʼt get the full value of AI investments
Investments in engineering platforms will be amplified.
With more code being written faster, it will be critical to have
highly functional, self-service engineering platforms capabilities
to enable rapid build, test, deployment and issue resolution to
take full advantage of productivity gains.
Organizations that have underinvested in platform engineering
may see greater contention on shared build and test
infrastructure and may not see throughput gains materialize as
faster time to market.

52. 52
© 2024 Thoughtworks, Inc. Commercial in Confidence
Focus on accelerating
product engineering flow
With more code being written faster, product managers will under
pressure to get more features specified in less time.
Like we with witnessed with the advent of agile delivery, some
product organizations may need to add capacity or upskill to
ensure a more high performing delivery engine has sufficient
backlog of well-specified, customer value-add stories epics and
stories. Engineering manager can expect
Example metrics to monitor:
●Feature lead time
●Story bounce back rates
●etc
© 2023 Thoughtworks | Confidential
Wasted capacity
Standard capacity
Ready backlog
Bottleneck on product backlog
GenAI Efficiency Expands
Potential Throughput
Product Managers

53. 53
© 2024 Thoughtworks, Inc. Commercial in Confidence
Engineering practices still matter, if not more.
Good practices help
deal with bottlenecks
from higher coding
throughput.
GenAI amplifies your
status quo - the good
gets even better, but
the bad can get worse.
Good practices
mitigate the GenAI
risks, and help manage
the quality of more
code.
PotentialWasted capacity
If you can code faster,
can you ship faster?
If you can code faster,
can you fill the
backlog faster?
If you can produce more
code, can you also keep
your technical debt in
check?
If you can produce more
features, how are you
tracking their value?

54. 54
© 2024 Thoughtworks, Inc. Commercial in Confidence
Summarising

55. 55
© 2024 Thoughtworks, Inc. Commercial in Confidence
Summarising
55
Why now
How to start
Prove with data
Engineering Practices
Information over data
Prioritise for the business
Everyone is under pressure to cut costs and show that we are being efficient & effective
What can you do today that might uncover waste in the value stream, without the need to
request new investment. Use a combination of lightweight assessments and involve the right
people
Different types of data are needed to build a picture of where to focus your next optimisation
initiative
Regardless of what the data shows, it's near-guaranteed that benefits will come from better
engineering practices. These require discipline, and prioritising.
Building the business case for change: when sharing outwards, ensure itʼs pitched at the right
level
Aspire for ROI calculations as much as possible. Do you have product representation across
your internal services?
GenAI for SDLC
Use with strong intention and utmost caution: without these youʼre more likely to contribute to
problems than solutions

56. 56
© 2024 Thoughtworks, Inc. Commercial in Confidence
Thank You
Max Griffiths
Head of Platform Engineering - Europe
[email protected]
56
Scan to download
Complimentary
Forrester + Thoughtworks Report