Entropy, Software Quality, and Innovation (presented at Princeton Plasma Physics Laboratory)

In his book, The Nature of the Physical World, Sir Arthur Eddington commented that “We have to appeal to the one outstanding law — the second law of thermodynamics — to put some sense into the world.” This sense-making goes beyond the physical world, too. Entropy is also essential in the fields of
information and communication theory.

During this lecture for the Princeton Plasma Physics Laboratory, lecturer Andrea Goulet discussed the application of entropy-related concepts in two communication systems: software and collaborative teams. She examined how concepts that help us understand systemic statistical disorder, such as ergodic systems, Lyapunov exponents, Kolmogorov-Sinai entropy, and Shannon-entropy can help us optimize for both software quality and innovation. She also provided several domain-specific models: Lehman’s Laws and Conway’s Law for software, as well as new models from her own research that relate to entropy and innovation.

Entropy helps us understand the world and achieve great things. There is an underlying beauty in its principles that we can use to advance scientific discovery. When we understand the subtleties related to balancing surprise and structure, we increase our chances for effective collaboration and finding novel solutions to complex problems.

1. ©2024 Andrea Goulet. All Rights Reserved.
Entropy, Software
Quality, and Innovation
Princeton Plasma Physics Laboratory
PRESENTED BY: ANDREA GOULET
Entropy, Software
MAY 9, 2024

2. ©2024 Andrea Goulet. All Rights Reserved.
About the Presenter
Andrea Goulet is on a mission to embed empathy into the
software industry to help humanity collaborate on the
complex problems we face today. She is a sought-after
international keynote speaker, experienced software
entrepreneur, and award-winning industry leader. Her
expertise centers on applying empathy as a technical skill
to improve organizational effectiveness and modernize
legacy and mission-critical software systems.
Through her online courses, Andrea has taught over
75,000 students how to level up their empathy and
communication skills to create better software. She is the
author of the forthcoming book, Empathy-Driven Software
Development, and the founder of
Empathy in Tech and
Legacy Code Rocks, two an online communities where
code and compassion connect.
●LinkedIn: https://www.linkedin.com/in/andreamgoulet
●Email: [email protected]
●Website: https://www.andreagoulet.com

3. ©2024 Andrea Goulet. All Rights Reserved.
Abstract
In his book, The Nature of the Physical World, Sir Arthur Eddington commented that “We have to
appeal to the one outstanding law — the second law of thermodynamics — to put some sense into the
world.” This sense-making goes beyond the physical world, too. Entropy is also essential in the fields of
information and communication theory.
During this lecture, we’ll discuss the application of entropy-related concepts in two communication
systems: software and collaborative teams. We’ll examine how concepts that help us understand
systemic statistical disorder, such as ergodic systems, Lyapunov exponents, Kolmogorov-Sinai entropy,
and Shannon-entropy can help us optimize for both software quality and innovation. We will also look at
domain-specific models: Lehman’s Laws and Conway’s Law for software, as well as new models from
the presenter’s own research that relate to entropy and innovation.
Entropy helps us understand the world and achieve great things. There is an underlying beauty in its
principles that we can use to advance scientific discovery. When we understand the subtleties related
to balancing surprise and structure, we increase our chances for effective collaboration and finding
novel solutions to complex problems.

4. ©2024 Andrea Goulet. All Rights Reserved.
“We have to appeal to the one
outstanding law — the second
law of thermodynamics — to put
some sense into the world.”
Sir Arthur Eddington
The Nature of the Physical World, 1929

5. ©2024 Andrea Goulet. All Rights Reserved. ©2024 Andrea Goulet. All Rights Reserved.
Entropy is both beautiful…

6. ©2024 Andrea Goulet. All Rights Reserved.
…and frustrating

7. ©2024 Andrea Goulet. All Rights Reserved.
Knowledge
Cascade of Science
Communication
1.Relevance
why does this matter?
2.Accessibility
general understanding
3.Transferability
associative learning
4.Quality Assurance
shared learning with nuance
Jensen, Eric A., and Alexander Gerber. “Evidence-Based Science Communication.” Frontiers in Communication 4 (January 23, 2020). https://doi.org/10.3389/fcomm.2019.00078.

8. ©2024 Andrea Goulet. All Rights Reserved.
“When one meets the concept of
entropy in communication theory,
he has a right to be rather excited—a
right to suspect that one has hold of
something that may turn out to be
basic and important.”
Warren Weaver
The Mathematical Theory of Communication, 1963

9. ©2024 Andrea Goulet. All Rights Reserved. ©2024 Andrea Goulet. All Rights Reserved.
Shannon, Claude Elwood, and Warren Weaver. The Mathematical Theory of Communication. Illini Books edition, 1963. Urbana: Univ. of Illinois Press, 1949.
SOURCE TRANSMITTER
CHANNEL
RECEIVER DESTINATION
MESSAGE
(encoding)
MESSAGE
(decoding)
What is entropy in a communication system?
“…[logarithmic measure] of various probabilities involved—those of getting
to certain stages in the process of forming messages, and the probabilities
that, when in those stages, certain symbols be chosen next.”
Claude Shannon: Information Theory
NOISE

10. ©2024 Andrea Goulet. All Rights Reserved.
INFERENCE EXPRESSION
INFORMATION CHANNEL
RELATIONSHIP CHANNEL
SENSES INTERPRETATION
ENCODING DECODINGREAPPRAISAL
INTENTIONINFERENCE EXPRESSION
Warren Weaver: Communication Theory
A.Technical - How accurately can the symbols of communication be transmitted? (Information Theory)
B.Semantic - How precisely do the transmitted symbols convey the desired meaning?
C.Effectiveness - How effectively does the received meaning affect conduct in the desired way?
©2024 Andrea Goulet. All Rights Reserved.
Shannon, Claude Elwood, and Warren Weaver. The Mathematical Theory of Communication. Illini Books edition, 1963. Urbana: Univ. of Illinois Press, 1949.

11. ©2024 Andrea Goulet. All Rights Reserved.
Non-Linear System
Entropy*
• Ergodic Systems
• Lyapunov Exponents
• Kolmogorov-Sinai (metric
entropy)
• Shannon (information entropy)
* The application of these concepts in this presentation are intended as abstractions to aid in conceptual accessibility and transferability. More discourse and
research is likely needed to explore the effectiveness of them as direct mappings.

12. ©2024 Andrea Goulet. All Rights Reserved.
Shannon entropy
(information)
HOW CAN WE PREDICT UNCERTAINTY IN
A MESSAGE SENT BETWEEN POINTS?
●Information = freedom of choice in selecting a
message at a source
●Redundancy = structure of message governed
by system
●Signal = channel capacity
●Coding = signal/information conversion
●Noise = average uncertainty in a message
when signal is known

13. ©2024 Andrea Goulet. All Rights Reserved.
Ergodic Systems
WHAT’S THE LONG-TERM BEHAVIOR
OF A DYNAMIC SYSTEM?
•Observing a single trajectory over
time allows you to infer the statistical
properties of the whole system.
• Statistical behavior of the system is
independent of initial conditions.
Trajectory (collaboration & software quality)
How fast are we identifying problems and
deploying effective solutions?

14. ©2024 Andrea Goulet. All Rights Reserved.
Lyapunov Exponents
HOW FAST DO POINTS MOVE
AWAY FROM EACH OTHER?
Humans = collaboration
Software = developer productivity*
CHAOTIC - fast frustration
STABLE - slow frustration
* There are several new methods to measure developer productivity: DORA, SPACE, and most recently DevEx (cognitive load, feedback loops & flow state)

15. ©2024 Andrea Goulet. All Rights Reserved.
Lyapunov Exponents in Collaboration
●Experience
●Knowledge
●Cognitive Processing
●Values
●Motivation
●Sensitivity
●Power
●Regulation
●Access
●Trust
●Language
●Affect
●Ability
●Energy
●Culture
●Belonging

16. ©2024 Andrea Goulet. All Rights Reserved.
Lyapunov Exponents in Software
●Complexity
●Duplication
●Churn
●Test Coverage
●Dependencies
●Tech Stacks
●Compatibility
●Domain Models
●Customer Knowledge
●Tooling
●Documentation
●Bureaucracy
●Org Goals
●System Access
●Security
●Legibility

17. ©2024 Andrea Goulet. All Rights Reserved.
Kolmogorov-Sinai
entropy (metric)
HOW MUCH DISORDER IS IN A SYSTEM OVER TIME?
Metric entropy for the system can be determined by looking at the sum of positive Lyapunov Exponents.
Chaotic LE → Chaotic system
Stable LE → Stable system

18. ©2024 Andrea Goulet. All Rights Reserved.
Balancing
Structure &
Surprise

19. ©2024 Andrea Goulet. All Rights Reserved.
Hey, Scott.
Got a sec?

20. ©2024 Andrea Goulet. All Rights Reserved.
Finding & Fixing Frustration Points
Andrea Scott
Negotiated
LE Score*
(0 = stable
10 = chaotic)
Sales Software 8
Extrovert Introvert 6
Maker Mender 5
Context Switch Deep Focus 10
Top-Down Bottom-Up 7
Female Male 3
1.Decompose system frustration
into its Lyapunov Exponents
2.Isolate the highest contributing
factor
3. Explore underlying reasons of
each person’s frustration
(empathy)
4.Co-create a protocol to add
redundancy (system defined
message structure)
*These scores are highly local, subjective, and relative, since they are determined collaboratively by the people who are directly involved in the frustrating situation. Therefore, they may not align with population-level averages. Inductive reasoning determines a pattern from a population, but it
cannot be used to infer that an individual within that population conforms to the average.

21. ©2024 Andrea Goulet. All Rights Reserved.
Co-Created Structure Aids Collaboration

22. ©2024 Andrea Goulet. All Rights Reserved.
Entropy &
Innovation

23. ©2024 Andrea Goulet. All Rights Reserved.
A New Paradigm
of Empathy
1.Collect - Build your personal
capacity, emotional regulation, and
self-awareness
2.Connect - Collect quality data and
attune to another perspective
3.Communicate - Anticipate
miscommunications and optimize
accordingly.

24. ©2024 Andrea Goulet. All Rights Reserved.
Innovation Emerges from
Productive Conflict
●Comprehension - effort is made to clarify
misunderstandings
●Shared Purpose - introduces a boundary condition that
enables us to use entropy principles effectively
●Prosocial Behavior - personal wellbeing and system
goals are regularly balanced
●Self-Organization - individuals feel empowered to adapt
and optimize their local conditions
●Experimentation - new methods can be tested and
learnings are shared
●Co-Creation - people create together what no single
person could create alone.

25. ©2024 Andrea Goulet. All Rights Reserved.
Entropy &
Code Quality

26. ©2024 Andrea Goulet. All Rights Reserved.
Lehman’s Laws Of
Software Evolution
●S-Type: system is static as both the
problem and solution are clearly defined. (ex: compute area of circle)
●P-Type: system models a real-world
problem. As the world changes, the system must change to remain valid. (ex: weather prediction)
●E-Type: system is embedded in the
world it models, making it the most complex type (ex: air traffic control)
Lehman, M.M., J.F. Ramil, P.D. Wernick, D.E. Perry, and W.M. Turski. “Metrics and Laws of Software Evolution-the Nineties View.” In Proceedings Fourth International Software Metrics
Symposium, 20–32. Albuquerque, NM, USA: IEEE Comput. Soc, 1997. https://doi.org/10.1109/METRIC.1997.637156.
Lehman, M.M., J.F. Ramil, P.D. Wernick, D.E. Perry, and W.M. Turski. “Metrics and Laws of Software Evolution-the Nineties View.” In Proceedings Fourth International Software Metrics
1.Continuing Change - Systems must evolve. If there is not continuous adaptation, the system will become less satisfactory.
2.Increasing Complexity - Entropy matters. Complexity increases unless work is done to maintain or reduce it.
3.Self-Regulation - Systems are dynamic. We can use statistically
determinable trends to self-regulate if entropy is stable.
4.Conservation of Organizational Stability - The stability between
businesses and software systems are strongly linked.
5.Conservation of Familiarity - People need to understand the
system they are working on. Confusion increases complexity.
6.Continuing Growth - Systems grow to satisfy the needs of its users. As functionality increases, so does size and complexity.
7.Declining Quality - Without concerted and rigorous efforts to maintain a system, its quality will decline over time.
8.Feedback System - Input from a variety of stakeholders is
needed, which requires processes that enable feedback loops.

27. ©2024 Andrea Goulet. All Rights Reserved.
Conway’s Law
Any organization that designs a system will inevitably
produce a design whose structure is a copy of the
organization's communication structure.
Communication structures (ex: protocols) have a direct
impact on software maintainability. When the entropy
balance of communication systems is not maintained, this
will inevitably lead to problems in the code, too.
Conway, Melvin E. “How Do Committees Invent?” Datamation Magazine, April 1968, 28–31. https://www.melconway.com/Home/Committees_Paper.html.

28. ©2024 Andrea Goulet. All Rights Reserved.
Key Takeaways

29. ©2024 Andrea Goulet. All Rights Reserved.
Key Takeaways
•Principles of entropy are essential to both human
collaboration and software quality
•Frame frustration points as Lyapunov Exponents.
Co-create structure through productive conflict.
•Reducing entropy requires effort. You have to
constantly work on the system to minimize disorder.
•Too much cohesion can be just as detrimental as too
much chaos. Balance structure and surprise.

30. ©2024 Andrea Goulet. All Rights Reserved.
Appendix

31. ©2024 Andrea Goulet. All Rights Reserved.
Follow Up to Discussion
After the presentation, a Q&A session took place. During that time, a key question
emerged: What definition of entropy should be used? This appendix provides
additional context to help answer that question in greater detail.
Entropy as a concept started in the field of thermodynamics. Over the course of
150 years, many different types of entropy have been named based on specific
contexts. The following slides provide a historical overview of these terms,
including the date they were introduced and a brief synopsis.
The author used ChatGPT-4o to aid her analysis. You can find more detail about
specific contributors, computations, and citations in her chat, which is available
here:
https://chatgpt.com/share/11e3eadb-2a0b-4a29-b5e2-99dcdd4a8faf

32. ©2024 Andrea Goulet. All Rights Reserved.
General Entropy
Entropy is a mathematical concept that quantifies uncertainty, or the amount of statistical
disorder within a system. Key characteristics across most types of entropy include:
1.Logarithmic functions
2. Probability distributions
Entropy plays a key role in understanding:
• Physical states
• Dynamical systems
• Information systems
• Networks
• Social systems
• Software systems, including machine learning

33. ©2024 Andrea Goulet. All Rights Reserved.
Historical Overview
Type of Entropy Year Context
2nd Law of
Thermodynamics
(Clausius-entropy)
1865
Entropy as a term was coined by Rudolf Clausius to describe mathematically
how energy is dispersed in a thermodynamic system, which was theorized in
the Carnot cycle. This kind of entropy is later formalized as the 2nd law of
thermodynamics.
Boltzmann-entropy 1877
Uses system mechanics to mathematically relate microscopic states of a
thermodynamic system to its macroscopic properties.
Gibbs-entropy 1902
Generalizes Boltzmann’s work so entropy is described by a probability
distribution and can be applied to more complex systems.
von Neumann-entropy 1927 Measures entropy within quantum systems.
Shannon-entropy
(Information entropy)
1948
Establishes information theory based on mathematical models of entropy in
the context of communication systems.

34. ©2024 Andrea Goulet. All Rights Reserved.
Historical Overview (cont.)
Type of Entropy Year Context
Kolmogorov-Sinai-entropy
(metric entropy)
1959
Extends entropy to explain the behavior of chaotic systems by measuring
the amount of complexity and unpredictability of dynamical systems.
Rényi-entropy 1961
Introduces a parameter to Shannon-entropy that allows for more
generalized and flexible cases of uncertainty.
Hawking-Bekenstein
entropy
1972
Demonstrates how black holes follow the laws of thermodynamics by
developing a framework that connects quantum mechanics,
thermodynamics, and general relativity.
Social entropy 1956
Applies work from general systems theory to human ecology, providing a
theory for how entropy could be used to determine disorder and complexity
in social systems.

35. ©2024 Andrea Goulet. All Rights Reserved.
Historical Overview (cont.)
Type of Entropy Year Context
Topological entropy 1965
Builds on dynamical systems theory to measure the rate at which distinct
trajectories of the system diverge over time.
Software entropy 1980
Uses entropy to determine practices that reduce the complexity of a
software system so that it is easier to modernize and maintain.
Path entropy 2001
Determines how entropy impacts message routing through a network by
quantifying the diversity of paths.
Degree Distribution entropy 2003
Compares the complexity of different networks, which provides
mechanisms for detecting structural changes in dynamic networks.
Node entropy 2005 Measures the distribution of connections among nodes in a network.
Edge entropy 2005
Determines how the strength or capacity of a connection between nodes
influences the complexity of a system.