How to Improve Your Ability to Solve Complex Performance Problems

This talk is really about problem solving. It’s about how we think about problems and how we resolve those problems in a deeply technical context. The main goal of the talk is the relay the lessons learned from a couple of decades working with and observing some of the best performance troubleshooters in the world.

The talk will be broken into 3 main parts.

1. Explain the basic process we must go through to solve a complex performance problem
2. Discuss some of the main factors that can inhibit our efforts
3. Discuss some of the techniques we can apply to improve our chances, including an almost fool proof method to reach a successful outcome

Specific technical examples from large enterprise customers using relational databases (Oracle primarily) will be used to illustrate the concepts.

1. How to Improve Your Ability to Solve Complex Performance Problems Kerry Osborne Database Black Belt, Team Lead, Google

2. ‹#› whoami – Kerry Osborne Database Black Belt Team Lead @ Google Blog: kerryosborne.oracle-guy.com Twitter: @KerryOracleGuy Alumni Technical by Nature Dev->DBA->Sys Admin->Biz->Tuner->’ Biz->Author->Chief Bottle Washer-> … Entrepreneur (3 time offender) Wrote a Couple of Books Never Worked directly for Oracle But Working with Oracle Since V2

3. ‹#› whoami – really Extremely Analytical Also Creative Pragmatic Optimistic Love Working with People Hate Working with People Solved Problems for a Living I’ve Spent a Lot of Time Thinking About Thinking Interviewed a Few Thousand People

4. ‹#› Outline - In Four Movements Characteristics of Good Problem Solvers Basics of How Our Brains Work Some Approaches to Solving Problems How Can We Get Better

5. Movement 1: Characteristics of Good Problem Solvers

6. ‹#› What makes a good problem solver? Smart Hard Worker Technical Background Communication Skills All important, but none guarantee success Relaxed Optimistic Experienced Failures in the Past

7. ‹#› Smart (and a hard worker) “I divide my officers into four classes; the clever, the lazy, the industrious, and the stupid. Each officer possesses at least two of these qualities. Those who are clever and industrious are fitted for the highest staff appointments. Use can be made of those who are stupid and lazy. The man who is clever and lazy however is for the very highest command; he has the temperament and nerves to deal with all situations. But whoever is stupid and industrious is a menace and must be removed immediately!" Best Problem Solvers*

8. ‹#› Technical Background It Takes Years to Acquire Learning How to Learn is an Important Part The Harder You Work the Faster You Progress Fundamentals are Important Diversified Background is Extremely Helpful

9. ‹#› Digression – How We Learn Most Effective When Feedback is Quick - Learning to drive a car is easier than learning to drive a big boat Breaking Things (in Test) is a Great Way to Learn - Jonathan Lewis – “How to Become an Expert” - Only talked about how to create test cases

10. ‹#› Communication Skills Doesn’t matter if you’re the smartest guy in the room if you can’t convince other people.

11. Movement 2: Basics of How Our Brains Work

12. Two Modes of Thought Intuitive System 1 Automatic Analytical System 2 Requires Effort* *our brains are naturally lazy 1 + 1 = 2 19 X 35 = 665

13. ‹#› System 1 - Intuition Correct Most of the Time In Control Most of the Time Biases Lazy – Substitution, WYSIATI You Can’t Really Turn It Off - Müller-Lyer Illusion

14. ‹#› Problem Characteristics (for us) Complex Multiple Solutions Solutions Vary Benefit Cost Monetary Time to Implement Risk

15. ‹#› The large list of findings makes it not trivial to define a linear path in the implementation of said recommendations, especially since some of those recommendations, whilst at the very top of the list in terms of urgency, come with a significant impact on the system and will require extensive testing before being moved to production. Furthermore most of the top findings are correlated to each other, but due to the high number of them and the significant impact they have, it’s not possible to clearly determine the exact order under which they come into play and how much one impacts the other. The following list of recommendations is a balance between urgency and ease of implementation, slightly leaning towards urgency. Excerpt from a Performance Review

16. Movement 3: Approaches to Solving Problems

17. ‹#› Basic Problem Solving Steps Defining the Problem Gathering Data Postulating a Reason for the Problem Listing Possible Solutions Ordering the Possible Solutions Attempting the Possible Solutions

18. ‹#› Postulate: All modern computer systems are made up of many parts (smaller discrete “things” that can be measured). It follows that making computer systems run faster depends on being able to identify which of those “things” are consuming the most time. This also implies that instrumentation is critical for identifying how to make computer systems run faster. Digression – More Thoughts on Gathering Data

19. ‹#› Two Basic Approaches Intuitive & Methodical Both use the same basic steps Vary in time spent on steps Vary in order of steps

20. ‹#› Hallmarks – Intuitive Approach Jumps Ahead to Theorizing About Reason (Step 3) Pluses and Minuses - Often Misses Important Data (Step 2) - Often Less Creative in Possible Solutions (Step 4) + Willingness to Quickly Abandon an Option + Flexibility to Re-order Options

21. ‹#› Hallmarks – Methodical Approach Mental Discipline to Postpone Theorizing (Step 3) Pluses and Minuses + Problem More Well Defined (Step 1) + More Data Available (Step 2) + More Creative Solutions are Possible - Takes More Time

22. ‹#› 3 rd Option - Combined Approach Jumps Quickly to Theorizing (Step 3) Validates (or Eliminates) Usual Suspects Falls Back to Methodical if Intuitive Doesn’t Work * Recognition-Primed Decision making model (RPD) * Also known affectionately as the Tanel Method ™

23. ‹#› Results (my opinion) Combined (RPD) Usually Delivers Best Results - for reasonably experienced people Methodical Approach Better Than Intuitive More Creative Solutions Generally in Less Time (overall) Generates More Confidence Intuitive Approach Alone Not Good But Occasionally Can Be Fastest - for very experienced people - or if you’re lucky

24. Final Thoughts for the First Half

25. ‹#› There are only 3 ways to make a “thing” go faster. Speed it up Do it less Don’t do it at all * With a nod to Cary Millsap

26. Kerry Osborne [email protected] @KerryOracleGuy Blog: kerryosborne.oracle-guy.com Thank you! Let’s connect.