DECODING JAVA THREAD DUMPS: MASTER THE ART OF ANALYSIS

Are you ready to unlock the secrets hidden within Java thread dumps? Join us for a hands-on session where we'll delve into effective troubleshooting patterns to swiftly identify the root causes of production problems. Discover the right tools, techniques, and best practices while exploring *real-world case studies of major outages* in Fortune 500 enterprises. Engage in interactive lab exercises where you'll have the opportunity to troubleshoot thread dumps and uncover performance issues firsthand. Join us and become a master of Java thread dump analysis!

1. Shooting the troubles: Crashes, Slowdowns, CPU spikes Ram Lakshmanan Architect: yCrash

2. https://blog.fastthread.io/2018/12/13/how-to-troubleshoot-cpu-problems/ Troubleshooting CPU spike

3. Step 1: Confirm ‘top’ tool is your good friend

4. Step 2: Identify Threads top –H –p {pid}

5. Step 3: Identify Lines of code

6. How to take Thread Dumps? 9 options https://blog.fastthread.io/how-to-take-thread-dumps-7-options/

7. 1. GC Log 10. netstat 12. vmstat 2. Thread Dump 9. dmesg 3. Heap Dump (optional) 360-degree data 6. ps 8. Disk Usage 5. top 13. iostat 11. ping 14. Kernel Params 15. App Logs 16. metadata 4. Heap Substitute 7. top -H Open-source script : https://github.com/ycrash/yc-data-script ./ yc –p <PROCESS_ID>

8. 2019-12-26 17:13:23 Full thread dump Java HotSpot (TM) 64-Bit Server VM (23.7-b01 mixed mode): "Reconnection-1" prio =10 tid =0x00007f0442e10800 nid =0x112a waiting on condition [0x00007f042f719000] java.lang.Thread.State : WAITING (parking) at sun.misc.Unsafe.park (Native Method) - parking to wait for <0x007b3953a98> (a java.util.concurrent.locks.AbstractQueuedSynchr ) at java.util.concurrent.locks.LockSupport.park (LockSupport.java:186) at java.lang.Thread.run (Thread.java:722) : : 1 2 3 1 Timestamp at which thread dump was triggered 2 JVM Version info 3 Thread Details - <<details in following slides>> Anatomy of thread dump "InvoiceThread-A996" prio =10 tid =0x00002b7cfc6fb000 nid =0x4479 runnable [0x00002b7d17ab8000] java.lang.Thread.State : RUNNABLE at com.buggycompany.rt.util.ItinerarySegmentProcessor.setConnectingFlight(ItinerarySegmentProcessor.java:380) at com.buggycompany.rt.util.ItinerarySegmentProcessor.processTripType0(ItinerarySegmentProcessor.java:366) at com.buggycompany.rt.util.ItinerarySegmentProcessor.processItineraryByTripType(ItinerarySegmentProcessor.java:254) at com.buggycompany.rt.util.ItinerarySegmentProcessor.templateMethod(ItinerarySegmentProcessor.java:399) at com.buggycompany.qc.gds.InvoiceGeneratedFacade.readTicketImage (InvoiceGeneratedFacade.java:252) at com.buggycompany.qc.gds.InvoiceGeneratedFacade.doOrchestrate (InvoiceGeneratedFacade.java:151) at com.buggycompany.framework.gdstask.BaseGDSFacade.orchestrate (BaseGDSFacade.java:32) at com.buggycompany.framework.gdstask.BaseGDSFacade.doWork (BaseGDSFacade.java:22) at com.buggycompany.framework.concurrent.BuggycompanyCallable.call (buggycompanyCallable.java:80) at java.util.concurrent.FutureTask$Sync.innerRun (FutureTask.java:334) at java.util.concurrent.FutureTask.run (FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:615) at java.lang.Thread.run (Thread.java:722)

9. "InvoiceThread-A996" prio =10 tid =0x00002b7cfc6fb000 nid =0x4479 runnable [0x00002b7d17ab8000] java.lang.Thread.State : RUNNABLE at com.buggycompany.rt.util.ItinerarySegmentProcessor.setConnectingFlight(ItinerarySegmentProcessor.java:380) at com.buggycompany.rt.util.ItinerarySegmentProcessor.processTripType0(ItinerarySegmentProcessor.java:366) at com.buggycompany.rt.util.ItinerarySegmentProcessor.processItineraryByTripType(ItinerarySegmentProcessor.java:254) at com.buggycompany.rt.util.ItinerarySegmentProcessor.templateMethod(ItinerarySegmentProcessor.java:399) at com.buggycompany.qc.gds.InvoiceGeneratedFacade.readTicketImage (InvoiceGeneratedFacade.java:252) at com.buggycompany.qc.gds.InvoiceGeneratedFacade.doOrchestrate (InvoiceGeneratedFacade.java:151) at com.buggycompany.framework.gdstask.BaseGDSFacade.orchestrate (BaseGDSFacade.java:32) at com.buggycompany.framework.gdstask.BaseGDSFacade.doWork (BaseGDSFacade.java:22) at com.buggycompany.framework.concurrent.BuggycompanyCallable.call (buggycompanyCallable.java:80) at java.util.concurrent.FutureTask$Sync.innerRun (FutureTask.java:334) at java.util.concurrent.FutureTask.run (FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:615) at java.lang.Thread.run (Thread.java:722) 1 2 3 4 5 6 7 1 Thread Name - InvoiceThread-A996 2 Priority - Can have values from 1 to 10 3 Thread Id - 0x00002b7cfc6fb000 – Unique ID assigned by JVM. It's returned by calling the Thread.getId () method. 4 Native Id - 0x4479 - This ID is highly platform dependent. On Linux, it's the pid of the thread. On Windows, it's simply the OS-level thread ID within a process. On Mac OS X, it is said to be the native pthread_t value. 5 Address space - 0x00002b7d17ab8000 - 6 Thread State - RUNNABLE 7 Stack trace -

10. Case Study: Troubleshooting CPU spike Major Trading application Analysis Report: https://tinyurl.com/wzs8kpb

11. 6 thread states RUNNABLE TERMINATED NEW TIMED_WAITING Thread.sleep (10); WAITING 03 02 01 06 05 public void synchronized getData() { makeDBCall(); } BLOCKED 04 Thread 1: Runnable Thread 2: BLOCKED wait(); Thread 1: Runnable

12. Case Study: Troubleshooting unresponsive app Analysis Report: https://tinyurl.com/wq95weo Travel App processes 70% N. America overseas booking TrafficJam Pattern

13. 9 types - OutOfMemoryError Java heap space https://blog.gceasy.io/2015/09/25/outofmemoryerror-beautiful-1-page-document/ 01 GC overhead limit exceeded 02 Requested array size exceed VM limit 03 Permgen space 04 Metaspace 05 Unable to create new native thread 06 Kill process or sacrifice child 07 reason stack_trace_with_native method 08 java.lang.OutOfMemoryError : <type> Direct Buff Memory 09

14. Case Study: OOMError : Unable to create new native thread One of world’s larges middleware app Analysis Report: http://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTcvMDMvMTQvLS10aHJlYWREdW1wLTIudHh0LS0xMi0yOC0zMw==&s=t

15. Java Heap Physical memory Physical memory Process-1 Process-2 Key: Threads are created outside heap, metspace threads Solution: Fix thread leak Increase the Thread Limits Set at Operating System( ulimit –u) Reduce Java Heap Size Kills other processes Increase physical memory size Reduce thread stack size (- Xss ). Note: can cause StackOverflowError OOM: Unable to create new native thread metaspace Java Heap metaspace - Xmx - XX:MaxMetaspaceSize - Xmx - XX:MaxMetaspaceSize

16. Case Study: Troubleshooting Microservices/Big data app Major Financial institution in N. America Analysis Report: https://tinyurl.com/yywdmvyy Same RSI Pattern

17. Case Study: Deadlock Open-Source apache library Analysis Report: Deadlock in Apache pdfbox library - yCrash Answers Deadlock Pattern

18. Unresponsiveness in backend (Good use case of Flame graph) Analysis Report: https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMjIvMDcvMzEvdGhyZWFkX2thc3RsZV8yNjA3MjIudHh0LS03LTMwLTMzLS0xNi0zMy0zNg==&&s=t What’s typically reported in APM? AWS Cloud watch + yCrash = Monitoring + RCA – yCrash All roads lead to Rome Pattern

19. HTTP 502 in AWS – EBS Analysis Report: Troubleshooting HTTP 502 bad gateway in AWS EBS  – yCrash Kernel Logs

20. EBS Architecture

21. Clue: Nginx Error

22. 1. GC Log 10. netstat 12. vmstat 2. Thread Dump 9. dmesg 3. Heap Dump (optional) 360-degree data 6. ps 8. Disk Usage 5. top 13. iostat 11. ping 14. Kernel Params 15. App Logs 16. metadata 4. Heap Substitute 7. top -H Open-source script : https://github.com/ycrash/yc-data-script

23. Degradation: Porting datacenter  public cloud Major cloud provider Load Average

24. 1. GC Log 10. netstat 12. vmstat 2. Thread Dump 9. dmesg 3. Heap Dump (optional) 360-degree data 6. ps 8. Disk Usage 5. top 13. iostat 11. ping 14. Kernel Params 15. App Logs 16. metadata 4. Heap Substitute 7. top -H Open-source script : https://github.com/ycrash/yc-data-script

25. Thank You my Friends! Ram Lakshmanan [email protected] @tier1app linkedin.com /company/ gceasy This deck will be published in: https://blog.fastthread.io