How to Troubleshoot 9 Types of OutOfMemoryError Session

Even though at surface level ‘java.lang.OutOfMemoryError’ appears as one single error; underlyingly there are 9 types of OutOfMemoryError. Each type of OutOfMemoryError has different causes, diagnosis approaches and solutions. This session equips you with the knowledge, tools, and techniques needed to troubleshoot and conquer OutOfMemoryError in all its forms, ensuring smoother, more efficient Java applications.

1. 1

2. Does Java process memory utilization go beyond – Xmx ? 2 – XX:MaxMetaspaceSize - Xmx Young Old Metaspace Threads JNI misc GC Direct Buff Code Cache Heap Memory Native Memory Java Process Memory - XX:MaxDirectMemorySize

3. 3 Demo OutOfMemoryError

4. 4 How to know the type of OutOfMemoryError ? java.lang.OutOfMemoryError : <type>

5. 5 Java Heap Space 1 Metaspace Threads JNI misc GC Direct Buff Code Cache Heap Memory Native Memory JVM Memory Most common type - Xmx

6. 6 GC Behavior of a Healthy Application - Full Garbage Collection Event

7. 7 GC Behavior of Acute Memory Leak - Full Garbage Collection Event

8. 8 GC Behavior when there is a Memory Leak - Full Garbage Collection Event

9. 9 How to study GC Behavior?  -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:<file-path> Till Java 8 -Xlog:gc*:file=<file-path> From Java 9 Enable GC logs (always) Almost zero overhead

10. 10 GCeasy GC Log Analysis Demo

11. 11 How to capture Heap Dumps? 8 options: https://blog.heaphero.io/2017/10/13/how-to-capture-java-heap-dumps-7-options/

12. 1. GC Log 10. netstat 12. vmstat 2. Thread Dump 9. dmesg 3. Heap Dump 6. ps 8. Disk Usage 5. top 13. iostat 11. ping 14. Kernel Params 15. App Logs 16. metadata 4. Heap Substitute 7. top -H 12 Open-source script: https://github.com/ycrash/yc-data-script 360° Troubleshooting artifacts ./ yc –p <PROCESS_ID>

13. 13 HeapHero How to analyze Heap Dump?

14. 14 OutOfMemoryError : Java Heap Space Causes Memory leak due to buggy code Increase in Traffic Volume Solutions Fix Memory Leak in the code Increase heap size (- Xmx ) Artifacts Garbage Collection Log Heap Dump Tools GCeasy (GC log analysis) HeapHero Eclipse MAT JVisualVM

15. 15 GC overhead limit exceeded 2 Metaspace Threads JNI misc GC Direct Buff Code Cache Heap Memory Native Memory JVM Memory - Xmx

16. 16 OutOfMemoryError : GC overhead limit exceeded Diagnosis: Same as ‘Java Heap Space’  Java process is spending more than 98% of its time doing garbage collection and recovering less than 2% of the heap and has been doing so far the last 5 consecutive garbage collections

17. 17 Requested array size exceeds VM limit 3 Young Old Metaspace Threads JNI misc GC Direct Buff Code Cache Heap Memory Native Memory JVM Memory

18. 18 Demo OutOfMemoryError : Requested array size exceeds VM limit

19. 19 OutOfMemoryError : Requested array size exceeds VM limit Causes Allocated array larger than heap size Solutions Fix code which allocates large array size Increase heap size (- Xmx ) Artifacts Application Log or std error Tools

20. 20 Metaspace 4 Young Old Threads JNI misc GC Direct Buff Code Cache Heap Memory Native Memory JVM Memory

21. 21 Metaspace Memory Leak - Full Garbage Collection Event

22. 22 Demo OutOfMemoryError : Metaspace

23. 23 How to study Metaspace Behavior?   java { app_name } - verbose:class Till Java 8 java { app_name } - Xlog:class+load =info:<filename> From Java 9

24. 24 OutOfMemoryError : Metaspace Causes Creating large number of Dynamic classes (Java Reflection, Groovy type of scripting languages) Loading large number of Classes (3 rd party libraries/frameworks) Loading large number of Classloaders Solutions Fix Memory Leak code Increase - XX:MetaspaceSize and - XX:MaxMetaspaceSize . Artifacts Java 8 & below: java { app_name } - verbose:class Java 9 & above: java { app_name } - Xlog:class+load =info:<filename> jcmd { pid } GC.class_histogram Heap Dump Tools GCeasy (GC log analysis) HeapHero Eclipse MAT

25. 25 Permgen space 5 Young Old Threads JNI misc GC Direct Buff Code Cache Heap Memory Native Memory JVM Memory Happens only in Java 7 & below

26. 26 OutOfMemoryError : Permgen Space Diagnosis: Same as ‘Metaspace’ 

27. 27 Unable to create new native threads 6 Young Old Metaspace JNI misc GC Direct Buff Code Cache Heap Memory Native Memory JVM Memory

28. 28 How to capture Thread Dump? 9 options: https ://blog.fastthread.io/how-to-take-thread-dumps-7-options/

29. 1. GC Log 10. netstat 12. vmstat 2. Thread Dump 9. dmesg 3. Heap Dump 6. ps 8. Disk Usage 5. top 13. iostat 11. ping 14. Kernel Params 15. App Logs 16. metadata 4. Heap Substitute 7. top -H 29 Open-source script: https://github.com/ycrash/yc-data-script 360° Troubleshooting artifacts

30. 30 Real Case Study Slowdown in a Major Financial Institution’s Middleware in USA http://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTcvMDMvMTQvLS10aHJlYWREdW1wLTIudHh0LS0xMi0yOC0zMw==&s=t

31. 31 OutOfMemoryError : Unable to create new native threads Causes Threads are leaking Solutions Fix thread leak Increase the Thread Limits Set at Operating System( ulimit –u) Kill other processes Increase RAM capacity Reduce Java Heap Size Reduce thread stack size (- Xss ). Note: can cause StackOverflowError Artifacts Thread Dump Tools FastThread Text Editor

32. 32 Direct buffer memory 7 Young Old Metaspace Threads JNI misc GC Code Cache Heap Memory Native Memory JVM Memory Becoming Pervasive in modern Spring Boot Frameworks

33. 33 Demo OutOfMemoryError : Direct buffer Memory

34. 34 OutOfMemoryError : Direct buffer memory Causes Increase(or Leak) in direct buffers usage java.nio package Moving from Spring RestTemplate to WebClient Image Processing Libraries Networking Libraries Some JDBC Drivers Solutions Fix Memory Leak code Increase - XX:MaxDirectMemorySize Upgrade to Java 17 (There some issues in Java 11) Artifacts App Log or Std error Native Memory Tracking Tools

35. 35 Kill process or sacrifice child 8 JVM Kernel X

36. 1. GC Log 10. netstat 12. vmstat 2. Thread Dump 9. dmesg 3. Heap Dump 6. ps 8. Disk Usage 5. top 13. iostat 11. ping 14. Kernel Params 15. App Logs 16. metadata 4. Heap Substitute 7. top -H 36 Open-source script: https://github.com/ycrash/yc-data-script 360° Troubleshooting artifacts

37. 37

38. 38 Real Case Study Intermittent HTTP 502 errors in AWS EBS Service

39. EBS Architecture 39

40. Clue: Nginx Error 40

41. 41

42. 42 OutOfMemoryError : Kill process or sacrifice child Causes Lack of RAM capacity High Memory utilization by other processes in the environment Memory Leak in the code Solutions Kill other processes Increase RAM capacity Fix Memory Leak in the code Artifacts dmesg GC Log Heap Dump Native Memory Tracking Tools HeapHero Eclipse MAT yCrash

43. 43 Reason stack_trace_with_native_method 9 Young Old Metaspace Threads misc GC Direct Buff Code Cache Heap Memory Native Memory JVM Memory

44. 44 OutOfMemoryError : Reason stack_trace_with_native_method Don’t worry, if you are not using JNI 

45. 45 OutOfMemoryError : Reason stack_trace_with_native_method Causes Heavy Usage of Native Methods Recursive Native Method Calls Solutions Fix the problem in the native app Tools OS native tools Dtrace pmap pstack Artifacts Application Log or std error

46. 46 You need data GC Log Heap Dump 1. Java Heap Space 2. GC overhead limit exceeded 3. Requested array size exceeds VM limit 7. Direct buffer Memory App Logs Verbose Class Log 4. Metaspace 5. PermGen Space Thread Dump 6. Unable to create native threads dmesg 8. Kill process or sacrifice child GC Log

47. 1. GC Log 10. netstat 12. vmstat 2. Thread Dump 9. dmesg 3. Heap Dump 6. ps 8. Disk Usage 5. top 13. iostat 11. ping 14. Kernel Params 15. App Logs 16. metadata 4. Heap Substitute 7. top -H 47 Open-source script: https://github.com/ycrash/yc-data-script You need 360° data

48. Ram Lakshmanan [email protected] @tier1app https://www.linkedin.com/company/ycrash This deck will be published in: https://blog.heaphero.io If you want to learn more … 48 THANK YOU FRIENDS