TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR

Even though at surface level ‘java.lang.OutOfMemoryError’ appears as one single error; underlyingly there are 9 types of OutOfMemoryError. Each type of OutOfMemoryError has different causes, diagnosis approaches and solutions. This session equips you with the knowledge, tools, and techniques needed to troubleshoot and conquer OutOfMemoryError in all its forms, ensuring smoother, more efficient Java applications.

1. 1

2. Does Java process memory utilization go beyond – Xmx ? 2 – XX:MaxMetaspaceSize - Xmx Young Old Metaspace Threads JNI misc GC Direct Buff Code Cache Heap Memory Native Memory Java Process Memory

3. 3 Demo OutOfMemoryError

4. 4 How to know the type of OutOfMemoryError ? java.lang.OutOfMemoryError : <type>

5. 5 Java Heap Space 1 Metaspace Threads JNI misc GC Direct Buff Code Cache Heap Memory Native Memory JVM Memory Most common type

6. 6 GC Behavior of a Healthy Application - Full Garbage Collection Event

7. 7 GC Behavior of Acute Memory Leak - Full Garbage Collection Event

8. 8 GC Behavior when there is a Memory Leak - Full Garbage Collection Event

9. 9 GC Behavior when App needs more memory - Full Garbage Collection Event

10. 10 How to study GC Behavior?  -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:<file-path> Till Java 8 -Xlog:gc*:file=<file-path> From Java 9 Enable GC logs (always) Almost zero overhead

11. 11 GCeasy GC Log Analysis Demo

12. 12 How to capture Heap Dumps? 8 options: https://blog.heaphero.io/2017/10/13/how-to-capture-java-heap-dumps-7-options/

13. 1. GC Log 10. netstat 12. vmstat 2. Thread Dump 9. dmesg 3. Heap Dump 6. ps 8. Disk Usage 5. top 13. iostat 11. ping 14. Kernel Params 15. App Logs 16. metadata 4. Heap Substitute 7. top -H 13 Open-source script: https://github.com/ycrash/yc-data-script 360° Troubleshooting artifacts

14. 14 HeapHero Heap Dump Analysis Demo

15. 15 OutOfMemoryError : Java Heap Space Causes Increase in Traffic Volume Memory leak due to buggy code Solutions Fix Memory Leak in the code Increase heap size (- Xmx ) Artifacts Garbage Collection Log Heap Dump Tools GCeasy (GC log analysis) HeapHero Eclipse MAT JVisualVM

16. 16 GC overhead limit exceeded 2 Metaspace Threads JNI misc GC Direct Buff Code Cache Heap Memory Native Memory JVM Memory

17. 17 OutOfMemoryError : GC overhead limit exceeded Diagnosis: Same as ‘Java Heap Space’  Java process is spending more than 98% of its time doing garbage collection and recovering less than 2% of the heap and has been doing so far the last 5 consecutive garbage collections

18. 18 Requested array size exceeds VM limit 3 Young Old Metaspace Threads JNI misc GC Direct Buff Code Cache Heap Memory Native Memory JVM Memory

19. 19 Demo OutOfMemoryError : Requested array size exceeds VM limit

20. 20 OutOfMemoryError : Requested array size exceeds VM limit Causes Allocated array larger than heap size Solutions Fix code which allocates large array size Increase heap size (- Xmx ) Artifacts Application Log or std error Tools

21. 21 Metaspace 4 Young Old Threads JNI misc GC Direct Buff Code Cache Heap Memory Native Memory JVM Memory

22. 22 Demo OutOfMemoryError : Metaspace

23. 23 Metaspace Memory Leak - Full Garbage Collection Event

24. 24 Metaspace Analysis Java 8 & below: java { app_name } - verbose:class Java 9 & above: java { app_name } - Xlog:class+load =info:<filename>

25. 25 OutOfMemoryError : Metaspace Causes Creating large number of Dynamic classes (Java Reflection, Groovy type of scripting languages) Loading large number of Classes (3 rd party libraries/frameworks) Loading large number of Classloaders Solutions Fix Memory Leak code Increase - XX:MetaspaceSize and - XX:MaxMetaspaceSize . Artifacts Java 8 & below: java { app_name } - verbose:class Java 9 & above: java { app_name } - Xlog:class+load =info:<filename> jcmd { pid } GC.class_histogram Heap Dump Tools GCeasy (GC log analysis) HeapHero Eclipse MAT

26. 26 Permgen space 5 Young Old Threads JNI misc GC Direct Buff Code Cache Heap Memory Native Memory JVM Memory Happens only in Java 7 & below

27. 27 OutOfMemoryError : Permgen Space Diagnosis: Same as ‘Metaspace’ 

28. 28 Unable to create new native threads 6 Young Old Metaspace JNI misc GC Direct Buff Code Cache Heap Memory Native Memory JVM Memory

29. 29 How to capture Thread Dump? 9 options: https ://blog.fastthread.io/how-to-take-thread-dumps-7-options/

30. 1. GC Log 10. netstat 12. vmstat 2. Thread Dump 9. dmesg 3. Heap Dump 6. ps 8. Disk Usage 5. top 13. iostat 11. ping 14. Kernel Params 15. App Logs 16. metadata 4. Heap Substitute 7. top -H 30 Open-source script: https://github.com/ycrash/yc-data-script 360° Troubleshooting artifacts

31. 31 Real Case Study Slowdown in a Major Financial Institution’s Middleware http://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTcvMDMvMTQvLS10aHJlYWREdW1wLTIudHh0LS0xMi0yOC0zMw==&s=t

32. 32 OutOfMemoryError : Unable to create new native threads Causes Threads are leaking Solutions Fix thread leak Increase the Thread Limits Set at Operating System( ulimit –u) Kill other processes Increase RAM capacity Reduce Java Heap Size Reduce thread stack size (- Xss ). Note: can cause StackOverflowError Artifacts Thread Dump Tools FastThread Text Editor

33. 33 Direct buffer memory 7 Young Old Metaspace Threads JNI misc GC Code Cache Heap Memory Native Memory JVM Memory Becoming Pervasive in modern Spring Boot Frameworks

34. 34 Demo OutOfMemoryError : Direct buffer Memory

35. 35 OutOfMemoryError : Direct buffer memory Causes Increase(or Leak) in direct buffers usage java.nio package Moving from Spring RestTemplate to WebClient Image Processing Libraries Networking Libraries Some JDBC Drivers Solutions Fix Memory Leak code Increase - XX:MaxDirectMemorySize Upgrade to Java 17 (There some issues in Java 11) Artifacts App Log or Std error Native Memory Tracking Tools

36. 36 Kill process or sacrifice child 8 JVM Kernel X

37. 1. GC Log 10. netstat 12. vmstat 2. Thread Dump 9. dmesg 3. Heap Dump 6. ps 8. Disk Usage 5. top 13. iostat 11. ping 14. Kernel Params 15. App Logs 16. metadata 4. Heap Substitute 7. top -H 37 Open-source script: https://github.com/ycrash/yc-data-script 360° Troubleshooting artifacts

38. 38

39. 39 Real Case Study Intermittent HTTP 502 errors in AWS EBS Service

40. EBS Architecture 40

41. Clue: Nginx Error 41

42. 42

43. 43 OutOfMemoryError : Kill process or sacrifice child Causes Lack of RAM capacity High Memory utilization by other processes in the environment Memory Leak in the code Solutions Kill other processes Increase RAM capacity Fix Memory Leak in the code Artifacts dmesg GC Log Heap Dump Native Memory Tracking Tools HeapHero Eclipse MAT yCrash

44. 44 Reason stack_trace_with_native_method 9 Young Old Metaspace Threads misc GC Direct Buff Code Cache Heap Memory Native Memory JVM Memory

45. 45 OutOfMemoryError : Reason stack_trace_with_native_method Don’t worry, if you are not using JNI 

46. 46 OutOfMemoryError : Reason stack_trace_with_native_method Causes Heavy Usage of Native Methods Recursive Native Method Calls: Solutions Fix the problem in the native app Tools OS native tools Dtrace pmap pstack Artifacts Application Log or std error

47. Ram Lakshmanan [email protected] @tier1app https://www.linkedin.com/company/ycrash This deck will be published in: https://blog.heaphero.io If you want to learn more … 47 THANK YOU FRIENDS