3 Troubleshooting Memory Leaks

Creating a Heap Dump

A heap dump provides detailed information on the allocation of heap memory. There are several ways to create a heap dump:

Obtaining a Heap Histogram

You can try to quickly narrow down a memory leak by examining the heap histogram. It can be obtained in several ways:

• If the Java process is started with the -XX:+PrintClassHistogram command-line option, then the Control+Break handler will produce a heap histogram.

• You can use the jmap utility to obtain a heap histogram from a running process:

  jmap -histo pid
  

  The output shows the total size and instance count for each class type in the heap. If a sequence of histograms is obtained (for example, every 2 minutes), then you might be able to observe a trend that can lead to further analysis.

• You can use the jmap utility to obtain a heap histogram from a core file:

  jmap -histo core_file
  

  For example, if you specify the -XX:+HeapDumpOnOutOfMemoryError command-line option while running your application, then when an OutOfMemoryError exception is thrown, the JVM will generate a heap dump. You can then execute jmap on the core file to get a histogram, as in the following example:

  $ jmap -histo \ /java/re/javase/6/latest/binaries/solaris-sparc/bin/java core.27421
  
  Attaching to core core.27421 from executable 
  /java/re/javase/6/latest/binaries/solaris-sparc/bin/java, please wait...
  Debugger attached successfully.
  Server compiler detected.
  JVM version is 1.6.0-beta-b63
  Iterating over heap. This may take a while...
  Heap traversal took 8.902 seconds.
  
  Object Histogram:
   
  Size      Count   Class description
  -------------------------------------------------------
  86683872  3611828 java.lang.String
  20979136  204     java.lang.Object[]
  403728    4225    * ConstMethodKlass
  306608    4225    * MethodKlass
  220032    6094    * SymbolKlass
  152960    294     * ConstantPoolKlass
  108512    277     * ConstantPoolCacheKlass
  104928    294     * InstanceKlassKlass
  68024     362     byte[]
  65600     559     char[]
  31592     359     java.lang.Class
  27176     462     java.lang.Object[]
  25384     423     short[]
  17192     307     int[]
  :
  

  The example shows that the OutOfMemoryError exception was caused by the number of java.lang.String objects (3,611,828 instances in the heap). Without further analysis it is not clear where the strings are allocated. However, the information is still useful and the investigation can continue with tools such as HPROF and jhat to find where the strings are allocated, as well as what references are keeping them alive and preventing them from being garbage collected.

Monitoring the Number of Objects Pending Finalization

When the OutOfMemoryError exception is thrown with the "Java heap space" detail message, the cause can be excessive use of finalizers. To diagnose this, you have several options for monitoring the number of objects that are pending finalization:

Tracking All Memory Allocation and Free Calls

A very common practice is to track all allocation and free calls of the native allocations. This can be a fairly simple process or a very sophisticated one. Many products over the years have been built up around the tracking of native heap allocations and the use of that memory.

Tracking Memory Allocation in a JNI Library

If you write a JNI library, it would probably be wise to create some kind of localized way to make sure your library does not leak memory, using a simple wrapper approach.

The following procedure is an easy localized allocation tracking approach for a JNI library. First, define the following lines in all source files:

#include <stdlib.h>
#define malloc(n) debug_malloc(n, __FILE__, __LINE__)
#define free(p) debug_free(p, __FILE__, __LINE__)

Then you can use the following functions to watch for leaks.

/* Total bytes allocated */
static int total_allocated;
/* Memory alignment is important */
typedef union { double d; struct {size_t n; char *file; int line;} s; } Site;
void *
debug_malloc(size_t n, char *file, int line) 
{ 
    char *rp;
    rp = (char*)malloc(sizeof(Site)+n); 
    total_allocated += n; 
    ((Site*)rp)->s.n = n;
    ((Site*)rp)->s.file = file;
    ((Site*)rp)->s.line = line;
    return (void*)(rp + sizeof(Site));
}
void 
debug_free(void *p, char *file, int line)
{
    char *rp;
    rp = ((char*)p) - sizeof(Site);
    total_allocated -= ((Site*)rp)->s.n;
    free(rp);
}

The JNI library would then need to periodically (or at shutdown) check the value of the total_allocated variable to make sure that it made sense. The above code could also be expanded to save in a linked list the allocations that remained and report where the leaked memory was allocated. This is a localized and portable way to track memory allocations in a single set of sources. You would need to make sure that debug_free() was called only with the pointer that came from debug_malloc(), and you would also need to create similar functions for realloc(), calloc(), strdup(), and so forth, if they were used.

A more global way to look for native heap memory leaks would involve interposition of the library calls for the entire process.

Tracking Memory Allocation With OS Support

Most operating systems include some form of global allocation tracking support.

Using the libumem Tool to Find Leaks

First introduced in Solaris 9 OS update 3, the libumem.so library and the modular debugger mdb can be used to debug memory leaks. Before using libumem, you must preload the libumem library and set an environment variable as follows:

$ LD_PRELOAD=libumem.so
$ export LD_PRELOAD
$ UMEM_DEBUG=default
$ export UMEM_DEBUG

Now, run the Java application but stop it before it exits. The following example uses truss to stop the process when it calls the _exit system call:

$ truss -f -T _exit java MainClass arguments

At this point you can attach the mdb debugger, as follows:

$ mdb -p pid
>::findleaks

The ::findleaks command is the mdb command to find memory leaks. If a leak is found, the findleaks command prints the address of the allocation call, buffer address, and nearest symbol.

It is also possible to get the stack trace for the allocation which resulted in the memory leak by dumping the bufctl structure. The address of this structure can be obtained from the output of the ::findleaks command. The description of the commands to perform these functions, as well as more information on using libumem to identify memory managements bugs, is located at