The java.lang.

OutOfMemoryError:
PermGen Space error demystified
Edward Chou
Frank Kieviet
SOA/BI
Sun Microsystems

BOF-9982

2007 JavaOneSM Conference | Session BOF-9982 |

Goal
What we'll tell you:

java.lang.OutOfMemoryError: PermGen Space:

What is it?
What can you do about it?

2007 JavaOneSM Conference | Session BOF-9982 | 2

A confusing error...
● Out of memory? I have plenty of memory!
● Complexity
● Classes
● Classloaders
● Container code / application code
● Application server at fault?

2007 JavaOneSM Conference | Session BOF-9982 | 3

An often queried error...
● Blog
● About 200 hits per day
● > 50% on entry on memory leaks
● > 80% through Google queries

2007 JavaOneSM Conference | Session BOF-9982 | 4

Agenda
● Basics of memory, GC
● Basics of Classloaders
● Basics of Leaks
● How to fix?
● Demo of tools to find leaks

2007 JavaOneSM Conference | Session BOF-9982 | 5

The basics: garbage collection
void f() {
Object x = new Object();
}
In method f()

After exiting method f()

x GC!

2007 JavaOneSM Conference | Session BOF-9982 | 6

 Objects in memory
Roots

Objects

2007 JavaOneSM Conference | Session BOF-9982 | 7

 Objects in memory
Roots
Reference chain
Root objects
- Bootstrap ClassLoader
- native references
- stack frames

Objects
Reachable

Unreachable

2007 JavaOneSM Conference | Session BOF-9982 | 8

 12

GC: first mark all objects as unreachable

Roots

Objects

(simplified model)

2007 JavaOneSM Conference | Session BOF-9982 | 9

 11

GC: next, mark reachable objects as reachable

Roots

Objects

2007 JavaOneSM Conference | Session BOF-9982 | 10

 10

Roots

Objects

2007 JavaOneSM Conference | Session BOF-9982 | 11

 9

Roots

Objects

2007 JavaOneSM Conference | Session BOF-9982 | 12

 8

Roots

Objects

2007 JavaOneSM Conference | Session BOF-9982 | 13

 7

Roots

Objects

2007 JavaOneSM Conference | Session BOF-9982 | 14

 6

Roots

Objects

2007 JavaOneSM Conference | Session BOF-9982 | 15

 5

Roots

Objects

2007 JavaOneSM Conference | Session BOF-9982 | 16

 4

Roots

Objects

2007 JavaOneSM Conference | Session BOF-9982 | 17

 3

Roots

Objects

2007 JavaOneSM Conference | Session BOF-9982 | 18

 2

Roots

Objects

2007 JavaOneSM Conference | Session BOF-9982 | 19

 1

Roots

Objects

2007 JavaOneSM Conference | Session BOF-9982 | 20

 0

Roots

Objects

2007 JavaOneSM Conference | Session BOF-9982 | 21

 Lastly GC all unreachable objects
Roots

Objects

2007 JavaOneSM Conference | Session BOF-9982 | 22

Memory leaks
● Objects are accidentally referenced
● Objects accumulate

2007 JavaOneSM Conference | Session BOF-9982 | 23

The basics: memory generations
● Young generation
● Newly created objects
● Tenured generation
● Objects that survived several rounds of GC
● Permgen space (permanent generation)
● Class objects
● String.intern()

2007 JavaOneSM Conference | Session BOF-9982 | 24

The basics: classes and classloaders
● Each object is an instance of a class
● A class itself is an object (class object)
● instance of the class Class
● Each class references its classloader
● A classloader references all classes it loaded
● Class objects hold static members
java.lang.Class

statics
* 1
*
java.lang.ClassLoader 1 java.lang.Object
*

2007 JavaOneSM Conference | Session BOF-9982 | 25

Why classloaders?
● Containers use classloaders to
● dynamically load applications
● isolate applications from eachother
● dynamically unload applications

2007 JavaOneSM Conference | Session BOF-9982 | 26

Example: empty servlet
package com.stc.test;

import java.io.*;
import java.net.*;
import javax.servlet.*;
import javax.servlet.http.*;

public class Servlet1 extends HttpServlet {

private static final String STATICNAME = "Simple";
protected void doGet(HttpServletRequest request,
HttpServletResponse response)
throws ServletException, IOException {
// nothing
}
}

2007 JavaOneSM Conference | Session BOF-9982 | 27

Deployed

Undeployed

2007 JavaOneSM Conference | Session BOF-9982 | 28

Classloader leaks
● A classloader cannot be GC-ed if any of the
instances of any of the classes it loaded are
reachable.
● Such a classloader keeps all its classes with all
their static members in memory.
● Not immediately apparent from a memory dump what
is a leak and what is not.
● Cause of the leak difficult to find.

2007 JavaOneSM Conference | Session BOF-9982 | 29

Example: a leaking servlet
package com.stc.test;

import java.io.*;
import java.util.logging.*;
import javax.servlet.*;
import javax.servlet.http.*;

public class LeakServlet extends HttpServlet {

private static final String STATICNAME = "Leak!";
static Level CUSTOMLEVEL = new Level("OOPS", 555) {};
protected void doGet(HttpServletRequest request,
HttpServletResponse response)
throws ServletException, IOException {
// Log at a custom level
Logger.getLogger("test").log(CUSTOMLEVEL, "x called");
}
}
2007 JavaOneSM Conference | Session BOF-9982 | 30
Deployed

2007 JavaOneSM Conference | Session BOF-9982 | 31

java.util.logging.Level class
private static List known = new ArrayList();

protected Level(String name, int value) {

this.name = name;
this.value = value;
synchronized (Level.class) {
known.add(this);
}
}

2007 JavaOneSM Conference | Session BOF-9982 | 32

A leak...

Undeployed

2007 JavaOneSM Conference | Session BOF-9982 | 33

Reality:
● Hundreds of leaked classes
● Thousands of leaked objects
● Bafflement...

2007 JavaOneSM Conference | Session BOF-9982 | 34

Profilers
Profilers...
● Take memory snapshots

● Find reference chains to root objects

● Most see class objects as root objects

2007 JavaOneSM Conference | Session BOF-9982 | 35

 Profilers find reference chains
Roots

Objects

2007 JavaOneSM Conference | Session BOF-9982 | 36

 Profilers may not find reference chains...
Roots

Objects

Class object

2007 JavaOneSM Conference | Session BOF-9982 | 37

JHat
● A tool that reads hprof memory dumps
● Comes with JDK 6

2007 JavaOneSM Conference | Session BOF-9982 | 38

Steps to find the cause of a memory leak:

● Trigger a heap dump

● -XX:+HeapDumpOnOutOfMemoryError
● -Xrunhprof:heap=dump,format=b
● jmap -dump:format=b,file=heap.bin <pid>
● Analyze heap dump using Jhat
● jhat -J-mx1024m heap.bin
● http://localhost:7000
● Using built-in or custom queries to narrow down leak
suspects
● Identify an object or class in the application
● List reference chains

2007 JavaOneSM Conference | Session BOF-9982 | 39

DEMO
A real life example

2007 JavaOneSM Conference | Session XXXX | 40

 4

Multiple reference chains

Roots

Objects

2007 JavaOneSM Conference | Session BOF-9982 | 41

 3

Roots

Objects

2007 JavaOneSM Conference | Session BOF-9982 | 42

 2

Roots

Objects

2007 JavaOneSM Conference | Session BOF-9982 | 43

 1

Roots

Objects

2007 JavaOneSM Conference | Session BOF-9982 | 44

 0

Roots

Objects

2007 JavaOneSM Conference | Session BOF-9982 | 45

java.lang.Thread class
private ClassLoader contextClassLoader;
private AccessControlContext inheritedAccessControlContext;

private void init(ThreadGroup g, Runnable target,

String name, long stackSize) {

this.contextClassLoader = parent.contextClassLoader;
...
...
this.inheritedControlContext =
AccessController.getContext();
}

2007 JavaOneSM Conference | Session BOF-9982 | 46

javax.security.auth.Policy class
private static ClassLoader contextClassLoader;

static {
contextClassLoader = (ClassLoader)
java.security.AccessController.doPrivileged(
new java.security.PrivilegedAction() {
public Object run() {
return
Thread.currentThread().getContextClassLoader();
}
});
};

2007 JavaOneSM Conference | Session BOF-9982 | 47

java.sql.DriverManager class
private static java.util.Vector writeDrivers =
new java.util.Vector();

public static synchronized void registerDriver(

java.sql.Driver driver) throws SQLException {
...

DriverInfo di = new DriverInfo();

di.driver = driver;
di.driverClass = driver.getClass();
di.driverClassName = di.driverClass.getName();

writeDrivers.addElement(di);

...
}

2007 JavaOneSM Conference | Session BOF-9982 | 48

Common problems
● Dangling thread
● Commons logging
● java.util.logging.Level
● Thread context classloader
● new Thread()
● Bean util
● Details
● SQL Driver

2007 JavaOneSM Conference | Session BOF-9982 | 49

Determining the leaked classloader(s)
● Use a known class as a marker
● Does not work in systems with many classloaders
● Identify classloaders with identical sets of classes
● e.g. 10 classloaders out of 100 classloaders load
classes {A, B, C, D}
● Identify “orphaned” classloaders
● i.e. no reference chains to classloader besides classes

2007 JavaOneSM Conference | Session BOF-9982 | 50

“Orphaned” classloader

2007 JavaOneSM Conference | Session BOF-9982 | 51

Summary
● jhat is an effective tool to track down classloader
leaks
● Approach:
● Obtain a dump
● Identify suspect classloader
● Identify a known class
● Duplicate sets of classes
● “Orphaned” classloaders
● Trace reference chains

2007 JavaOneSM Conference | Session BOF-9982 | 52

Q&A
http://blogs.sun.com/fkieviet
http://blogs.sun.com/edwardchou

This picture © 2007 Software Test and Performance Magazine

2007 JavaOneSM Conference | Session XXXX | 53

Optional Demo
● How to write new jhat query?
● Object Query Language (OQL)
● Java

2007 JavaOneSM Conference | Session BOF-9982 | 54