Notes to self

Whatever I feel like writing

PermGen Strikes Back!

Thursday, March 15, 2007 3:34:58 PM

,

Bugger, bugger, bugger!

Just as I thought that the issue with the PermGen OutOfMemoryError was over, it turns out: it's not.

Here's why: by setting the -XX:MaxPermSize=128m, I was only postponing the eventual death of my app server. This postponing was apparently so good, that I didn't incounter the PermGen issue during my, ahem, "tests".

In fact, my "test" was not only manual, it was also naïve. What I did was to run the app server without the GC tuning parameters, and make redeployments until it crashed. Turns out I could make 5 redeployments with an app called Payout (named after my current project). Then I applied the parameters and redeployed again and again until I felt confident it worked. Apparently, I felt confident after 10 redeployments.

My failure was that I didn't do any actual measuring of the state of the memory in the JVM. :clown:

Nervous sysadmin

My sysadmin contacted me today and said that he was worried that the memory usage pattern hadn't changed significantly. The JVM was still, slowly but surely, eating up all its memory. I pointet out that the MaxPermSize had been increased, and the GC probably didn't bother to reclaim memory because there was still plenty of it available. My rationale was, that once the JVM got closer to the max, it would start to do more garbage collection.

And then, I started to feel like testing my claims.

I wrote a small web application that used the MemoryMXBean to plot the NonHeap memory stats to an array list every minute, and then print the list, CSV formattet, on a JSP file. I then followed the memory usage as I start to redeploy again and a again, to see what would happen when I reached the max. As you figured, I got PermGens.
Then I snatched the memory statistics from my little plotter app, and importedd it into Excel to generate this graph:


What have we learned?

Two things:
  1. The PermGen issue is harder to crack than I first thought.
  2. One should properly test one's claims before bragging about it on blogs and dzone and whatnot.

So I guess it's back to the drawing board! furious

Good Riddance, PermGen OutOfMemoryError !Spambot Talk

Comments

Anonymous Thursday, March 15, 2007 8:29:50 PM

sgtmcd writes: I did several hours of testing on this issue. java 1.5_06-b05 on a windows 2003 server. I plotted the permgen for hours on end, both with these switches and without. The one switch I left constant between each test was -XX:MaxPermSize=128m. without Switches: Overall, permgen grew quite rapidly to about 140meg within 10 minutes, then settled out at about 80meg and stayed roughly steady for 3 hours (when I finally got tired of looking at it.) with the above switches: permgen grew much slower and peaked at about 90meg before garbage collection brought it down to about 55meg where it stayed steady for about 3 hours (when I got really tired and went home!) During both of the above tests I reloaded my webapp 20 times in 15 minutes(that was as fast as I could hit reload). I also ran jmeter on a loop hitting pages in the application over and over during the entire test. I never encountered a permgen memory error with either test. So, in my opinion, the switches are a good thing with lower overall memory usage and much faster garbage collection.

Christian Vest Hansenkarmazilla Friday, March 16, 2007 10:18:13 AM

That's odd. Why does is work for you, but not for me?

This is the complete bootstrap for my app server:

===============================================================================

  JBoss Bootstrap Environment

  JBOSS_HOME: C:\Programmer\jboss-4.0.5.GA\bin\\..

  JAVA: C:\Program Files\Java\jdk1.5.0_08\bin\java

  JAVA_OPTS:  -Dprogram.name=run.bat -XX:+PrintGC -Xms128m -Xmx512m -XX:+CMSPermGenSweepingEnabled -XX:MaxPermSize=128m -XX:+
CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600
000

  CLASSPATH: C:\Program Files\Java\jdk1.5.0_08\lib\tools.jar;C:\Programmer\jboss-4.0.5.GA\bin\\run.jar

===============================================================================


I saw no difference in running with and without the -server switch (other than the client uses slightly less memory).

Another thing: does the application you testet with use Hibernate?

Anonymous Friday, March 16, 2007 12:41:37 PM

sgtmcd writes: I think, maybe, the biggest difference is that I am using tomcat rather than jboss. I am using some pretty intense frameworks as far as memory usage... spring, ibatis, sitemesh are the main three that suck up a lot of cpu/memory on startup. I do see a couple switches that I don't use. -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600

Christian Vest Hansenkarmazilla Friday, March 16, 2007 8:57:01 PM

Tomcat uses memory the same way as JBoss. In fact, Tomcat is the servlet container in JBoss.

However, I think Hibernate is a particulary nasty framework in exposing this particular issue. I think it creates and loads new classes on the fly at run time, and thus drawing heavily on the (possibly non-heap allocated) method area of the JVMs memory space.

If you try and test again with an application that uses Hibernate in any extent, then I bet you'll see different results.

Anonymous Friday, March 23, 2007 8:26:35 PM

Anonymous writes: Hi You should try a larger PermGen to see if the problem is temporary in that eventually that space will be corrected. Take a look at http://www.unixville.com/~moazam/stories/2004/05/17/maxpermsizeAndHowItRelatesToTheOverallHeap.html

Christian Vest Hansenkarmazilla Saturday, March 24, 2007 12:02:31 PM

That will only delay the problem.

Read again:
"The permanent generation is used to hold reflective data of the VM itself such as class objects and method objects. These reflective objects are allocated directly into the permanent generation, and it is sized independently from the other generations."

The problem is, that the class and method objects properly unloaded in some cases, when you redeploy a web application on a running application server. This will slowly cause the permanent generation to fill up, and when it does, your server will die with an "OutOfMemoryError: PermGen".

The most effective way to expose the issue, that I know of, is to redeploy a web application that uses Hibernate on an application server that uses the Tomcat class loader, such as Tomcat, JBoss and Glassfish. Though I should note that I haven't actually tested on Glassfish, and they have made some changes to the class loader.

Anonymous Wednesday, April 25, 2007 12:47:54 PM

Galder writes: Are you using Hibernate and Javassist? 4.0.5.GA comes with Hibernate 3.2.0.GA and Javassist 3.3 which faces the issue explained in JBoss's JIRA -> JASSIST-28 This leads to permanent space memory leak. Upgrade to Hibernate 3.2.1.GA or newer and Javassist 3.4 to fix the issue.

Christian Vest Hansenkarmazilla Wednesday, April 25, 2007 3:02:39 PM

That's not it. I'm using CGLIB rather than Javassist.

I just tried updating to Hibernate 3.2.3.ga - still the same.

Anonymous Thursday, April 26, 2007 3:58:11 PM

scottw writes: Your CGLIB reference suggested the following thread to me. See if it helps: http://forum.springframework.org/showthread.php?t=21383&highlight=cglib+cache

Christian Vest Hansenkarmazilla Thursday, April 26, 2007 5:12:44 PM

Apparently from the Tomcat FAQ and that link above:

Q. Why does the memory usage increase when I redeploy a web application?

A. Because the Classloader (and the Class objects it loaded) cannot be recycled. They are stored in the permanent heap generation by the JVM, and when you redepoy a new class loader is created, which loads another copy of all these classes. This can cause OufOfMemoryErrors eventually.



So I guess the real answer is twofold:

1. Fix Tomcat if you got the time, or
2. Don't use Tomcat or any derived App Server (JBoss, Glassfish)

Someone from that post mentioned that Jetty didn't experience this issue; I'm not so sure about that. It's a while ago but I've also tried to redeploy on Jetty and as far as I recall, the memory usage had a steady growth and eventually killed the server.

EDIT:
This info might also be useful:

Since using Tomcat 6.0 my PermGen space OutOfMemory exceptions are gone.


Only thing is figuring out how to plug Tomcat 6 into the belly of JBoss. (asuming it isn't already using it)

Anonymous Thursday, June 7, 2007 2:13:44 PM

tschwenk writes: I'm using Spring and Hibernate with JBoss 4.2, and the only thing that I could do to get the OOM errors to stop occurring on redeploy was to switch to BEA's JRockit JDK. I was using Sun 1.5.0_11. -Tom

Christian Vest Hansenkarmazilla Friday, June 8, 2007 6:12:35 AM

Please note that JRocket dosn't solve the problem of classes that dosn't get garbage collected (and hence take up increasingly more perm-gen space), rather, JRocket allows the perm-gen to grow up to 4 GBs.

This means that using JRockit will make the problem appear to be gone, when it really isn't.

Anonymous Wednesday, June 27, 2007 9:23:47 AM

Tomas writes: Hi We encountered the same the dreaded Perm Gen space problem also. I solved it for our application in 2 different ways. We used: Jboss 4.0.5GA Hibernate 3.2.0 javaassist 3.3 We changed to: Hibernate 3.2.4.SP1 javaassist 3.4GA We also made changes in how we use Hibernates session. After every transaction we flush and clear the session. This is perhaps the standard way to do this, but we did not see Hibernate session beeing cleared in any tutorial on the net. This worked for us. From adding nearly 2MB to the heap for every request made, now it only adds to the heap when calling a bean method for the first time.

Anonymous Friday, February 1, 2008 3:17:43 AM

Sheepy writes: We are using Tomcat 6.0.14 on Sun Java 1.5.0.14, without Hibernate or Spring, but we also got PermGen OutOfMemory on redeploy, very regularly before we increase MaxPermGen. Actually we were just updating class files on the fly.

Anonymous Wednesday, May 7, 2008 10:23:01 AM

Anonymous writes: Frank Kieviet has a few good posts about this error. Take a look here: http://blogs.sun.com/fkieviet/entry/classloader_leaks_the_dreaded_java http://blogs.sun.com/fkieviet/entry/how_to_fix_the_dreaded

Christian Vest Hansenkarmazilla Wednesday, May 7, 2008 6:41:33 PM

@Anonymous:

Exactly. I believe I've also noted those blog posts here:
http://my.opera.com/karmazilla/blog/2007/09/29/return-of-the-permgen

Anonymous Tuesday, July 28, 2009 9:10:22 AM

Anonim writes: I recommend reading this post: http://blogs.sun.com/fkieviet/entry/classloader_leaks_the_dreaded_java

Anonymous Thursday, January 7, 2010 5:24:31 AM

mm writes: Few pointers here as well, http://minmaxmim.blogspot.com/2010/01/javalangoutofmemoryerror-java-heap.html http://minmaxmim.blogspot.com/2010/01/javalangoutofmemoryerror-permgen-space.html ~~~ http://www.captivecaptures.blogspot.com/

Anonymous Monday, July 5, 2010 8:52:08 PM

Jason Gritman writes: We were experiencing some serious PermGen leak problems with JBoss 4.0.5. After reading the advice above, we found that upgrading javassist fixed the issue.

Anonymous Thursday, September 29, 2011 9:14:01 PM

Harish writes: All of a Sudden encountering this issue for past few days on our JBOSS Server 4.05 GA. Thanks for all the suggestions. Not sure exactly what's causing it. Going to try few things see how it responds. I think that's only way out. GC parmaters seems to provide little help. Crossing my fingers and waiting Best Regards

How to use Quote function:

  1. Select some text
  2. Click on the Quote link

Write a comment

Comment
(BBcode and HTML is turned off for anonymous user comments.)

If you can't read the words, press the small reload icon.


Smilies

Links