Good Riddance, PermGen OutOfMemoryError !
Tuesday, 13. March 2007, 20:44:47
Hi, you're probably here because you are hot-deploying some Java web applications and you have problems with some PermGen: OutOfMemoryErrors. Then you somehow found a link to an article that claimed to solve this very problem.
Well, that article was wrong. So I put up a big red note, but people still link to it.
So, in an attempt to set things straight again, I present a short list of facts that I have collected since I wrote the original article:
The only solution I have found that is feasible, reliable and portable is this: reboot your servers!
P.S. Don't bother giving me links to other articles on this issue. Even if you happen to find something I haven't already read, chances are that it is simply rehashing stuff I already know.
Thanks!
Bellow this line is the original article with all it's faulty conclusions.
-----------------------------------------------------------------------------------------------------------------------
I did it!
Yes, lads & lasses! If you've been annoyed with having JBoss or Tomcat die with an OutOfMemoryError every fifth time you redeploy your beloved brainchild of a web application, then this is your lucky day! Because I found a fix! It's true! Yay! ....!!!!!11one (can you tell this has been a pain to me?)
Boring but Serious Theory and Hypothesis part:
The "PermGen" error happens, when the Java virtual machine runs out of memory in the permanent generation. Recall that Java has a generational garbage collector, with four generations: eden, young, old and permanent.
In the eden generation, objects are very short lived and garbage collection is swift and often.
The young generation consists of objects that survived the eden generation (or was pushed down to young because the eden generation was full at the time of allocation), garbage collection in the young generation is less frequent but still happens at quite regular intervals (provided that your application actually does something and allocates objects every now and then).
The old generation, well, you figured it. It contains objects that survived the young generation, or have been pushed down, and garbage collection is even less infrequent but can still happen.
And finally, the permanent generation. This is for objects that the virtual machine has decided to endorse with eternal life - which is precicely the core of the problem. Objects in the permanent generation are never garbage collected; that is, under normal circumstances when the jvm is started with normal command line parameters.
So what happens when you redeploy your web application is, that your WAR file is unpacked and its class files loaded into the jvm. And here's the thing: almost always ends up in the permanent generation... Because, seriously, who wants to garbage collect their classes?!? Well, apparently application servers do, and here's how we make that happen;
PermGen, The Fix:
The standard garbage collector can't collect in the permanent generation, but the concurrent collector can. So the first thing we need to do is to make sure that the jvm uses the concurrent garbage collector. This is done by putting this:
> -XX:+UseConcMarkSweepGC
In java's command line arguments. But this is not enough. We must also specifically tell it to collect in the permanent generation, and this is done with this command line argument:
> -XX:+CMSPermGenSweepingEnabled
Good, now the concurrent collector will take the permantent generation under its wings. But wait! Classes are special, and the jvm is reluctant to let go of them, so we must also explicitly allow classes to be unloaded:
> -XX:+CMSClassUnloadingEnabled
Now we're certain that the permanent generation will be properly cleaned. But this raises another issue: what if the jvm unloads classes that might still be needed? I imagine it can be hard for a collector to tell whether or not a class might still be needed with the amount of reflection that goes on in an application server. Therefor, we might want to tweak the amount of memory allocated for the permanent generation, and this is done with this command line parameter:
> -XX:MaxPermSize=128m
Which will set the maximum size of our permanent generation to 128 megabytes - tweak it to fit your needs.
With these parameters properly applied to the jvm that runs your application server, your chances of running into a PermGen OutOfMemoryError will be considerably lessened.
Take care!
Well, that article was wrong. So I put up a big red note, but people still link to it.
So, in an attempt to set things straight again, I present a short list of facts that I have collected since I wrote the original article:
- Bugs in the Sun garbage collector is one source of this problem.
- There are (were? can't be arsed to check) more than one bug, hence confusion.
- Your app failing to undeploy completely will leak all classes in your app, every time you redeploy - this is another source of the problem.
- None of these sources can be fixed with GC flags.
- The JRocket JVM does NOT fix this problem - it simply lets the permgen grow unbounded, until something bad happens, like swap-thrashing or the OS kills it.
The only solution I have found that is feasible, reliable and portable is this: reboot your servers!
P.S. Don't bother giving me links to other articles on this issue. Even if you happen to find something I haven't already read, chances are that it is simply rehashing stuff I already know.
Thanks!
Bellow this line is the original article with all it's faulty conclusions.
-----------------------------------------------------------------------------------------------------------------------
I did it!
Yes, lads & lasses! If you've been annoyed with having JBoss or Tomcat die with an OutOfMemoryError every fifth time you redeploy your beloved brainchild of a web application, then this is your lucky day! Because I found a fix! It's true! Yay! ....!!!!!11one (can you tell this has been a pain to me?)
Boring but Serious Theory and Hypothesis part:
The "PermGen" error happens, when the Java virtual machine runs out of memory in the permanent generation. Recall that Java has a generational garbage collector, with four generations: eden, young, old and permanent.
In the eden generation, objects are very short lived and garbage collection is swift and often.
The young generation consists of objects that survived the eden generation (or was pushed down to young because the eden generation was full at the time of allocation), garbage collection in the young generation is less frequent but still happens at quite regular intervals (provided that your application actually does something and allocates objects every now and then).
The old generation, well, you figured it. It contains objects that survived the young generation, or have been pushed down, and garbage collection is even less infrequent but can still happen.
And finally, the permanent generation. This is for objects that the virtual machine has decided to endorse with eternal life - which is precicely the core of the problem. Objects in the permanent generation are never garbage collected; that is, under normal circumstances when the jvm is started with normal command line parameters.
So what happens when you redeploy your web application is, that your WAR file is unpacked and its class files loaded into the jvm. And here's the thing: almost always ends up in the permanent generation... Because, seriously, who wants to garbage collect their classes?!? Well, apparently application servers do, and here's how we make that happen;
PermGen, The Fix:
The standard garbage collector can't collect in the permanent generation, but the concurrent collector can. So the first thing we need to do is to make sure that the jvm uses the concurrent garbage collector. This is done by putting this:
> -XX:+UseConcMarkSweepGC
In java's command line arguments. But this is not enough. We must also specifically tell it to collect in the permanent generation, and this is done with this command line argument:
> -XX:+CMSPermGenSweepingEnabled
Good, now the concurrent collector will take the permantent generation under its wings. But wait! Classes are special, and the jvm is reluctant to let go of them, so we must also explicitly allow classes to be unloaded:
> -XX:+CMSClassUnloadingEnabled
Now we're certain that the permanent generation will be properly cleaned. But this raises another issue: what if the jvm unloads classes that might still be needed? I imagine it can be hard for a collector to tell whether or not a class might still be needed with the amount of reflection that goes on in an application server. Therefor, we might want to tweak the amount of memory allocated for the permanent generation, and this is done with this command line parameter:
> -XX:MaxPermSize=128m
Which will set the maximum size of our permanent generation to 128 megabytes - tweak it to fit your needs.
With these parameters properly applied to the jvm that runs your application server, your chances of running into a PermGen OutOfMemoryError will be considerably lessened.
Take care!
WIDGETIZE
Anonymous # 14. March 2007, 05:33
Will try this out. Have been plagued with OutOfMemory errors on development boxes recently with 5-15 different applications deployed. And almost always occurs on a new deploy.
If it works ... thanks in advance.
karmazilla # 14. March 2007, 09:30
They have deployed it on one of our production servers with, I think, is under medium load. There's two sides to the test: How will these GC settings affect the overall perfomance of the server? And will it now be able to run for more than a week without having to be restarted?
If I remember it, I'll let you know how it turns out. If it fails misurably, I guess I'll learn the hard way
Anonymous # 14. March 2007, 10:59
I was haunted by PermGen OutOfMemory for years, but have never been able to solve it. I surfed the whole web to find the solution, but never got one.
It is strange that there is no mention of this trick on sun's site!?
Anonymous # 14. March 2007, 14:20
"The standard garbage collector can't collect in the permanent generation"
I don't know every jvm, but the 1.5.0_10 (and 11) 64-bit Sun (server) JVM _does_ class unloading with the default parallel garbage collector.
karmazilla # 14. March 2007, 17:44
We use 1.5.0_06-b05 on the production server that I'm deploying to, and it's a 32-bit machine.
karmazilla # 15. March 2007, 15:36
Anonymous # 16. March 2007, 09:32
Hi,
there is alternative solution - use another JVM (BEA or IBM).
Regards,
Oleksandr Alesinskyy
Anonymous # 19. March 2007, 10:06
I'd recommend to upgrade to the latest JVM. At release 10 there were heavy bugfixes of garbage collecting subsystem.
karmazilla # 19. March 2007, 14:38
karmazilla # 22. March 2007, 14:22
Anonymous # 14. May 2007, 09:16
incredible! much respect
Anonymous # 21. May 2007, 07:49
Would have been to good to be true...
Thanks for your effort and article, but it didn't work for me here. Adding the JVM OPTS described to my Tomcat 5.5.20 startup script only now lead to the fact, that I can't undeploy my application because the ANT Task Undeploy is not able to remove log4j.properties from WEB-INF/classes and so another redeploy is not possible as the application still exists under webapps/myapp.
The memory was not released when the undeploy occured. (which might stem from the fact that log4j.properties was not removed)
Anyway thanks for this contribution. Hope that I will find the culprit. :)
karmazilla # 22. May 2007, 07:25
Anonymous # 22. October 2007, 00:55
I modified my eclipse.ini file as per your recommendation and it has not crashed for two days straight (compared to every 15 minutes). Keep up the good work. (Below are my eclipse.ini settings)
-showsplash
org.eclipse.platform
--launcher.XXMaxPermSize 256m
-vmargs
-XX:MaxPermSize=256m
-XX:+UseConcMarkSweepGC
-XX:+CMSPermGenSweepingEnabled
-XX:+CMSClassUnloadingEnabled
-Xms40m
-Xmx256m
karmazilla # 6. November 2007, 15:58
cd to the eclipse application directory (most likely /Applications/eclipse) and just type:
emacs $(find . -name eclipse.ini)
To edit the memory settings for eclipse without having to worry about where the file actually is.
Anonymous # 8. January 2008, 04:38
Don't mean to start (yet) another IDE war ... My NetBeans IDE never experiences those out of memory errors a couple of you are talking about here. And, the most recent version of NetBeans IDE 6.0 is very much improved. I find it much better than Eclipse.
Anonymous # 10. January 2008, 11:25
On CentOS4.6, JDK 1.5.0_12_64bit and Tomcat 5.5.23 with a Spring/Hibernate/CGlib application, the workaround doesn't help.
15 restarts of the application, and the PermGen is full.
Anonymous # 25. August 2008, 17:36
I tried your solution. I added it to my registry in windows. It has been one day and no PermGen yet. Crossing my fingers.
Anonymous # 8. October 2008, 05:25
I had to add the vm parameters in Eclipse's Preferences under Installed JREs.. Edit..., and this solved the Tomcat-in-Eclipse permgen overflow problem. Thanks!
Anonymous # 20. January 2009, 07:34
These are default netBeans 6.0.1 parameters. still i can see permgen. jvm is 1.6.0_02-b06
VM arguments:
-Xss2m
-Xms32m
-XX:PermSize=32m
-XX:MaxPermSize=200m
-Xverify:none
-Xmx367m
-XX:+UseConcMarkSweepGC
-XX:+CMSClassUnloadingEnabled
-XX:+CMSPermGenSweepingEnabled
Anonymous # 12. February 2009, 10:19
try working with this option:
sun.rmi.dgc.server.gcInterval
sun.rmi.dgc.client.gcInterval
sun.rmi.dgc.checkInterval
Anonymous # 9. June 2009, 10:35
Thank you for all the info, but just a question, I thought that when a FullGC is done the PermGen zone is also collected and cleaned, isn't this right?
If it is, what will be the difference between doing a fullGC and the JVM parameters you suggest?
Thank you!
Anonymous # 14. June 2009, 12:13
I have tried with the option which was mentioned and monitored using Jconsole. Based on the jconsole monitoring, the heap memory was gc very often but the non-heap memory (perm gen and code cache) wasn't gc at all. Although increasing the perm gen memory can delay the out of memory from happening, it does not completly eradicate the issue.
Anonymous # 17. July 2009, 15:38
How do I configure this for my setDomainEnv.sh file for Weblogic 10.0
I adjust my memory setting and I can get up to 10-12 deployments.. but then I get PermGen, and then I know its not unloading class files, because all my webservices will then start to get handshake_failure exception.
I really need this fix.. I spend about 2 hours each time trying to get the server back up and exceptions resolved.
Anonymous # 28. July 2009, 09:12
I recommend reading this post:
http://blogs.sun.com/fkieviet/entry/classloader_leaks_the_dreaded_java
Anonymous # 15. September 2009, 14:41
Christian,
thank you very much for the solution. These annoying errors on tomcat have driven me almost crazy - but at least I found your comments!!
Anonymous # 29. October 2009, 13:27
Well, not for production but for development those settings were OK.
So (for development!!!) I start server with:
set JAVA_OPTS= -Xdebug -Xnoagent -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n %JAVA_OPTS% -Xmx512m -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256m
thanks anyway