Permgen: Our Workable solution.
Sunday, 31. August 2008, 00:09:51
The majority of the visitors to this blog, I just noticed, come here because they are interested in my permgen saga:
The last installment concluded with newfound hope in the JDK6 profiling tools and a promise to post back as more knowledge was uncovered. Well... that was a while ago.
So, why haven't I spent time blogging about all of the exciting findings I must have made along my quest to solve this problem, since the 29th of September, 2007? The answer is simple: I have not spent any time at all on this problem. The reason for this lack of effort is simple: permgens have not been a problem to me since about that time.
To properly explain what has happened, let me draw a parallel to writing code - multi-threaded code, to be presice: What is the primary concern in multi-threaded/concurrent code? Shared mutable state. The easiest way to solve this problem? Do not share state, or make shared state immutable.
Neither of these proposed solutions actually solve the problem of shared mutable state. Instead they remove it; make the problem moot - a non-issue.
I have not solved the permgen problem; I have made it moot. Well, the operations department have made it moot and I have been cheering from the side-line.
The Moot Permgen:
Recall the original problem: every so often we redeploy a Java web application on our application servers, without restarting said servers. An intricate object graph prevents the old class files from being unloaded, this equates to a memory leak in the permanent generation of the JVM that eventually causes the server to die with an OutOfMemoryError.
The thing that gets the hair ball rolling here, is the deployment without restarting the server. To make this problem moot, all we have to do is to always restart the server every time we deploy an application.
Simple, right?
It's like when you go to the doctor and say "It hurts when I do this" and the doctor says "Well, don't do that." Same exact thing. Deployment without a restart: Just stop it!
The Other Way around the problem:
We basically changed the way we deploy applications. Not in a big way, just some small adjustments.
First and foremost, we don't deploy anything without restarting the server - but you already knew that.
Second, we deploy less stuff on our servers - the fewer apps you run on a server, the faster it'll restart. Preferably, we would only run one application per JVM- this way, the restartes would only affect a single application.
Lastly, we're moving our production systems to load-balanced setups. When an application is running on two (or more) servers behind a load-balancer, then restarting a server won't incur any down-time.
It's not rocket science. It's a practical and workable way to simply avoid having to deal with the dreaded OutOfMemoryError: permgen.
The last installment concluded with newfound hope in the JDK6 profiling tools and a promise to post back as more knowledge was uncovered. Well... that was a while ago.
So, why haven't I spent time blogging about all of the exciting findings I must have made along my quest to solve this problem, since the 29th of September, 2007? The answer is simple: I have not spent any time at all on this problem. The reason for this lack of effort is simple: permgens have not been a problem to me since about that time.
To properly explain what has happened, let me draw a parallel to writing code - multi-threaded code, to be presice: What is the primary concern in multi-threaded/concurrent code? Shared mutable state. The easiest way to solve this problem? Do not share state, or make shared state immutable.
Neither of these proposed solutions actually solve the problem of shared mutable state. Instead they remove it; make the problem moot - a non-issue.
I have not solved the permgen problem; I have made it moot. Well, the operations department have made it moot and I have been cheering from the side-line.
The Moot Permgen:
Recall the original problem: every so often we redeploy a Java web application on our application servers, without restarting said servers. An intricate object graph prevents the old class files from being unloaded, this equates to a memory leak in the permanent generation of the JVM that eventually causes the server to die with an OutOfMemoryError.
The thing that gets the hair ball rolling here, is the deployment without restarting the server. To make this problem moot, all we have to do is to always restart the server every time we deploy an application.
Simple, right?
It's like when you go to the doctor and say "It hurts when I do this" and the doctor says "Well, don't do that." Same exact thing. Deployment without a restart: Just stop it!
The Other Way around the problem:
We basically changed the way we deploy applications. Not in a big way, just some small adjustments.
First and foremost, we don't deploy anything without restarting the server - but you already knew that.
Second, we deploy less stuff on our servers - the fewer apps you run on a server, the faster it'll restart. Preferably, we would only run one application per JVM- this way, the restartes would only affect a single application.
Lastly, we're moving our production systems to load-balanced setups. When an application is running on two (or more) servers behind a load-balancer, then restarting a server won't incur any down-time.
It's not rocket science. It's a practical and workable way to simply avoid having to deal with the dreaded OutOfMemoryError: permgen.
WIDGETIZE
Anonymous # 19. September 2008, 17:30
Well, I hoped you wouldn't get to this at the end. That's what I had thought as well, but thought to myself: "c'mon, this is suppose to work!!". You let me without much hope. I had a look on the kieviet's solution, and it appears to be quite complex...
emerson
karmazilla # 19. September 2008, 19:49
Anonymous # 1. September 2009, 17:36
Can you post the "small application that used the MemoryMXBean to plot the NonHeap memory stats to an array list every minute, and then print the list, CSV formattet, on a JSP file"? It would be very helpful for a number of use cases.
karmazilla # 1. September 2009, 20:07
Originally posted by anonymous:
You are probably better off with VisualVM or visualgc. Otherwise if you don't mind deploying to the same app server, or even in the app itself, then it's dead easy to fetch the relevant mbeans from java.lang.management.ManagementFactory.
Anonymous # 15. October 2009, 13:57
http://community.eapps.com/showthread.php?p=326
karmazilla # 15. October 2009, 19:28
Originally posted by anonymous:
Nope. Does not work!
I updated my original blog post a while back to state as much: http://my.opera.com/karmazilla/blog/2007/03/13/good-riddance-permgen-outofmemoryerror
In the updated article, I present a short list of facts about this problem and one of the items on that list is "None of these sources [of the permgen problem] can be fixed with GC flags."
The only reliable solution I've found is a reboot on deployment.