Permgen: Our Workable solution.
Sunday, 31. August 2008, 00:09:51
The majority of the visitors to this blog, I just noticed, come here because they are interested in my permgen saga:
The last installment concluded with newfound hope in the JDK6 profiling tools and a promise to post back as more knowledge was uncovered. Well... that was a while ago.
So, why haven't I spent time blogging about all of the exciting findings I must have made along my quest to solve this problem, since the 29th of September, 2007? The answer is simple: I have not spent any time at all on this problem. The reason for this lack of effort is simple: permgens have not been a problem to me since about that time.
To properly explain what has happened, let me draw a parallel to writing code - multi-threaded code, to be presice: What is the primary concern in multi-threaded/concurrent code? Shared mutable state. The easiest way to solve this problem? Do not share state, or make shared state immutable.
Neither of these proposed solutions actually solve the problem of shared mutable state. Instead they remove it; make the problem moot - a non-issue.
I have not solved the permgen problem; I have made it moot. Well, the operations department have made it moot and I have been cheering from the side-line.
The Moot Permgen:
Recall the original problem: every so often we redeploy a Java web application on our application servers, without restarting said servers. An intricate object graph prevents the old class files from being unloaded, this equates to a memory leak in the permanent generation of the JVM that eventually causes the server to die with an OutOfMemoryError.
The thing that gets the hair ball rolling here, is the deployment without restarting the server. To make this problem moot, all we have to do is to always restart the server every time we deploy an application.
Simple, right?
It's like when you go to the doctor and say "It hurts when I do this" and the doctor says "Well, don't do that." Same exact thing. Deployment without a restart: Just stop it!
The Other Way around the problem:
We basically changed the way we deploy applications. Not in a big way, just some small adjustments.
First and foremost, we don't deploy anything without restarting the server - but you already knew that.
Second, we deploy less stuff on our servers - the fewer apps you run on a server, the faster it'll restart. Preferably, we would only run one application per JVM- this way, the restartes would only affect a single application.
Lastly, we're moving our production systems to load-balanced setups. When an application is running on two (or more) servers behind a load-balancer, then restarting a server won't incur any down-time.
It's not rocket science. It's a practical and workable way to simply avoid having to deal with the dreaded OutOfMemoryError: permgen.
The last installment concluded with newfound hope in the JDK6 profiling tools and a promise to post back as more knowledge was uncovered. Well... that was a while ago.
So, why haven't I spent time blogging about all of the exciting findings I must have made along my quest to solve this problem, since the 29th of September, 2007? The answer is simple: I have not spent any time at all on this problem. The reason for this lack of effort is simple: permgens have not been a problem to me since about that time.
To properly explain what has happened, let me draw a parallel to writing code - multi-threaded code, to be presice: What is the primary concern in multi-threaded/concurrent code? Shared mutable state. The easiest way to solve this problem? Do not share state, or make shared state immutable.
Neither of these proposed solutions actually solve the problem of shared mutable state. Instead they remove it; make the problem moot - a non-issue.
I have not solved the permgen problem; I have made it moot. Well, the operations department have made it moot and I have been cheering from the side-line.
The Moot Permgen:
Recall the original problem: every so often we redeploy a Java web application on our application servers, without restarting said servers. An intricate object graph prevents the old class files from being unloaded, this equates to a memory leak in the permanent generation of the JVM that eventually causes the server to die with an OutOfMemoryError.
The thing that gets the hair ball rolling here, is the deployment without restarting the server. To make this problem moot, all we have to do is to always restart the server every time we deploy an application.
Simple, right?
It's like when you go to the doctor and say "It hurts when I do this" and the doctor says "Well, don't do that." Same exact thing. Deployment without a restart: Just stop it!
The Other Way around the problem:
We basically changed the way we deploy applications. Not in a big way, just some small adjustments.
First and foremost, we don't deploy anything without restarting the server - but you already knew that.
Second, we deploy less stuff on our servers - the fewer apps you run on a server, the faster it'll restart. Preferably, we would only run one application per JVM- this way, the restartes would only affect a single application.
Lastly, we're moving our production systems to load-balanced setups. When an application is running on two (or more) servers behind a load-balancer, then restarting a server won't incur any down-time.
It's not rocket science. It's a practical and workable way to simply avoid having to deal with the dreaded OutOfMemoryError: permgen.
WIDGETIZE