Best programmers 28 times better than the worst? I'm not sure.
Wednesday, 18. March 2009, 22:24:33
In his book Facts and Fallacies of Software Engineering, Robert L. Glass cites from a research paper from 1968, that the best programmers can be up to 28 times better than the worst.
That paper is Sackman, H., W. I. Erikson, and E. E. Grant. "Exploratory Experimental Studies Comparing Online and Offline Programming Performance." A brief, but publically available, treatment of it can be found here and most like elsewhere too if you ask Google.
Now, I know it's bad form to critisize a paper I have not read, but you need an ACM account to get at the real thing, so I will instead base my opinion the linked blog post, and otherwise be brief about it.
I have two points of critique, and the first and most obvious one is regarding the age of the paper. It was published 41 years ago and its primary purpose was to figure out whether time-sharing or batch processing systems were the most productive - which it did. And then, "almost in passing," apparently, they present numbers showing that the best programmers in their study were up to 28 times better than the worst.
I have to wonder whether the numbers have changed. Have the worst programmers gotten better, or worse? And are we even able to distinguish such a number from a difference among the best programmers? I have absolutely no idea. Most people I know, myself included, wasn't even born 41 years ago.
The second peeve I have with the unkown content of that paper, or perhaps rather the 28:1 quote in the context of that paper, is the dataset. While I don't know how many programmers participated, I do know that they were tasked with solving two programming problems.
While such a method is able to show peaks such as "up to 28 times better," it fails to show whether this grade-28 master is able to keep ahead in the long run. The two programming problems they were presented, were pretty small in size. Especially compared to many of the projects that are going on in the real world right now.
Is the master able to keep his times-28 advantage throughout a project that takes a year to complete - or at least is "normally" estimated to a years worth or work? Assuming they will finish at some point, how long will that same project take the worst programmer to complete?
It is possible that operating under the assumption that the worst programmer will, eventually, finish the project, is a pretty far stretch. But so do the "28 times better" seem when considered in the long run. Given the numbers are based on a mere two programming problems, the 28 times may have been a fluke. Maybe the guy was just lucky - you can argue that it takes skill to be that lucky, but still.
I once spent one and a half months hunting nasty memory leak (and I don't ever want to do that again), but once I found it I just had to change two lines and it was fixed. I could have been lucky and found those lines after a week, or never introduced the leak in the first place, but unfortunately it didn't happen like that.
In conclusion, I don't think that paper has the material to justify "up to 28 times better" as fact. The difference is there, for sure. And many people seem to feel in their gut that it may be around 10 times better. That's also a nice round number, but I won't claim it's a fact though.
That paper is Sackman, H., W. I. Erikson, and E. E. Grant. "Exploratory Experimental Studies Comparing Online and Offline Programming Performance." A brief, but publically available, treatment of it can be found here and most like elsewhere too if you ask Google.
Now, I know it's bad form to critisize a paper I have not read, but you need an ACM account to get at the real thing, so I will instead base my opinion the linked blog post, and otherwise be brief about it.
I have two points of critique, and the first and most obvious one is regarding the age of the paper. It was published 41 years ago and its primary purpose was to figure out whether time-sharing or batch processing systems were the most productive - which it did. And then, "almost in passing," apparently, they present numbers showing that the best programmers in their study were up to 28 times better than the worst.
I have to wonder whether the numbers have changed. Have the worst programmers gotten better, or worse? And are we even able to distinguish such a number from a difference among the best programmers? I have absolutely no idea. Most people I know, myself included, wasn't even born 41 years ago.
The second peeve I have with the unkown content of that paper, or perhaps rather the 28:1 quote in the context of that paper, is the dataset. While I don't know how many programmers participated, I do know that they were tasked with solving two programming problems.
While such a method is able to show peaks such as "up to 28 times better," it fails to show whether this grade-28 master is able to keep ahead in the long run. The two programming problems they were presented, were pretty small in size. Especially compared to many of the projects that are going on in the real world right now.
Is the master able to keep his times-28 advantage throughout a project that takes a year to complete - or at least is "normally" estimated to a years worth or work? Assuming they will finish at some point, how long will that same project take the worst programmer to complete?
It is possible that operating under the assumption that the worst programmer will, eventually, finish the project, is a pretty far stretch. But so do the "28 times better" seem when considered in the long run. Given the numbers are based on a mere two programming problems, the 28 times may have been a fluke. Maybe the guy was just lucky - you can argue that it takes skill to be that lucky, but still.
I once spent one and a half months hunting nasty memory leak (and I don't ever want to do that again), but once I found it I just had to change two lines and it was fixed. I could have been lucky and found those lines after a week, or never introduced the leak in the first place, but unfortunately it didn't happen like that.
In conclusion, I don't think that paper has the material to justify "up to 28 times better" as fact. The difference is there, for sure. And many people seem to feel in their gut that it may be around 10 times better. That's also a nice round number, but I won't claim it's a fact though.
WIDGETIZE
Anonymous # 16. August 2009, 03:11
I don't doubt that the best programmers are *at least* 28 times better than the worst programmers. A really bad programmer can spend a year writing a lousy program that a really good programmer can auto-generate by writing a program in less than a day that generates a better version of the lousy program.
Christian Vest Hansen # 16. August 2009, 12:39
Anonymous # 16. August 2009, 15:07
It's difficult to draw hard and fast conclusions here because there are so many variables. For example, in today's economy the difference may not be as great as it was a few years ago because many of the "bad programmers" are currently unemployed as programmers.
What I can say is that I've encountered people who are "negatively productive". That is to say that they create more work for others.
I can also say that I've worked on systems which were architected to the technology, not the problem domain. It's been my experience that the former require at least an order of magnitude to extend and maintain over those where domain level abstraction prevailed. In essence one creates a new language (call it a DSL if you like) for solving problems in the application domain as opposed to a solution engineered to solve a single problem. When one considers the total cost of ownership, the costs disparity is further magnified.
I can't speak for what a "common problem" is although I have a feeling it refers to smaller, simpler problems. It could be that when it comes to less involved projects the disparity is not as great although I will say that I've seen cases where one person solved a problem by typing a single line of "awk" where another person took a day or two to write a custom C program. Part of the process is recognizing what code *doesn't* have to be written.
In large systems though, software tends to scale hierarchically, so finding a good abstraction/representation at the foundational levels can easily yield order of magnitude payoffs as vertical scaling occurs.
Anonymous # 9. October 2009, 21:35
I think the 28 times is an understatement when you come to a real world situation. I have to wonder what the nature of the memory leak is that took 9 and a half months to find. Surely you could have coded up a substitute memory allocator in a day that recorded a stack trace for each memory allocation and logged the unfreed memory at the end?
Or better for a complex project, surely you already used a custom memory allocator that reported the existence of a leak on exit so you found the leak as soon as it was put in?
The skill difference arises from both raw talent and experience. A brand new developer with a lot of talent will perform a bit better than an average developer with a lot of experience (and will completely outclass a bad developer with a lot of experience), but a developer with a lot of talent and a lot of experience will draw on that experience to solve problems in far less time (possibly by solving other intermediate problems first) than either the average experience, or the talented inexperienced programmer.
Christian Vest Hansen # 11. October 2009, 16:58
Originally posted by anonymous:
It was one and a half months. The nature was that dynamically creating log4j loggers caused the system to leak interned strings in the permanent generation of the JVM heap. The leak didn't manifest under our regular stress tests because they weren't similar enough to the real world, so we had to code up a new one. Then running this test took about four hours and this slowed down our search for the commit that introduced the bug. We were also slowed down by disbelief and lack of appreciation for the nature of the problem, and experience in dealing with memory leaks in general.
Anonymous # 30. December 2009, 05:00
I have witnessed many times large teams of mediocre programmers mess up projects - both in schedule and quality - that could have been done well if the team was made of fewer good programmers. As Greg Schmidt rightly said, "Total Cost of Ownership" is the right key for comparison.