Software Development

So we hear desperate cries for a silver bullet--something to make software costs drop as rapidly as computer hardware costs do.

In the past two years developing Project V in my free time and in past projects before that, one of the main things I've noticed is that asking the wrong question is usually much worse than finding the wrong answer. A wrong question will never let you reach the correct solution and right here, the NSB paper is off to a bad start. The rest of the paper is invalidated at only the third sentence.

Here, Brooks is asking if there isn't something to make software costs drop as much as hardware costs do. Costs are one thing, but what each product does is another. Let's look at the Intel PIII 550 released in May 1999. It cost $744 each for 1000 units. By Feb 2000, it cost $193 each for 1000 units. In 9 months, the price went down by $551. By Feb 2001 (two years later), you could buy a 1.5GHz PIV for $637 and 1.7Ghz PIV a few months later for $701, which dropped to $352 the very next month. So the price drops are quick. Within a few months, the prices drop a lot and within a few years, that product is likely to have been replaced by something else.

Next, we get to software. You can look around, but prices don't fluctuate that much. If you look at NewTek's Lightwave 3D package, it used to be over $1000 many years ago, but for the past several years, it's stayed constant at around $800 for several versions.

Unfortunately, this is an incorrect comparison. We're only looking at the costs. It looks like costs of hardware are getting cheaper while getting faster. But software seems to remain constant even with these hardware advantages. But that's the wrong question. It's the wrong thing to look at.

Let's look at the functionality. Processors do the same thing as they've always done. Since the 8088, it still does those same operations. Some were added over time. The 286 added 16bit protected mode. The 386 added 32bit operations and registers. The Pentium added MMX and the PIII added SSE. But these are all the same thing. Registers and opcodes. More cache, more registers and more opcodes that operate on those new registers. Ultimately, there is nothing new there. What is new is the speed. And that's admirable. But there's no extra functionality other than that speed. There's nothing in those new opcodes that you couldn't do before. The doubling of transistors is more of the same. If we apply this doubling of resources to software, we can already do that. How many images can a drawing package handle? 10? Then you've just achieved a ten-fold increase in functionality within seconds. 10 * 2years * 365days * 24hours * 60minutes * 60seconds = 630720000. So we have here a 630720000 times increase in productivity or reduction in costs compared to the two years that it takes for hardware. But we don't think that way. Brooks picks and chooses what he wants to compare. Take into account the reproduction costs. Hardware can decrease costs because there's actually a cost involved. In software, how do you reduce costs of reproduction when that cost is near zero already? Again, invalid comparisons.

The fact that software is slower than hardware speeds up could be an indication that we can insert functionality faster than hardware speeds up. It could mean that we don't optimize our code very well, but that's beside the point. If we talk about features of speed in hardware vs. functionality in software, software clearly has more abilities than ever before. If you look at just the amount of features included in software, there's a clear indication that software is using whatever resource is available. The multi-core crisis is another indication of this. If there's extra resources to be had, programmers want to use it. As for doubling the amount of features, I don't think that's necessarily a goal either. So Brooks goes to the opposite end and asks if we can't reduce the costs just as hardware. That's a trap one must be very careful not to fall into. Because when you go there, you forget exactly what it is you're comparing. The costs must be attributed to the same things. In hardware, it's number of resources (transistors). In software, we already outpace hardware by several orders of magnitude (in the billions or trillions). Show me one programmer that cannot use most of the resources on a machine within seconds! So the cost comparison is invalid as stated.

Here's a reference to CPU price charts used for the pricing info above.

Not only are there no silver bullets now in view, the very nature of software makes it unlikely that there will be any--no inventions that will do for software productivity, reliability, and simplicity what electronics, transistors, and large-scale integration did for computer hardware.

I'd like to use a favourite technique and to replace words in quotes to remove any preconceived notions. When you label things as A and B, you don't get the same attachment as you do for terms like hardware and software. So let me requote that.

Not only are there no silver bullets now in view, the very nature of A makes it unlikely that there will be any--no inventions that will do for A's productivity, reliability, and simplicity what X, Y, and Z did for B.

Note that it's an EXACT duplicate of the original except for replacing some terms with capital letters. Now I'll requote the above, but add a phrase at the end.

Not only are there no silver bullets now in view, the very nature of A makes it unlikely that there will be any--no inventions that will do for A's productivity, reliability, and simplicity what X, Y, and Z did for B even when A possesses all the power and functionality of B.

If you read the above for the first time, would you not say there's some logical fallacy in that statement? How can A be less reliable than B if A can do everything that B can do? And we know that software can do everything that hardware can do. We can simulate it if we wish. There is no setup found in hardware that cannot be replicated in software for use by software. So there's zero reason that we can't use the same techniques that make hardware so reliable in our software. At least, not on a technical basis. Other reasons will come up later.

The next line in the paper is this:

We cannot expect ever to see two fold gains every two years.

What can't see a two fold gain? Reliability? Nope, we can replicate hardware techniques. Productivity? Nope! Simplicity? Nope! If it's found in hardware, we can replicate it in software. The "two years" is what's key here. Brooks doesn't say, but it seems to indicate Moore's Law where the number of transistors doubles every two years. Software can double its usage of transistors right now. Today. Yesterday. 50 years ago. So the above statement is false. It's tempting to compare the number of transistors for hardware and reliability in software, but it's not a fair comparison. They're different things. Look at the above quote again and see how there's no qualificaition of what gains are made.

Please note that I'm showing how Brooks' arguments are false and am NOT going to the opposite spectrum. I don't need to show that there is a Silver Bullet. Not having found one does not mean that it does not exist. Ironically, I think Brooks has mentioned several times what this silver bullet is, but doesn't want to admit it.

Again, about hardware:

No other technology since civilization began has seen six orders of magnitude in performance-price gain in 30 years. In no other technology can one choose to take the gain in either improved performance or in reduced costs.

We get a little bit of qualification, yet it still falls short. Are we comparing performance or reliability? These aren't thing you can compare and then put a price tag on and say there's something out of whack. These are two different topics. If Brooks wants to talk about performance, then so be it. But if he wants to talk about reliability, that's fine too. He can even talk about both. But he must talk about each item as they relate to both software and hardware. Note that in the above quote, he does NOT talk about software. This leaves the impression that there is some equivalent in software that is somehow inferior or that hardware has some magical property. The kinds of statements above serve only to inject a preferred state of mind into the reader. Also note the issue of cost again when it doesn't belong. Mass production of software duplication is already at zero. Software already has hardware beat in terms of cost as it relates to duplication.

The essence of a software entity is a construct of interlocking concepts: data sets, relationships among data items, algorithms, and invocations of functions. This essence is abstract in that such a conceptual construct is the same under many different representations. It is nonetheless highly precise and richly detailed.

Now we get to Brooks famous essential complexity. These are things that must be dealt with no matter what. Brooks says that software has essential building blocks that cannot be avoided. Essense is the word he chooses. And the above quote shows the items he believes are essential to a software entity. Different tools can reduce accidental complexity, but software must always DO something. So that part, the DOING and how it is constructed, is essential and is where much complexity comes from according to Brooks (quote found below) and cannot be removed, hence No Silver Bullet. More on this below.

I can use Project V or simply nodes as in Lightwave to disprove the above quote where software must have the concepts mentioned. The quote is totally false and I've shown it with two examples. An algorithm is an imperative construct. It's a set of operations that must be done in a certain order. Even in functional programming, there is an order to the sequence of functions through its nesting mechanism though there is potential for more flexibility. This is not the only way to describe software. You can look at my previous blog entry for an example where I demonstrate no algorithm and no invocations of functions. None! Not one function can be seen. Yet, it's still software. This means that algorithms and functions are not essential as Brooks thought, but are accidental. Also look at the node system in NewTek's Lightwave 3D package. It is definitely software construction usable by artists. This means that Brooks' essence of software is misguided at best. You can indeed use the things he says to build software, but you don't have to. It's not the whole picture. Any assumptions made on the quoted claim must be tossed aside.

He does try to say that these things can have different representations, but just lumps it in with the rest without as much of a comment. As a professional developer, this is a statement that I cannot endorse. If different tools yield better results depending on the circumstance, then Brooks' statement about what he believes are the essense of software cannot hold up.

I believe the hard part of building software to be the specification, design, and testing of this conceptual construct, not the labour of representing it and testing the fidelity of the representation. We still make syntax errors, to be sure; but they are fuzz compared with the conceptual errors in most systems.

If this is true, building software will always be hard. There is inherently no silver bullet.

This all sounds good. When you take into consideration everything I've mentioned before, it makes for a weak argument though. I too believe that the hard parts are the things that come before implementation. These lump in both necessary and unnecessary components though Brooks doesn't see it that way. The best people can reduce and choose the best ways to accomplish the task. I also believe that the best way to implement this task is difficult. Not the typing part, but deciding what to use. It's difficult to tell what part Brooks is talking about. To me, it seems like he's saying that anyone that knows a language can deal with it. The higher level thought processes that form your design are not considered. I would agree with the notion that once you know a language, this isn't what is going to stop you. What I have a problem with is that it's not clear what point he's trying to make and then jumps the gun to use this to support the biggest conclusion in computing history.

What he seems to be basically saying is that representing and inserting the software onto the computer is easy. I agree with that. This is the typing part. Anyone that knows a language can type stuff in. What goes into that software is a different story. And the tools you have that enable this also affect what you do and with what ease. So the conceptual errors are indeed one factor, but the tools you use that enable these concepts are also involved. If we have tools that better suit the concepts, then we'd have more reliability and simplicity, no? So these tools are not fuzz compared to the concepts. Brooks says that high level languages pretty much resolved any accidental properties of inserting software and can't get much better. I disagree. How can he know? How can a man who believes that function invocation is an essential part of software know? Here, it's not so much that what he says isn't true (and it isn't), but that there's a credibility problem and that's there's nothing supporting his argument. His argument is shown to be false in further topics below.

Now, we get to his four pillars of the software holy grail. Complexity, conformity, changeability, and invisibility.

Complexity. Software entities are more complex for their size than perhaps any other human construct because no two parts are alike (at least above the statement level). If they are, we make the two similar parts into a subroutine--open or closed. In this respect, software systems differ profoundly from computers, buildings, or automobiles, where repeated elements abound.

First, don't let Brooks type in vehicle recall in Google. I realise the paper is from the 80's, but if you're going to quote an example, you should look it up. And he shouldn't look here either. That leaves buildings and computers. Let's not tell him about the Pentium divide bug. I've heard of a few other recalls as well in different hardware. Didn't the XBox have a faulty power supply at one time? And finally, there are buildings. I've worked in construction in the past year as my blog readers know and I can tell you building codes are ridiculous. Tell anyone in any large city if they can get competent people to do work. Large buildings used to fall down for numerous reasons. In the past, it was because the land was not inspected properly and the type of weather in the area was not taken into account. This includes frost levels and earthquakes. Buildings are ONE type of "things" that humans have been doing for thousands of years at least. And anyone that has tried doing repairs will laugh at anyone in software trying to say that it's so much easier in construction.

I also object to Brooks' assertion that software components are different. What usually happens is that we need to reuse something, but in a slightly different way. Functions don't allow for this flexibility. That's where generics come in. You can use the same functions for different data types. Even so, it'd be nice to have more flexibility. This means there is still more we can do. I'm working on that right now. It's still up in the air how this will turn out, but giving up is not the answer. In any case, it's been shown over and over again that reuse is and can be done. Again, Lightwave has shown this and you can read this chapter in J. Paul Morrison's book. It'd also be curious to know why people like extensive libraries which are often attributed to Java's success.

Likewise, a scaling-up of a software entity is not merely a repetition of the same elements in larger sizes; it is necessarily an increase in the number of different elements. In most cases, the elements interact with each other in some nonlinear fashion, and the complexity of the whole increases much more than linearly.

The irony knows no bounds. If that's the problem, then fix it. If it's possible in hardware, it's possible in software. Perhaps it's the tools we use to connect these different parts that are at fault. Brooks gives no consideration to this option.

The complexity of software is an essential property, not an accidental one. Hence, descriptions of a software entity that abstract away its complexity often abstract away its essence.

Almost every sentence can be disproved. How can Brooks say that complexity is an essential property of software? He makes an assumption. There is nothing in the paper backing up his claim. I already showed that you don't need to use functions. Something Brooks thought was an essential part of software. How then can we take his claim of software's essential properties at face value when we know he's been wrong about this before? We can't.

We can go further though. We can show that his claims are false. Let's assume that complexity is NOT an essential property, but an accidental one. This would mean that abstracting away complexity would not abstract away its essence, but rather its accidental properties. That would be a huge plus. Is it possible to show that there is some accidental complexity in software? Yes. It's rather easy too.

Take a function that calls or invokes two other functions where the data from the first function is passed to the second (A is the primary function and B and C are the two delegate functions). This primary function A must know properties of two other functions (B and C). The two delegate functions (B and C) must also return to the invocator A either in execution or in data (or both). Is it possible to simplify things? What we can do is not use functions and instead of having the first called function B return values, it could pass it on directly to the second function C. This isn't possible in conventional languages. What we have is A->B->C in a linear fashion. There are two connections instead of four. A also need not know anything about B. It can connect to anything that supports that kind of data. In function form, A must know the name of what it calls. Not so when we chain the functionality in this manner. A has no clue who it connects to. And it doesn't care. It also only connects to ONE entity, not two. So its complexity is reduced in half just with that. And since it need not know what it connects to, then complexity is driven down even more. What's more, in some languages, the only use for A is to connect the two called functions. With our new scenario, we can dispense entirely with A and only use B and C with B->C thus simplifying things further. Some languages have names for this kind of data flow construct like arrows and monads. That's where those things come from.

There's a real reason why these constructs are starting to appear. There is more on the horizon too. Brooks perhaps had no way to know, but these things did exist over 30 years ago. Maybe the popularity was at fault. Still, excuses aside, it ultimately means that there's a lot of complexity in software that is not essential and that Brooks' claim is wrong again.

Another thing to note which is very important is that software can be built in two different ways. There are only two fundamental properties of software. Imperative and data flow. Everything else is built on top of these properties. You can read up on it here. What Brooks talks about is the imperative way. Data flow is like the hardware way. They can both coexist in peace. Unfortunately, Brooks' paper is more of an attack on imperative than anything else. It does not consider data flow.

About conformity,

These differ from interface to interface, and from time to time, not because of necessity but only because they were designed by different people, rather than by God.

This is a weak argument. We had the same problem with DOS. Too many hardware interfaces. MacOS, AmigaOS and other OS's didn't have these problems. Today, even Windows API is usable. The problem mentioned by Brooks does show up all the time though. It's a matter of getting together and resolving the issue as we did in the past. The web is one such case that is in dire need of a cleanup. As Brooks says, it's a human problem, not a necessary problem. We've solved it in the past and can solve it again. It's just a matter of doing it. Does this mean conformity is solvable? Absolutely. It's been done in the past on numerous occasions. Next.

About change,

Buildings do in fact get changed, but the high costs of change, understood by all, serve to dampen the whims of the changes.

Brooks' argument amounts to physical things costing more than software. Isn't the NSB supposed to argue the opposite? That physical things can cut costs? Arguing on both sides of the fence is not impressive.

At point in this section is how software can live forever, but physical things cannot. There's one thing that Brooks misses though. With the side effect of physical things breaking down, the people who build them must come up with ways to reproduce the thing they are building from scratch. If we were to do the same with ONE particular piece of software, I can guarantee that prices would drop rather fast and that reliability would go up. The problem is that initial costs would be grossly disproportional to the gains. With hardware, you don't have a choice. Look at what it costs to build a fabrication factory. It's in the billions of dollars. Who in their right mind would do the same for software? There's an issue of perspective that's missing in Brooks' arguments.

Software is invisible and unvisualizable.

This is clear evidence that Brooks has never heard of data flow programming. While not currently popular, it is gaining a LOT of traction. And it is visual. Again, NewTek's Lightwave 3D node editor is very visual. It's used for 3D, but can be generalised quite easily by providing your own nodes. I couldn't find a good Lightwave screenshot, but Blender has added nodes, so here's a preview of that. So that's one very concrete example where Brooks is wrong and that's enough for this point. Perhaps he was talking about representing the current way of programming and converting that into something visual. If so, that's a really shortsighted view.

I just have to say something about the following.

The reality of software is not inherently embedded in space.

This shows a clear indication that Brooks does not understand the fundamental concepts that make computing possible. If he did, he'd have a solution to concurrency (that comes for free with one of these concepts and further reduces complexity). One concept is embedded in time, the other in space (exactly what Brooks says is not inherent). Imperative is embedded in time and there's a very real reason that it's called programming. Programming, in any field, always deals with timings. TV programming deals with when shows come on. Same thing for theatre. In computer programming, timing plays a crucial role as well. It's where you state what operations are done before the other.

Software can also be embedded in space. It's the flip side of being embedded in time. This is how hardware and data flow works. You can look at my last blog entry or the Blender preview just mentioned and see a couple of very visual examples of software in space. You can even follow the data as it travels from point A to point B. It doesn't get more visual than that.

Anyhow, this is getting tedious. I can pick almost any sentence from this paper and show real concrete examples of how it's wrong or show simple logic of how Brooks' arguments don't hold water.

At this stage, the four primary arguments of Brooks', complexity, conformity, changeability, and invisibility are unfounded. Please consider that even though I believe the opposite of NSB's claims, I am not arguing FOR that in this blog entry. I'm only showing how the NSB paper is false. Nothing more.

I'm skipping the past breakthroughs and a few other sections, not because they're necessarily valid, but because they show improvements (or not) and do nothing toward the argument of NSB. The only thing I'd like to mention is that programming languages are not the only way to build software. Brooks does not consider this and seems to not be able to. Just because one person or many cannot see different ways of doing things does not mean they don't exist.

You can read for yourself what Brooks says about AI. I agree that the term is used for limited tasks. It doesn't mean what we know as intelligence. At the same time, I do think that if we understood how to produce creations such as our brains, we would well be on our way to a silver bullet. Being able to grow software is something that would be remarkable. Our brains are incredibly reliable considering what it goes through. We do not maintain it like we do our cars or houses. It takes care of itself. If it does break down, it's usually due to some outside influence like viruses, injuries or other physical events. The brain is also built using a data flow mechanism with the use of interconnected neurons. If anything, AI should eventually produce some new discoveries. We know what's possible just by looking at another human being. Sure, there's potential for evil, but there's potential for greatness too. Regardless of the NSB paper, you can look at yourself in the mirror every day and know that there's something better waiting for us in software. Unless you don't like what you see.

Actually, I'll end it here. The rest of the paper considers things that have helped or not helped instead of focusing on the argument at hand. ADA aside, we should use critical thinking before accepting an argument of such magnitude. Besides, any argument that leads to the negative should be dispensed with. Not because it may not be true, but because one negative thought can be more destructive than any real life obstacle. It's even more devastating when the arguments end up actually being false such as with the No Silver Bullet paper. No other paper has had a more detrimental effect on computing than that one. May it Rest In Peace once and for all. For us. And for our future limited in its greatness only by our imaginations.