Notes to self

Writing correct concurrent code is hard. Writing correct and performant concurrent is significantly harder still, but before getting to that, the code must first be correct. Actually, even correct in the non-concurrent sense. Simply put, the set of potential bugs - failure modes - in a multi-threaded program is the set of failure modes for the single-threaded program, plus all of the concurrency specific failure modes.

So, a correct multi-threaded program implies a correct program in general. But in this post, I will only focus on the part that is specific to correct concurrency.

Getting concurrency right takes knowledge and analytical skills. I am an avid TDD practitioner, and I try to test-drive as much as I possibly can. However, in the world of concurrency we are faced with the problem that some bugs are simply impossible to write a test case for. Sometimes the timings required to surface a bug are just too tight and the chances too small, other times a bug is just does not exist on our specific combination of hardware and runtime.

Therefore, analyzing the code in terms of the concurrency guarantees provided by the underlying platform, is a key element of the path towards concurrency correctness. And it cannot be done without knowledge of the guarantees provided by the platform, whether it be the JVM, CLR or x86.

There are lots of failure modes specific to mutli-threaded programs. Many are quite esoteric, but three things stand out as the most basic problems: shared mutable state, publication and locking.

The first problem of concurrency is shared mutable state. There are cases where you can have deliberate under-synchronized shared mutable state, but those are as few as they are special. There are basically three ways to deal with shared mutable state:

1. Make state immutbale
   
2. Don't share state
   
3. Guard the shared mutable state by the same lock (two different locks cannot guard the same piece of state, unless you always use exactly these two locks)
   

The second problem of concurrency is safe publication. How do you make data available to other threads? How do you perform a hand-over? You will find the solution to this problem in the memory model of your platform and (hopefully) in the API. Java, for instance, has many ways to publish state and the java.util.concurrent package contains tools specifically designed to handle inter-thread communication. Make sure to give this problem ample focus; the bugs produced by unsafe publication can be extremely subtle, and it is easy to gloss over this problem and focus on locking instead.

The third (and harder) problem of concurrency is locking. Mismanaged lock-ordering is the source of dead-locks. Some dead-locks are practically guaranteed to always occur, while others require highly improbable timing. This problem can actually be generalized to everything that prevents a thread from making progress. Blocking operations, waits on conditions, spin-locks - these all have the same potential to stall a thread. And in this highly networked world, two threads don't even have to be on the same machine to dead-lock each other.

You can get pretty far with carefully constructed automatic test cases - and you should, but analysis is by far the most important step towards correctness in a multi-threaded program. However, you need to design and write your code with that in mind, otherwise the complexity of the code can quickly render such an analysis impossible to perform in practice.

I have noticed a pattern in my unit-testing recently. Ah, the word "pattern" is possibly a tad loaded for this purpose, so you might also call it a trend, a style or a convention. It's a recurring thing anyways, and it has to do with how I build my mock objects in my unit tests. I write primarily in Java, by the way, so this is where I noticed this pattern but it may just as well apply, with or without modification, to other languages.

Onwards.

The gist of it that if I have some class called a "Peg" that is used in all sorts of situations throughout my code and I need to build mocks of it for testing, then I will create a companion class called "PegThat" in the test sources tree.

This "PegThat" class will consist of nothing but static methods that all return various forms and instances of Pegs. These static methods will have names telling of certain properties of the Pegs they return. For instance, it might have an "isSquare" method that returns a square Peg.

The beauty of this naming convention is the readability that ensures in my test code. Behold:

@Test public void
squarePegsMustFitIntoSquareHoles() {
  Peg peg = PegThat.isSquare();
  Hole hole = HoleThat.hasEdges(4);
  
  assertThat(peg, fitsInto(hole));
}

That is arguable a very contrived example, but when you are building more complex mocks or instances, and these static methods need parameters or various types, then the benefit increases. Your tests can become a lot less messy and a good deal more readable.

The following is something I intend to sleep on:

Introduce abstractions out of need. Abstractions that are introduced through design rather than need are inherently over-designed.

Programming is about satisfying need.

Agility is about reacting to need rather than anticipating it.

TDD is a process of formulating need at a low level.

In his book Facts and Fallacies of Software Engineering, Robert L. Glass cites from a research paper from 1968, that the best programmers can be up to 28 times better than the worst.

That paper is Sackman, H., W. I. Erikson, and E. E. Grant. "Exploratory Experimental Studies Comparing Online and Offline Programming Performance." A brief, but publically available, treatment of it can be found here and most like elsewhere too if you ask Google.

Now, I know it's bad form to critisize a paper I have not read, but you need an ACM account to get at the real thing, so I will instead base my opinion the linked blog post, and otherwise be brief about it.

I have two points of critique, and the first and most obvious one is regarding the age of the paper. It was published 41 years ago and its primary purpose was to figure out whether time-sharing or batch processing systems were the most productive - which it did. And then, "almost in passing," apparently, they present numbers showing that the best programmers in their study were up to 28 times better than the worst.

I have to wonder whether the numbers have changed. Have the worst programmers gotten better, or worse? And are we even able to distinguish such a number from a difference among the best programmers? I have absolutely no idea. Most people I know, myself included, wasn't even born 41 years ago.

The second peeve I have with the unkown content of that paper, or perhaps rather the 28:1 quote in the context of that paper, is the dataset. While I don't know how many programmers participated, I do know that they were tasked with solving two programming problems.

While such a method is able to show peaks such as "up to 28 times better," it fails to show whether this grade-28 master is able to keep ahead in the long run. The two programming problems they were presented, were pretty small in size. Especially compared to many of the projects that are going on in the real world right now.

Is the master able to keep his times-28 advantage throughout a project that takes a year to complete - or at least is "normally" estimated to a years worth or work? Assuming they will finish at some point, how long will that same project take the worst programmer to complete?

It is possible that operating under the assumption that the worst programmer will, eventually, finish the project, is a pretty far stretch. But so do the "28 times better" seem when considered in the long run. Given the numbers are based on a mere two programming problems, the 28 times may have been a fluke. Maybe the guy was just lucky - you can argue that it takes skill to be that lucky, but still.

I once spent one and a half months hunting nasty memory leak (and I don't ever want to do that again), but once I found it I just had to change two lines and it was fixed. I could have been lucky and found those lines after a week, or never introduced the leak in the first place, but unfortunately it didn't happen like that.

In conclusion, I don't think that paper has the material to justify "up to 28 times better" as fact. The difference is there, for sure. And many people seem to feel in their gut that it may be around 10 times better. That's also a nice round number, but I won't claim it's a fact though.

I read Java Concurrency in Practice (Brian Goetz) and it taught me something important. It showed me a whole new "dimension" (I can't explain it any further than that - words fail me) of reasoning about code. This was not the intent of the book, but rather a side-effect of the examples in it and its approach and focus on correctness.

I now write all my code with this same focus and attention to correctness, and I always reason about my code through this new dimension that I have discovered, and I am always concious about the thread-safety of my code.

Then I read Release It! (Michael T. Nygaard) and it showed me that my newfound attention to correctness could, and should, be extrapolated to a much higher inter-system scale. The appraoch is now less about fundamental reasoning, and instead more about making concious design decisions.

The situations where I need to apply this knowledge are fewer and farther between, but their effects are equally profound and, I dare say, even more noticable and visible to the people who surround the systems I work on, and my fellow programmers.

Then I read Java Management Extensions (J. Steven Perry) and it sucked. Moving on.

Then I read Facts and Fallacies of Software Engineering (Robert L. Glass) and while my code is unaffected, I still learned something important. Actually, I learned many things from this book. But the book itself is like a window through which, if you care to look, sense something called experience.

Here's a guy who at the time of writing (2002) had been a software practitioner for 45 years. He has seen things, tried things, evaluated things, researched things and written a ton of code. He does not use the word "fact" because he is pompous. When he says "fact," he implies anectdotal evidence and research papers (that he has actually read and understood).

Where was I going with this? Oh, yes. The facts and fallacies are worth remembering. And that's about it. This stuff is good to know, because, it just is. Okay?

Now I'm reading Clean Code (Robert C. Martin) and I have only just made it past the introduction, but already sense something. I get this weird feeling. You'd think that any reasonably experienced programmer knows what clean code is, knows it when he sees it, knows how to write it and knows how to change bad code into clean code.

Do you know what clean code is? Can you tell me? If you think "yes" to these questions, then try to imagine yourself in the classical elevator-pitch situation and tell me, right to my face, what clean code is. Do you still know what clean code is? Can you even point to an example that you are certain is clean code?

I have just made it through the introduction and just realized that I can't do any of those things. And what's more: clean code is more important than I thought before I started on this book.

I can see my horizon expanding but I have no idea how far it will go - that's the feeling I have about that book, right now.

I wonder what I'll read next