There is no spoon

I have been involved in the last couple of years in a project that uses semantic technology, and I will try in this post to give a feeling about what it is and how it can be used.

I'm sure that we've all heard about semantics, in various contexts: language, parsers and compilers and lately related to web. However, until I've worked and thought of this concept, I didn't have a very good understanding of what semantics is in general.

According to wikipedia, semantics is "the study of meaning in communication", and actually you can substitute semantics with meaning in almost all contexts. Semantic analysis in grammar is looking at a sentence from the point of view of meaning. Semantic analysis in parsers and compilers is looking at the meaning of the programming language constructs put together. The idea of semantic web is to add machine-readable and inferable meaning.

The world we're living in contains semantics, a semantics built by three main characteristics: things, relations between them and actions that you can apply to a thing. Take the following example:

My(relation) name(thing) is(relation) Alex(thing) and(relation) I(thing) write(action) a blog entry(thing) while(relation) sitting(relation) on(relation) a chair(thing).

And that's basically what semantics is.

The problem with semantics is that we have a great tool for processing it, one that machines lack: our brains. The brain evolved along with us in this environment, and it developed special powers that allow it to process and understand the semantics of this world. Machines have no idea about things that to us are very simple to understand, like the fact that "Alex" is the name of a person, or what "writing" means. Therefore, in order to make machines process such information, you need to explain everything to them, or to allow them to evolve. The first solution is very time consuming, while the second may give very strange results as you cannot control the evolution but only the environment and the rules for it.

Despite these problems, people have found ways to build machine-readable and inferable semantics, some of them very familiar to any of us.
The web is the first example that comes to mind. It has a few types of objects (page, image, sound, object), a single type of relation - links to - and one action - follow link. A wiki is the same thing, only that there are rules for the free text content of the page - you know that in wikipedia the page http://en.wikipedia.org/wiki/Semantics discusses "Semantics" and you have additional actions: edit, create, delete.

Video games build their own semantics. In a FPS, you can move, rotate, shoot, and each object may have additional attributes: weight, health, stamina etc. A RPG builds upon a more complex semantics that describes not only your current state but also your possibilities of evolution - nothing more than a kind of metadata. This is only natural, since a game tries to define a world with meaning for the player.

The easiest built semantics that you can find in games is in MUDs. Due to them being text-based, there's plenty of freedom in defining objects, relations and actions very quickly, and obtaining a world where anybody can play and try. This advantage of quick prototyping allows for a lot of user generated content, even if the community is rather small.

Social networks also build semantics. The main object is a person, with various other collateral objects: company, group, awards, schools etc. The main relation is "know": "Person X knows person Y". The actions are add and remove. The problem is that they are limited, because other elements are missing from the picture: trusts, prefers, likes etc.


If we think about technology from this perspective, we realize that the tendency is to add and to enrich the semantics of the applications. However, this is no easy treat. Trying to do too much can only bring failure. It's better to start small, and that's why technologies like the web, wikis, social networks are so successful, despite having very limited semantics.

However, it is clear that adding semantics - new types of objects, of relations and of actions - can significantly enrich the user experience, if done properly. That's why I think that in the long run, the future is semantic.

So everyone will know how to read?
So everyone will know how to write?
So everyone will know how to compute?
So many people will read what's written by certain authors?
So everyone will be able to talk to everybody else?
So everyone will be able to see/hear movies/music made by some people?
So everyone will be able to tape the movies and music from radio/TV?
So everyone will be able to make movies?
So everyone will be able to make music?
So everyone will be able to publish text, images or sound?
So everyone will be able to transfer text, music, videos to a lot of people?
So everyone will be able to publish anything and everyone will be able to see/hear/read/get a copy?
So everyone will be able to do all that without anybody knowing it?

In my many long days (and a few nights) of professional software development, I've had quite a few WTF moments. I'm sure you all know what I'm talking about - that moment when you look to somebody's else piece of code and you realize that he/she/them used the worst possible solution to the problem in hand, one of such devious ingenuity that you could never have created it, and therefore builds into you feelings of both fear and nervous laughter at the same time.

Every time I shared such a solution with one of my fellow developers who have the appropriate stature to understand the point, we were both thinking: this is nonsense! This violates the most basic mechanics of software development! This violates our common sense!

But common sense is a very elusive concept. How do you express it, how do you explain it to those people violating it in such an unimaginable manner? How can they learn it, use it, teach others? How did you get into its possession?

Hard questions, with no answer. Until now, when I have finally developed a theory of common sense. It may be proven right or wrong, but at least it's a theory that is, I believe, worth exploring.

My theory can be summarized as following:

1. We start learning programming because we are attracted by it
   
2. We continue to learn and read more and more because it's more and more interesting
   
3. The knowledge pieces we gather connect to each other
   
4. The connections solidify in patterns
   
5. The patterns become the second nature and the common sense is born
   

The first empirical evidence of this theory was available to me a long time ago. I realized that the programmers that had common sense also touched a wide area of topics related to software and computers. I could discuss with them about all kinds of things, from the architecture of Linux kernel to functional languages, including topics from telecommunications, networking, assembly, electrical circuits or even unrelated topics like quantum physics, astronomy or genetics. One does not read such topics if forced, so they must have liked it.
On the other hand, the programmers that did those mistakes could not discuss any interesting topic and had almost zero culture in programming. Most of them had learned one programming language and one platform (.NET in my case) and were programming for money.

However, knowledge is not enough. I have also met people who knew quite some things about programming but had a crippled common sense. They weren't as bad as the WTF cases, but they had some repeated glitches. The brain plays a major factor here; the way it filters and organizes information can create very good systems or good enough systems of patterns.

My theory is also supported by some scientific research. A Scientific American article talks about The Expert Mind.


A chess master expert. Photo licensed under a CC license by mysza831

How did he play so well, so quickly? And how far ahead could he calculate under such constraints? "I see only one move ahead," Capablanca is said to have answered, "but it is always the correct one."

The article shows that the mind of an expert is formed by learning, practicing and working for at least 10 years. This helps forming a specialized type of memory that the expert uses for his work and that makes a very big difference between him or her and a beginner.

On the other hand, software development studies have shown that a good programmer is at least an order of magnitude better than a mediocre programmer. This isn't so unexpected when you know about the expert mind, is it?

In programming specifically, many studies have shown order of magnitude differences in the quality of the programs written, the sizes of the programs written, and the productivity of the programmers. The original study that showed huge variations in individual programming productivity was conducted in the late 1960s by Sackman, Erikson, and Grant (1968). They studied professional programmers with an average of 7 years' experience and found that the ratio of initial coding time between the best and worst programmers was about 20:1; the ratio of debugging times over 25:1; of program sizes 5:1; and of program execution speed about 10:1. They found no relationship between a programmer's amount of experience and code quality or productivity. (Code Complete, page 548)

And the final piece of evidence that I can provide at this point is a study on the ability of people to learn how to program. The study was initially called in a very suggestive manner The camel has two humps because it shows a large difference between the people who can learn programming and those who can't.


Expert pattern built by an expert. CC licensed work by Vilhelm Sjostrom.

Abstract: All teachers of programming find that their results display a 'double hump'. It is as if there are two populations: those who can, and those who cannot, each with its own independent bell curve. Almost all research into programming teaching and learning have concentrated on teaching: change the language, change the application area, use an IDE and work on motivation. None of it works, and the double hump persists. We have a test which picks out the population that can program, before the course begins. We can pick apart the double hump. You probably don't believe this, but you will after you hear the talk. We don't know exactly how/why it works, but we have some good theories.

In conclusion, my take on common sense in software development is:

• Some people like programming and some don't - the camel has two humps
  
• The people that like programming study a lot, forming their expert mind
  
• The expert mind makes an order of magnitude difference
  
• What we call common sense are the patterns found in the expert mind
  

Until next time, may the common sense become more common!

Here we are to the last (for now) installment of this series. And what a road it has been! We've talked mainly about brain, about its internal hardwiring and about the limitations and mistakes it is prone to. We discussed about pattern matching, about how great it is and that it tends to be overused, about the (in)ability to predict future, about it being fooled by randomness and about how all those things affect software development.

I will try in this post to draw a conclusion from this discussion. During the writing of this series, I realized that the best practices that we now know and sometimes use in software development are related to the limitations of the brain and are actually means to overcome them. It's a natural process: we noticed some of the things that don't go well, tried to solve them and evolved the solution up to a certain level, but the resulted best practices solve fundamental issues with the brain.

Therefore, we are now in the position to apply the same process, but this time knowing what it's about. There are many studies on the brain, on its limitations and on its super-powers (include only scientifically valid ones please), so we could start from them and derive or improve the current best practices. We would then have a method for deriving software development best practices, which is equivalent to saying that we can end the empirical phase of organizing development teams.

Provided that we use this method, the way of teaching software development must change. We already know that, but we don't really know how to change. I believe that the study of brain's limitations is mandatory for any programming course, because successful software development depends first of all on the team members. In fact I will go as far as to say that the study of brain's limitations should be the cornerstone of any programming curriculum.

It can also be really fun. I provided in my previous articles links towards speeches and books about the brain's limitations and I have to say that they are the most fun scientific materials I've ever come across. It's so easy to find about the brain's limitations, because we all have one, and sometimes we're all tricked in the same way - due to our similar hardwiring of the brain. For instance optical illusions are the best proof that pattern matching is overused. Future prediction games can easily be designed, as well as randomness toys. This is good, because it is really easy to understand and internalize the idea that those issues exist, that you have to find ways to overcome them and that we already know some solutions that you can apply today. The main difficulty lies in finding better ways to overcome limitations, but given a similar mindset and a collective mind, I'm sure that we can get significant improvements faster than expected.

Of course, the beneficial effects of such a lecture are not limited to software development. The reasons I'm talking so much about it are (a) because it's my domain and (b) because software development works with ideas, and the brain is our only tool. Thus, any improvement in the way we work with it will show a dramatic increase in quality, performance, productivity and so on. Despite my personal interest, I am sure that the applicability of a lecture on brain's limitations is beneficial for many other domains, especially creative ones - which become more and more present in our lives due to the Internet and the scientific advances.

Therefore, my conclusion is that we should study brain's limitations, document their relation with software development best practices, derive or improve best practices from our knowledge about the brain and start teaching programmers, present or future, about those issues. This is exactly what I will do from now on, because this topic has captured my imagination and my thirst of knowledge.

Until we talk again, unlimit your brain - it's worth it.

PS: All the articles in this series will undergo a process of review and improvement. New horizons have opened for me during their writing, and new information has appeared - as it does every day. If you happen to have any connection with brain studies and have the time for a critical view on the series, please send me your remarks. If you know of any connected studies, books, speeches etc. please let me know. For me, this is only the beginning - although those are questions I've pondered since long ago. Any help will be appreciated and remembered. Thank you.

In the previous installments(first and second) of this article, we talked about the evolution of the brain and about the extraordinary pattern matching ability that we possess and how it can make us take erroneous decisions. We'll continue with another hardwired ability of the brain that is good most of the time and is sometimes bad for software development: the prediction of the future.


If we take a look at what the human race accomplished, we see lots of finalized projects: buildings, transportation, bridges, roads etc., some of them extraordinary, like this water bridge. No other species on Earth seems capable of doing this, so we must have something special.
It turns out that all those projects require a few steps: imagine the end product, plan it and execute it. The first two steps are of the foremost importance for any complex endeavor, and they have a thing in common: the ability to imagine something that doesn't exist yet.

Do we know how we do that? According to existing studies, this ability has a lot in common with the way we store memories: as a compressed lossy version. When we recall a specific memory, the brain fills in the details, using the present.
When trying to predict the future, the brain uses the same mechanism: starting from the present and from a specific piece of information (e.g. I want to build an online ticketing application), it fills it with details and creates an image.

We've learned from history that this ability works good enough, but we also know that sometimes it doesn't. The source of bad predictions is what we call "lack of imagination", and we have lots of examples. Let's see a few, just for fun:

"I think there is a world market for maybe five computers." - Thomas Watson, chairman of IBM, 1943.

"Where a calculator on the ENIAC is equipped with 18,000 vacuum tubes and weighs 30 tons, computers in the future may have only 1,000 vacuum tubes and weigh only 1.5 tons." - Popular Mechanics, 1949

"Radio has no future. Heavier-than-air flying machines are impossible. X-rays will prove to be a hoax." - William Thomson, Lord Kelvin, British scientist, 1899.

"It will be years - not in my time - before a woman will become Prime Minister." - Margaret Thatcher, 1974.

"The abdomen, the chest, and the brain will forever be shut from the intrusion of the wise and humane surgeon." - Sir John Eric Ericksen, British surgeon, appointed Surgeon-Extraordinary to Queen Victoria 1873.

When we analyze these bad predictions, we realize that the authors failed to imagine something that was entirely possible, due to being influenced too much by the present. And so we come to the next question: how do we avoid this influence?

Lucky for us, Arthur C. Clarke stated two laws that help us do just that:

When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.

The only way of discovering the limits of the possible is to venture a little way past them into the impossible.

There are two keys for a more accurate prediction:

• Details, many details
  
• Think what you need to do it, not if it's possible
  

The lack of imagination takes many aspects in software development.

During my career I have encountered many times the expression: "We can't do that", either in a technical context (e.g. we can't use a custom rule engine with built-in Workflow Foundation activities) or in an organizational context (e.g. we can't use unit testing now). Most of the times, I've proven them wrong by actually doing it. Actually, this is one of those things I constantly struggle with. However, this is the least damaging effect of lack of imagination.

The others, much more damaging in the long term, are:
- failing to imagine risks
- failing to imagine possible security breaches
- failing to estimate correctly

(It would be interesting to find a study showing the contribution had by the first and the third factor on project failure. I couldn't find any.)

Estimation technique called Planning Poker in action. Image released under a Creative Commons license by Wyscan

Let's look at some of the modern best practices in software development with respect to what we've established above:
- Release early, release often = you don't have enough details to accurately predict the user's needs, the risks etc.
- Design and implement incrementally = you don't have enough details to accurately predict the future, so think only of what you know for sure
- Estimate by describing the solution in detail = add more details to the prediction mechanism of your brain
- Threat modeling = detail all possible threats and treat them
and the list could go on and on.

To conclude, we have good news and bad news. The good news is that the human brain is able to imagine a possible future. The bad news is that it sometimes fails. The good news is that we can improve its ability by opening our minds and adding details to the picture.

Until next time, may your predictions be right!

P.S.: I'm a big time Douglas Adams fan. As a result, this trilogy will have five parts ;-)

---

Bibliography

Software Estimation - Demistifying the Black Art by Steve McConnel - A look into estimation techniques and best practices

Stumbling on Happiness by Daniel Gilbert - We destroy our happiness when we expect something very different than what we get. A very informative book about how the brain stores memories, recalls them and predicts the future - and about when it fails to do so correctly.

Dan Gilbert researches happiness, a TED talk about why and how we fail to predict the future

Wikipedia

Many articles about software development methodologies, best practices and so on

I would like to thank Felix for the help provided for improving this article.