The Humane Programmer

I have been involved in the last couple of years in a project that uses semantic technology, and I will try in this post to give a feeling about what it is and how it can be used.

I'm sure that we've all heard about semantics, in various contexts: language, parsers and compilers and lately related to web. However, until I've worked and thought of this concept, I didn't have a very good understanding of what semantics is in general.

According to wikipedia, semantics is "the study of meaning in communication", and actually you can substitute semantics with meaning in almost all contexts. Semantic analysis in grammar is looking at a sentence from the point of view of meaning. Semantic analysis in parsers and compilers is looking at the meaning of the programming language constructs put together. The idea of semantic web is to add machine-readable and inferable meaning.

The world we're living in contains semantics, a semantics built by three main characteristics: things, relations between them and actions that you can apply to a thing. Take the following example:

My(relation) name(thing) is(relation) Alex(thing) and(relation) I(thing) write(action) a blog entry(thing) while(relation) sitting(relation) on(relation) a chair(thing).

And that's basically what semantics is.

The problem with semantics is that we have a great tool for processing it, one that machines lack: our brains. The brain evolved along with us in this environment, and it developed special powers that allow it to process and understand the semantics of this world. Machines have no idea about things that to us are very simple to understand, like the fact that "Alex" is the name of a person, or what "writing" means. Therefore, in order to make machines process such information, you need to explain everything to them, or to allow them to evolve. The first solution is very time consuming, while the second may give very strange results as you cannot control the evolution but only the environment and the rules for it.

Despite these problems, people have found ways to build machine-readable and inferable semantics, some of them very familiar to any of us.
The web is the first example that comes to mind. It has a few types of objects (page, image, sound, object), a single type of relation - links to - and one action - follow link. A wiki is the same thing, only that there are rules for the free text content of the page - you know that in wikipedia the page http://en.wikipedia.org/wiki/Semantics discusses "Semantics" and you have additional actions: edit, create, delete.

Video games build their own semantics. In a FPS, you can move, rotate, shoot, and each object may have additional attributes: weight, health, stamina etc. A RPG builds upon a more complex semantics that describes not only your current state but also your possibilities of evolution - nothing more than a kind of metadata. This is only natural, since a game tries to define a world with meaning for the player.

The easiest built semantics that you can find in games is in MUDs. Due to them being text-based, there's plenty of freedom in defining objects, relations and actions very quickly, and obtaining a world where anybody can play and try. This advantage of quick prototyping allows for a lot of user generated content, even if the community is rather small.

Social networks also build semantics. The main object is a person, with various other collateral objects: company, group, awards, schools etc. The main relation is "know": "Person X knows person Y". The actions are add and remove. The problem is that they are limited, because other elements are missing from the picture: trusts, prefers, likes etc.


If we think about technology from this perspective, we realize that the tendency is to add and to enrich the semantics of the applications. However, this is no easy treat. Trying to do too much can only bring failure. It's better to start small, and that's why technologies like the web, wikis, social networks are so successful, despite having very limited semantics.

However, it is clear that adding semantics - new types of objects, of relations and of actions - can significantly enrich the user experience, if done properly. That's why I think that in the long run, the future is semantic.

Nassim Nicholas Taleb is a stock exchange trader who also happens to have a mathematical background. He's a very good trader because he has a secret: he knows how
to take advantage of randomness and how to avoid the traps of applied statistics.

Europeans believed swans can only be white until they discovered Australia. The Black Swan is a sample of a rare event, and also the name of one of Nassim Nicholas Taleb's books. Photo released under a Creative Commons license.

We wouldn't be talking about a stock exchange trader in this series of articles about the issues raised by the internal hardwiring of the human brain if he wasn't special in some way. Mr. Taleb sure is; he wrote a book that was selected by Fortune as one of "The Smartest Books of All Time" and another one that challenged the common view on random events. His main points are:

• We do not perceive correctly the randomness that surrounds us.
  
• Many events in the human history are actually rare and random.
  
• We retro-fit to those events a logical explanation that exclude randomness (our brains create causal links where they don't really exist)
  

As a result, his very successful trading technique is to bet on rare events that create high gains. Instead of removing the maximum and minimum values from the analysis, as most statistical methods do, he is searching for those values, he is betting that they will appear, because they make the difference.

His story is very interesting because this is exactly the way Scrum was created. When Jeff Sutherland and Ken Schwaber thought about a new, simplified and productive way of work, they took inspiration from those software development teams who had excellent results (it's worthwhile to mention that those teams include the IBM "surgical team" designed by Frederick Brooks). Ironically, the same teams were eliminated from the industry-wide statistics because normal statistical distribution is the norm (that's also why scientists didn't see the signs of global warming).
As Scrum proves, following the rare maximums is the way to go, instead of looking at the average. So we can say that for many years, software developers and managers were fooled by randomness - as Mr. Taleb would say.

This is not the only activity in software development where randomness intervenes. Estimation is another one. Not many people know, but an estimate for a task is actually a combination of two factors: how long it will take and the probability that the prediction is correct. The probability evolves over time, creating what is known as the cone of uncertainty. The interesting thing is that a software developer can estimate the probability for his/her estimation when asked, but doesn't think about it if not required. (I don't know how accurate the probability estimations are, but the important point is that anyone will admit that the estimation is not 100% accurate.)

Let's not forget about the influence of the business evolution on software projects. It is well known that requirements change all the time during development, and the waterfall approach tried to turn a blind eye to it - with notorious results. The continuous change of requirements is a very real random factor in the software development process, and it should be treated as such. Maybe with the help of ideas like the ones presented here we will improve the way we manage it. Until then, smart planning of short iterations is the best way to go.

A different view on causality. Photo released under a Creative Commons license.

Another point tackled by Mr. Taleb in "Fooled by Randomness" is that people, especially traders, can be fooled by consequent successes into believing that they always win. This is a direct consequence of the failure of perceiving randomness. Some of you may have met with software developers or project managers who think they are unbeatable, because most of their projects were successful; I sure did. Sometimes they were overestimating their results (on a closer look, the projects were actually close to disaster), other times they were overestimating their contribution to a good result. We know that everyone has a self-evaluation bias, people obtaining worse results thinking they perform better while people obtaining better results thinking they perform worse. But isn't there a random factor (besides the one we know) adding to the success of their projects? Weren't they just lucky random fools? That's difficult to say or measure, but it seems reasonable to assume it's true.

In the end, the main points are that our brains tend to under-evaluate the effect of randomness in our lives and in our projects. A conscious effort is necessary to overcome this limitation, and estimation techniques do just that by narrowing the cone of uncertainty. We need to include the random factor in our planning, and let go the illusion of control.

Until next time, may you achieve the wisdom of randomness.

In the previous installment, we saw that the brain is a tool for faster evolution. We will now start to examine the mistakes that the brain makes in the modern world, due to its internal hardwiring. Due to my personal knowledge, we will concentrate on examples from software development.

If we were to wonder about the best capabilities of our brains, I believe we would come with a short list that would include somewhere at the top the pattern matching ability. The fact that we can recognize in a very short time objects in various light conditions, colored differently, slightly deformed etc. and to contextualize them has yet to be beaten by any computer program, although many have tried.
Even more surprising, we are able to recognize not only complete objects, but also parts of an object. We are able to recognize sounds even if they are distorted, and, even more mind blowing, we are able to match ideas with abstract patterns.

Still, how can the brain be so good at pattern matching? More specifically, does it cheat in some way?
The short answer is that we don't know yet. In order to find out, scientists are looking at, guess what, patterns of brain activity in various situations. But, due to the internal complexity and to the speed with which the processes happen, it's very difficult to completely decompile our brain.

Illustration of overuse of patterns by our brains. Unknown author and rights, as far as I know part of public domain.

What we do know is that the same amazing ability that helps us live in the physical world, communicate with other people, recognize the sounds around us, understand the structure of the universe and so on is also harming us sometimes, by recognizing patterns where they don't really exist or by trying to use patterns where they don't match.

Those two fallacies are certainly not fatal to our existence; they are trade-offs of the evolutionary engineering. We can function and be successful with them, but they don't let us reach our maximum potential. There's one thing we can do: learn about them and know how to recognize the pattern of fallacy of pattern recognition. In other words, use the ability to circumvent the errors of the same ability. Sounds kind of meta-human, doesn't it?

The two errors often play an important role in software development, due to its essential difficulty. Let me illustrate this with a few examples.

Imagine that a customer comes to you and asks you to make an online ticket selling application, without giving any more details. At that exact moment, your brain will start to recover the patterns that match this idea of online ticket selling application. You may start to imagine an internet application, with an administrative backend, with a shopping cart in the frontend, a seat-selection screen and so on. You will think about the technologies you know best and of similar modules from other applications that you've implemented and you'll think of an estimation for the application, let's say 6 months with a team of 3 developers and one tester, if we're using php and mysql.

What the customer wanted and what he received at the end. Available under Creative Commons license from brajeshwar. Larger version here.

The amazing thing is that you see all this in your head before even asking more details about the requirements, effectively materializing the idea of the online ticket selling application into a very specific and subjective view, determined by past experiences and patterns stored in your brain. Most of the times, this view will be completely different from what your customer think he wants and most definitely different from what your customer needs. (Let's not forget that the customer is human too, and he has already been fooled by his own patterns when trying to solve his business problem). Even more, the whole process is automatic; you have no control over it.

Let's say that you ask the customer the business model, and he tells you that the application will be used by one office who sells tickets, and there's no option for users coming from the Internet to buy tickets directly. All the scaffolding you built in your head is brought to pieces by this very simple detail. The application can turn into a simple csv file that is sent by email, modified by the office employees and then sent back to the headquarters every day – enough to solve the actual business problem. You've just been fooled by your pattern matching ability!

Of course, this example is extreme. The exact sample is easily avoidable in most modern situations. However, I'm sure any software developer can come with a story when the implementation was more than needed, or not exactly what was needed. If you analyze it more thoroughly, you will find at some point the fallacy of matching the wrong patterns.

A more technical example refers to the use of design patterns. MVC, for example, is a very powerful pattern that has been overused in time. It's very useful when the same piece of data needs to be seen in different views, and when changing it in one view needs to update the other views of the same data. However, I've seen it used even in very simple scenarios, unnecessary complexifying the design. This human tendency has well known and captured in the saying “When you have a hammer, all objects around you seem nails”.

So how do you avoid those fallacies in software development? Simply by following a few well known best practices:

1.Understand the requirements, in all its gory details – talk to the customer, use job shadowing if you can etc.
2.Keep it Simple, Stupid – easier to say than done, this principle requires a continuous effort to create the patterns for simplicity in the brain and to follow them at all times
3.Release early, Release Often – so that even if you're fooled by your pattern matching, you find out as soon as possible
4.Describe the scenario you're estimating before crunching numbers – even if you're not consciously trying to use an estimation method, this simple trick allows for a much increased accuracy
5.Use patterns in context, not as solutions to all problems

If you look at the principles of agile development, you will see that it integrates many of the solutions to the problems presented above. This is because agile development focuses on people and on people improvement. This is also one of the reasons for which it fails – if the people don't make the effort, it won't work.

We will continue next time with more fallacies of the human brain. Until then, match your patterns with care!

---

Bibliography

Software Estimation - Demistifying the Black Art by Steve McConnel - A look into estimation techniques and best practices

Stumbling on Happiness by Daniel Gilbert - The catchy title hides an informative and amusing trip in the fallacies of the human brain, including the pattern matching failures

A talk on the same topic by Dan Gilbert; less about pattern matching, it is a short version of the book

I would like to thank Felix for the help provided for improving this article.

I've been very busy those two weeks trying to put together a training on security issues for software development. It wasn't an easy task, though I've had a continuous interest in security and in what it means for software projects.

The first difficulty is to concentrate a massive amount of information in sessions of 1h 15' - 1h 30'. I could talk for hours about encryption algorithms, about their history, what they mean in litterature and the amazing story of Enigma - how it was decrypted and the influence this had on our own history -, but there's never enough time. Then there are a lot of exciting facts from information theory that never get into such sessions. It's a pitty; on the other hand, when I train people on security topics, I'm never short of interesting stories.

The second difficulty is to present the human factor in security. The people who read this blog know that the human factor in software development is the central idea of my articles. It's not different with security. The need for security is a central fact in the human psychology. We have to understand this need if we want to produce secure applications.

I structured my training such that it's possible to deliver it no matter of the technology used by the trainees. It has a few common tracks, on the core topics of security, and it then has specific tracks that are technology specific; currently, I'm only covering .NET, but I can very simply add Java, PHP, C++ and other technologies on a need basis.

And here comes the final announcement: at some point in the (near) future, I will release the training materials under a Creative Commons license. I hope this will make them better and better, allowing many people to learn about the security issues in software development. So, stay tuned ;-).

---

Personal advertisement: I am PassionIT and help teams that develop high quality software. I train, I help improving and I work hands-on, depending on the needs. Contact me if you need help.
Alexandru Bolboaca-Diaconu

---

 

Quite often I hear developers or managers acting very surprised about the fact that an application "doesn't work".

I am never surprised when an application doesn't work. Probably because of my engineering background, and probably because I can still remember the second law of thermodynamics:

The entropy (=randomness, disorder) of an isolated system which is not in equilibrium will tend to increase over time, approaching a maximum value at equilibrium.

This law states something that we all know intuitively: it takes effort to move things in the desired direction. It doesn't matter whether we're talking about a carpenter trying to build furniture, a cleaning person trying to keep a room clean or a software developer trying to develop an application.

This means that the normal state of things is for an application not to work according to specifications. That's part of the reason why programming is hard: you have to fight entropy.

The reason why people are surprised with an application that doesn't work is that they don't realize just how hard it is to fight entropy in a software project, avoiding in the same time the other big enemy: complexity. Many people don't understand that programming actually means working with ideas.

What comes as a natural consequence of the above law is that I am always nervous when an application we develop seems to work fine. That's just not natural.
When I say "seems to work fine", I mean that there is no reasonable proof of the fact that the application is working fine. Of course, we can't prove that an application just works (no matter what marketing tries to tell us). But there are many examples of projects that were driven back because they lacked this proof and because they discovered at some point that inside the seemingly good working box there were many lurking bugs.

The first rule that results from this reasoning is:

Never trust your developers when they say the application is working fine.

Only trust them when they prove it to you with: unit testing, functional testing, beta testing (with real users) etc. Developers' mindset is about making the program work, not making it fail. That's why not many of them are good at testing their own modules.

In the end, remember that it's normal for an application not to work. It's unnatural when the people working on it say that it works without investing too much effort into it. The entropy is hard to beat.

---

Personal advertisement: I am PassionIT and help teams that develop high quality software. I train, I help improving and I work hands-on, depending on the needs. Contact me if you need help.
Alexandru Bolboaca-Diaconu

---