Alexandru Bolboaca-Diaconu

Posts tagged with "development"

A theory about common sense

Tuesday, 27. January 2009, 20:18:10

common sense, engineering, brain, development

In my many long days (and a few nights) of professional software development, I've had quite a few WTF moments. I'm sure you all know what I'm talking about - that moment when you look to somebody's else piece of code and you realize that he/she/them used the worst possible solution to the problem in hand, one of such devious ingenuity that you could never have created it, and therefore builds into you feelings of both fear and nervous laughter at the same time.

Every time I shared such a solution with one of my fellow developers who have the appropriate stature to understand the point, we were both thinking: this is nonsense! This violates the most basic mechanics of software development! This violates our common sense!

But common sense is a very elusive concept. How do you express it, how do you explain it to those people violating it in such an unimaginable manner? How can they learn it, use it, teach others? How did you get into its possession?

Hard questions, with no answer. Until now, when I have finally developed a theory of common sense. It may be proven right or wrong, but at least it's a theory that is, I believe, worth exploring.

My theory can be summarized as following:

We start learning programming because we are attracted by it
We continue to learn and read more and more because it's more and more interesting
The knowledge pieces we gather connect to each other
The connections solidify in patterns
The patterns become the second nature and the common sense is born

The first empirical evidence of this theory was available to me a long time ago. I realized that the programmers that had common sense also touched a wide area of topics related to software and computers. I could discuss with them about all kinds of things, from the architecture of Linux kernel to functional languages, including topics from telecommunications, networking, assembly, electrical circuits or even unrelated topics like quantum physics, astronomy or genetics. One does not read such topics if forced, so they must have liked it.
On the other hand, the programmers that did those mistakes could not discuss any interesting topic and had almost zero culture in programming. Most of them had learned one programming language and one platform (.NET in my case) and were programming for money.

However, knowledge is not enough. I have also met people who knew quite some things about programming but had a crippled common sense. They weren't as bad as the WTF cases, but they had some repeated glitches. The brain plays a major factor here; the way it filters and organizes information can create very good systems or good enough systems of patterns.

My theory is also supported by some scientific research. A Scientific American article talks about The Expert Mind.

A chess master expert. Photo licensed under a CC license by mysza831

How did he play so well, so quickly? And how far ahead could he calculate under such constraints? "I see only one move ahead," Capablanca is said to have answered, "but it is always the correct one."

The article shows that the mind of an expert is formed by learning, practicing and working for at least 10 years. This helps forming a specialized type of memory that the expert uses for his work and that makes a very big difference between him or her and a beginner.

On the other hand, software development studies have shown that a good programmer is at least an order of magnitude better than a mediocre programmer. This isn't so unexpected when you know about the expert mind, is it?

In programming specifically, many studies have shown order of magnitude differences in the quality of the programs written, the sizes of the programs written, and the productivity of the programmers. The original study that showed huge variations in individual programming productivity was conducted in the late 1960s by Sackman, Erikson, and Grant (1968). They studied professional programmers with an average of 7 years' experience and found that the ratio of initial coding time between the best and worst programmers was about 20:1; the ratio of debugging times over 25:1; of program sizes 5:1; and of program execution speed about 10:1. They found no relationship between a programmer's amount of experience and code quality or productivity. (Code Complete, page 548)

And the final piece of evidence that I can provide at this point is a study on the ability of people to learn how to program. The study was initially called in a very suggestive manner The camel has two humps because it shows a large difference between the people who can learn programming and those who can't.

Expert pattern built by an expert. CC licensed work by Vilhelm Sjostrom.

Abstract: All teachers of programming find that their results display a 'double hump'. It is as if there are two populations: those who can, and those who cannot, each with its own independent bell curve. Almost all research into programming teaching and learning have concentrated on teaching: change the language, change the application area, use an IDE and work on motivation. None of it works, and the double hump persists. We have a test which picks out the population that can program, before the course begins. We can pick apart the double hump. You probably don't believe this, but you will after you hear the talk. We don't know exactly how/why it works, but we have some good theories.

In conclusion, my take on common sense in software development is:

Some people like programming and some don't - the camel has two humps

The people that like programming study a lot, forming their expert mind

The expert mind makes an order of magnitude difference

What we call common sense are the patterns found in the expert mind

Until next time, may the common sense become more common!

2 comments

People evolve faster than their genes - part V

Thursday, 11. December 2008, 08:22:13

development, hardwiring, quality, ideas ...

Here we are to the last (for now) installment of this series. And what a road it has been! We've talked mainly about brain, about its internal hardwiring and about the limitations and mistakes it is prone to. We discussed about pattern matching, about how great it is and that it tends to be overused, about the (in)ability to predict future, about it being fooled by randomness and about how all those things affect software development.

I will try in this post to draw a conclusion from this discussion. During the writing of this series, I realized that the best practices that we now know and sometimes use in software development are related to the limitations of the brain and are actually means to overcome them. It's a natural process: we noticed some of the things that don't go well, tried to solve them and evolved the solution up to a certain level, but the resulted best practices solve fundamental issues with the brain.

Therefore, we are now in the position to apply the same process, but this time knowing what it's about. There are many studies on the brain, on its limitations and on its super-powers (include only scientifically valid ones please), so we could start from them and derive or improve the current best practices. We would then have a method for deriving software development best practices, which is equivalent to saying that we can end the empirical phase of organizing development teams.

Provided that we use this method, the way of teaching software development must change. We already know that, but we don't really know how to change. I believe that the study of brain's limitations is mandatory for any programming course, because successful software development depends first of all on the team members. In fact I will go as far as to say that the study of brain's limitations should be the cornerstone of any programming curriculum.

It can also be really fun. I provided in my previous articles links towards speeches and books about the brain's limitations and I have to say that they are the most fun scientific materials I've ever come across. It's so easy to find about the brain's limitations, because we all have one, and sometimes we're all tricked in the same way - due to our similar hardwiring of the brain. For instance optical illusions are the best proof that pattern matching is overused. Future prediction games can easily be designed, as well as randomness toys. This is good, because it is really easy to understand and internalize the idea that those issues exist, that you have to find ways to overcome them and that we already know some solutions that you can apply today. The main difficulty lies in finding better ways to overcome limitations, but given a similar mindset and a collective mind, I'm sure that we can get significant improvements faster than expected.

Of course, the beneficial effects of such a lecture are not limited to software development. The reasons I'm talking so much about it are (a) because it's my domain and (b) because software development works with ideas, and the brain is our only tool. Thus, any improvement in the way we work with it will show a dramatic increase in quality, performance, productivity and so on. Despite my personal interest, I am sure that the applicability of a lecture on brain's limitations is beneficial for many other domains, especially creative ones - which become more and more present in our lives due to the Internet and the scientific advances.

Therefore, my conclusion is that we should study brain's limitations, document their relation with software development best practices, derive or improve best practices from our knowledge about the brain and start teaching programmers, present or future, about those issues. This is exactly what I will do from now on, because this topic has captured my imagination and my thirst of knowledge.

Until we talk again, unlimit your brain - it's worth it.

PS: All the articles in this series will undergo a process of review and improvement. New horizons have opened for me during their writing, and new information has appeared - as it does every day. If you happen to have any connection with brain studies and have the time for a critical view on the series, please send me your remarks. If you know of any connected studies, books, speeches etc. please let me know. For me, this is only the beginning - although those are questions I've pondered since long ago. Any help will be appreciated and remembered. Thank you.

1 comment

The Importance of Data Structures in Software Development

Tuesday, 7. October 2008, 08:51:41

development, Java, .NET, Linus Torvalds ...

If there's one thing that I noticed that makes a lot of difference between programmers is their knowledge and usage of data structures. That's why one of my favorite quote from Linus Torvalds is:

I will, in fact, claim that the difference between a bad programmer and a good one is whether he considers his code or his data structures more important. Bad programmers worry about the code. Good programmers worry about data structures and their relationships.

Torvalds, Linus (2006-06-27). Message to Git mailing list. Retrieved on 2006-08-28.

The explanation for this fact is that it's related to the biggest problem in software: complexity management. There is an amount of essential complexity that one cannot minimize, but it can be placed so that it doesn't get in the way and it's also easy to understand. The best such place is a data structure, from a very simple reason: the way the brain functions.

Imagine the following scenarios, with the same result:

You have to perform a number of very specific and detailed steps, written in a language that you don't completely understand
You have a number of items, an initial configuration and need to move them to a known final configuration

Which one do you think would be simpler for you?

I would bet on the scenario B. It is much easier for our brain to picture how to move items than to follow instructions. This is because of evolution: for survival it's much more important to use the environment in your help than to follow instructions.

Well, scenario A is complex code, while scenario B is a complex structure. See the point?

Unfortunately, from my experience (mostly with Romanian developers), many developers don't know what data structures are available and how to use them properly. I've noticed that it happens more to .NET developers than to Java developers, probably because JDK has much more built-in collections and because Java community is much more motivated to follow best practices.
If you're a .NET developer and you don't know what a Dictionary, a HashTable or a red-black tree does, there are ways to learn. A very good one is the documentation of the C5 generic collection library (which is now included in Mono 2.0, but can be used with .Net Framework as well), but if you want more, add a comment; I am planning to release a tutorial on data structures and their practical uses, if there are interested people.

Until next time, may your HashTables be protected from collisions!

7 comments

Software development learning and development tools

Tuesday, 30. September 2008, 07:53:55

development, development tools, IDE, quality ...

I started recently to work with Sql Server 2008. (You see, this is where you realize I'm just a simple developer and have no connection with Microsoft. All posts that I've read about Sql Server 2008 contain words like "vision", "experience", "trusted", "productive", "bringing peace and prosperity" - ok, ok, I made that last one up.)

The first thing I've noticed in the Management Studio, besides the much improved splash screen, was that I wasn't able to save modifications to the tables structure anymore. Instead, I get a very nice and welcoming message:

The reason I say it's welcoming is that it says "Welcome to SQL DDL!" Yes, you read it right. Since the designer doesn't work anymore, I started to use SQL DDL to modify the tables structure, something I've never actually done before, because databases were not really my thing. But, based on two principles that I'm acting by those days:

The best way to learn is to do
The best way to do is to do

I learned a lot. I learned that you have to drop a column in order to make it identity (that was quite logical, by the way). I learned, by using, the syntax for alter table. Yes, it's been annoying at first, but now I'm starting to think: why would I need the Management Studio?

In fact, I'm about to make a scandalous proposal.

What if we were to make our development tools randomly shut down a feature for a week?

E.g.: you go into your IDE and you see that the Build button is grayed, so you need to write a build script, so you automatize your build and everybody's much happier! Or your forms designer doesn't work anymore, so you have to write the forms by hand, and you learn more about what's behind the scenes.

The point is, folks, that you learn more by doing that by anything else. And development tools are colored, have nice splash screens and allow you to make everything you need, except learning. Try to live without parts of them for a while, and see how much you've improved your knowledge. You'll be amazed.

P.S.: I can't help but notice that Microsoft did actually do something good for me by not allowing me to use the table designer. Of course, they didn't mean it.

Follow-up: As I was expecting, the solution to the problem above is really simple (the point is that I didn't search for it on purpose). You can find it here.

1 comment

The reverse of Carlsberg slogan applied to code

Monday, 30. June 2008, 19:08:53

development, C#, Entity Framework, IT ...

I sometimes get very annoyed with code I stumble upon. It happens more often these days, when I'm doing for one of my clients a thorough code review, cleanup, low level refactoring and so on.

Today was a special day with this respect. I found something that made me mad, in the most Monthy Pythonesque way.

I found a piece of code similar to this:

for(int i = 0; i < 100; ++i)
{
    SomeEntity entity = 
        new SomeEntity{SomeOtherEntity = someValue};
}

For the people less familiar with the marvelous Microsoft universe, let me explain that we are using Entity Framework, which is a ORM for .NET, developed by Microsoft. So SomeEntity actually corresponds to a table in the database. The entity instances are managed in a special space called a context. As for the constructor, it has this strange form because it initializes properties of the class with some values.

Now, when any decent developer looks at this code, it becomes immediately apparent that the variable entity is a local variable which is not used again. So, it's useless. Right?

Right, but with one catch. It so happens that whenever you set a property based on a relationship between two entities, as in the example, the entity is also saved in the context. Yes, you understood it correctly. It's like doing something like:

Dialog dlg = new Dialog();
dlg.BackgroundColor = Color.White;

and the dialog gets displayed.

The worst thing is that we didn't actually do anything to provoke something like this. This is how it works out of the box.

Disclaimer: it's true that we are using the beta 3 version of the Entity Framework. However, I am not very convinced that the final version will improve on this point...

No comments

Adopting Best Practices

Thursday, 5. June 2008, 07:47:45

best practices, IT, communication, development ...

I have been trying in those last days to write a document describing some best practices that should be followed by one of my projects - and, why not, by any other project. Trouble is, I find this a very difficult task, not because I wouldn't have enough material or inspiration, but because it's difficult to make it convincing.

It all starts from a simple fact: I know bad code when I see it. I know bad design when I see it. And, whenever I make those judgments I am aware that (A) I think of human nature and its limitations, (B) Many other people from the industry would agree with me and (C) My colleagues may not agree with me.

Regarding the first point, it is important to understand that people write software for people. And people are limited, they make mistakes, they have difficulties finding the simplest solution, they tend to overdo or to underdo, they don't know how to express something that will be easy to understand by others. The source code is nothing but a communication medium, only it's a three-fold communication medium: the developer who wrote it talks to the developer who reads it, and to the user who needs it, all through the code; the difficulty is that the user sees another representation of the code, which communicates differently.

If this communication is broken, problems start to appear. If you cannot read your code as a story, you are bound to have troubles at some point. If your application doesn't tell its story to the user, he/she won't be content.

Regarding the second point, there are countless books talking about best practices. "Code Complete", "The Mythical Man Month", "The Art of Unix Programming", "The Cathedral and The Bazaar", "Refactoring" and even "The Tao of Programming" are well-known books that are all cited, used as reference, implemented (Source Monitor's Complexity metrics is implemented based on "Code Complete") and required reading. But I still need to talk about them.

And this takes me to the last point. Many developers I know chose to be programmers because they felt they can do anything. To some extent, it is true: a programming job gives us power to mold ideas into something we like.
On the other hand, this view is terribly wrong. You cannot do anything, because you are limited as a human being.

This is a truth that's difficult to grasp, but it's the path towards mastering software development.

Personal advertisement: I am PassionIT and I help teams develop high quality software. I train, I help improving and I work hands-on, depending on the needs. Contact me if you need help. Alexandru Bolboaca-Diaconu

No comments

BadProgrammingException or On The Issue of Developer Mistakes Exceptions

Wednesday, 21. May 2008, 16:48:32

software, development, programming languages, best practices

Sometimes, in one of my Pai Mei moments, I think about introducing an exception called BadProgrammingException. Ideally, such an exception would also call svn blame and record the name of the developer who did the last modification in the code. (This might actually work from the technical point of view, if the project doesn't use branches and merges; or if you use a distributed vcs like bazaar or git).

The BadProgramming exception would be used in some of the following cases:
- on stack overflow
- on null pointer or reference
- on invalid casts
- on any exception thrown from the code that is not documented
- on value out of bounds
- on assert failures

and I'm sure you can think of some examples, too. Ideally, the introduction of such an exception would create more responsibility for the developers, and make them look twice at the code they're writing.

Of course, this is not possible and it won't work because of the human factor. But this idea raises an interesting question:

Should we try to eliminate "developer exceptions" like the above from our frameworks and products? Couldn't we avoid them in some way?

The answer is: Sure we could, and there's one simple way: design by contract. But, for some reason I can't understand, no mainstream programming language has design by contract built-in. But the possiblity exists; check for example the Nice programming language.

Instead, we are condemned to look at the code and search for all kinds of possible developer mistakes, like this one (C# + LINQ):

User u = db.User.Where(p=>p.ID=150).FirstOrDefault();
System.Out.WriteLine(u.Name);

or this one (C#):

Entity entity = someObject as Entity;
System.Out.WriteLine(entity.ID);

or, even worse, this one (C#):

Entity entity = (Entity)someObject;

In languages like C# or Java, the nullable pattern might work much better. Nullable was introduced for value types that should be allowed to be null. For example:

DateTime? dt = someObject as DateTime?;
if(dt.HasValue)
{
    System.Out.WriteLine(dt.Value);
}

(There are many other versions of syntax, including shorter ones.)

The idea is that the compiler will tell you if you try to convert someObject to DateTime instead of DateTime? (shortcut for NullableType<DateTime>). So why couldn't we use the same pattern for all types? Disallow them to be null, and only allow them when the programmer says so.

But this is just the beginning of a would-be-very-lengthy-discussion. Any ideas?

No comments

The Tester

Monday, 7. April 2008, 08:40:35

engineering, work, software, testing ...

The Tester is one of the most important member of a software development team. Unfortunately, in many projects I've worked, his/her role was severely underestimated, resulting in low product quality and high costs.

I believe this happens because testing is a more difficult task than the common sense tells us. Testing software is hard because it hits the same essential difficulty as all other activities in software development: software is constructed from ideas. It's even worse for a tester, because how can one check that the representation of an idea corresponds to the idea?

Because of this fact, a tester has to have a few key skills:

Smart, in order to understand the problem
Attention to details, in order to be able to find even the issues burried deep in the functionality and in order to be able to fill the missing details from the functional requirements (there always are)
Rigour, because he has to follow a process that allows him to check all functionalities, to recheck them periodically, to fill in the missing pieces and to improve the process
Empathy, because he needs to understand the users
Effective communicator, because he has to communicate both in writing (through test documents, bug reports etc.) and in speaking (with the developers, the business analyst, the project manager etc.)

If we add to those skills things like automatic testing, managing virtual machines and virtual networks, testing for security, stress testing and, the cream on the top, writing unit tests, we understand that a tester is a really complex character.

And a really important one, too. He has to work side by side with the development team right from the start of the project, because of simple economical reasons: studies have shown that bugs caught earlier are much less expensive than bugs caught late. Even a week difference between the moment when the bug is found may translate in a significant increase in costs: the developer locates the bug in a longer time, it's more difficult to fix it, it has to be re-tested and so on.

Therefore, if you want quality in your project, use a good tester and use him/her early. The price you pay for testing will be an investment, not an expense.

No comments

Business, more than usual

Thursday, 3. April 2008, 08:17:38

ranting, development, training, open source ...

I was very busy during the last weeks, and unfortunately my blog has suffered by having less posts than usual. The reasons are partly related to software development.

First of all, I've worked on the offer for trainings. It's almost ready, and will see the light of web in a few days.
The topics covered are various and are presented in a modern, interactive, hands-on format. And I'm not just saying that; we spent countless hours (mostly during the night) thinking what kind of games, workshops or exercises could help presenting the topics in the best way possible.

The topics go from best development practices (e.g. Software design) to programming languages (e.g. Javascript), from software project management (e.g. SCRUM) to specifics related to development (e.g. Application security) and we think that we add to the Romanian trainings pool the pieces that are currently missing. Our goal is to simply take the best we've learned and pass it on, helping people do their software development better and to feel better about it.

Second of all, I am still working on my DSL. It is a fairly complex task, but it turns out that I made the right choice when I decided to work on it in an evolutionary way: Write, Re-write, Refactor. It's still room for improvement, but I'm happy about the code quality and I'm happy that it starts to take shape.

Third of all, I'm still learning more and more about the Linux ways. I'm very happy with the Ubuntus installed on my computers, but I found out yesterday a few more things: prelink, an invaluable tool for making it run faster, and free maps for GPS applications - including Romania. Really cool!

Last, I have to say a few words about the adoption of OOXML as a standard. I can only use my common sense in this problems, and I have mixed feelings. On one hand, OOXML is nowhere near the maturity I would have expected from a standard: it's unnecessarily complex, it's not unitary, it's very difficult if not impossible for another vendor to implement it. On the other hand, maybe it's better that the format is now controlled by a standardization committee. But that's a very long discussion, and I would rather wait for the next developments before making any conclusions.

Well, that's it. See you soon!

No comments

The other half of agile practices

Tuesday, 1. April 2008, 07:37:39

management, continuous integration, development, best practices ...

I believe that agile practices are some of the less understood topics in software development today.

Just do the following, very simple exercise: answer the question "what do you think when you hear about agile?"

The list of answers I'd think of is:

less documentation
people are more important than processes
refactoring
the developers get involved in the project

It turned out that my list is wrong. Its substance is not wrong, but it's utterly incomplete.

I only got half of the picture, and I believe so do many other IT professionals.

The trouble is, if you take each of the above principles and apply them, you will get really bad results. I can't think of a better way to illustrate this than an older Dilbert cartoon:

I've recently had the chance to find out about what agile really means from one of the Scrum creators, Jeff Sutherland, through my wife (who is now a Certified Scrum Master). Even if I wasn't directly in connection with him (though I would really have wanted to), I found out a few very important things.

First of all, there's no such thing as agile. There's Scrum, there's XP, there's open source and so on. But if you say that you're working "agile", something is wrong.

Second of all, Scrum is about the common sense. That's something I already knew and fought for in the context of my previous projects. Some of the most common mistakes due to lack of common sense are: not testing the application, not allowing developers to do their magic, trying to plan and to adapt the plan to continuous change, not using continuous integration and the list may go on.

Third of all, Scrum is applied succesfully by many companies at different levels. The most extreme Scrum approach is to put all tasks from all projects in a common backlog and transform all your company in a self-organizing team that works on this backlog. Sounds scary, but it works.

And last, the role of the architect in Scrum is really really important. The architect is the one who looks a little bit forward, but not too far, who creates a general but not too general model. He/she has to walk a really thin line, and the team is depending on its success.

There are many other interesting aspects about Scrum that I can't treat here. I just wanted to show you that I was wrong about agile, like, I believe, many other IT professionals are.

I am open to discussions about this topic that interests me profoundly, especially because in Romania agile development is even less understood and properly used than in other parts of the world. Let's talk about it, shall we?

1 comment

1 2 3 4 Next »

Showing posts 1 - 10 of 32.