Skip navigation.

Notes to self

Whatever I feel like writing

Posts tagged with "agile"

Form Follows Function

Saturday, 18. July 2009, 06:47:37

, ,

If form follows function then why do we do up-front design when we develop software?

My work place has, for some time now, repeatedly asked me for estimates and designs, and this has gotten me to think about how I feel about those artifacts. In this blog post I am going to try and put my feelings about up-front design into words.

Design is the detailed description of how something is suppose to be built, but software development is plagued by uncertainty to such a degree that any design conceived before the first line of code has been written, is bound to be wrong.

Design is a model and a description of a destination. A place we want to end up. A model and a description of what we want to end up with.

When one embarks on a new software development project, I would argue that such a destination is not a terribly valuable artifact to have. Kent Beck uses the metaphor of driving to describe software development; constantly adjusting the direction as we go, making sure that we don't fall off the road.

Kent Beck is spot on, in my opinion. We don't need a destination when we start a project. What we need is a starting point and a direction. And the people that need those things are the team members, not management. The team members all needs to be on the same page with regards to the starting point and the direction. This is even more so important with a distributed team, which is what I currently have at hand at work.

When the starting point and the direction is all agreed upon, then the destination, the design, will emerge as we go along, and become self evident when we at some point decide to stop. This is the "emergent design" idea from extreme programming.

But before I get ahead of myself here, I have already described how design is the destination in this metaphor, so what are the starting and the direction? Briefly, the starting point is the most pressing and important unimplemented user story. The direction is knowledge of the business problem and a vague conceptual idea of how a system might solve this. I suppose part of what the direction is, is described in the current set of unimplemented user stories, but the direction as a whole is more than just user stories. The direction is sense that all team members are "on the same page," so to speak.

So when I title this blog post as "Form Follows Function," what I really mean is that "Design Follows Implementation" and that this idea is emergent design. When we write code, we are also implicitly designing, and this design that emerges from our code is the real design, so to speak. It is the truth, as oppose to some up-front class diagram describing classes that don't exist and incorporating engineering trade-offs to problems that also don't exist (unless they are designed into the system).

The up-front design describes either something that we don't want, or something that we won't get, or both. And therefor, I think that working to the level of detail that is implied by it, is wasted effort. And I don't like wasted effort.

1 comment

Code Need

Thursday, 18. June 2009, 23:20:53

, , ,

The following is something I intend to sleep on:

Introduce abstractions out of need. Abstractions that are introduced through design rather than need are inherently over-designed.

Programming is about satisfying need.

Agility is about reacting to need rather than anticipating it.

TDD is a process of formulating need at a low level.

No comments

Disconnected Quality

Monday, 4. June 2007, 08:55:32

, ,

Can software quality be increased by disconnecting user demand from developer obligation?

I find that a very interesting question. Normal reasoning, at first glance at least, would have that if the developers give the users what they want, they will become happy (the users, that is).

That's what the books taught us in comp.sci. school, right?

I think we should read the question again.

Can software quality be increased by disconnecting user demand from developer obligation?

Note the wording: we're disconnecting user demand from developer obligation, but we don't rule out communication in general.

I'm thinking this: the agile methodologies praise communication as their center pillar, and claim that the more communication we have, the more quallity we get, but when we implement the agile methodology, we often end up with an extremely tight coupling between user demand and developer obligation.

Now, the thing that leads me to think that disconnecting user demand from developer obligation will increase software quality is an interesting thread on the XOM mailing list, where Michael Kay makes this claim based on his own personal field experience.

Had it been any random Ph.D. making this claim, I wouldn't have given it a second thought, but since this is based on real world experience, I find it very interesting and worth thinking about.

As I said, I found this on the XOM interest list, so let me introduce you to the discussion. First, Olivier Lefevre makes this claim:

If you want to be a player in the enterprise software world, supporting
multiple versions is a burden you have to be willing to bear anyhow

And Michael Kay makes this reply:

No, the dynamics of open source development changes that entirely. When
users aren't paying, they can't make demands.

When I started doing open source I found that incredibly liberating, and it
enabled me to move forward at a far faster pace and produce much better
software than I had ever done in the commercial space.

Then Randall R Schulz challenges Michaels claims:

Does this not amount to saying that if you ignore your users' requests,
you can do anything you want? It's true, of course, but it seems like a
disservice to those you presumably want to use your software.

You say non-payers cannot make demands, but at the same time, you cannot
pretend to be satisfying their needs unless you respect their
requirements.

I fail to see how disconnecting user demand from developer obligation
can improve the quality of software or accelerate the overall advances
in the world of software and information technology.

Software is already, and for the most part rightly, seen as inadequate
both in its utility and in its quality and reliability. Telling users,
even if they're other programmers, that if they want something they
should create it themselves is just arrogant.

Randalls reply reflects what I belive most developers think, and therefor alot of software is also built using this mindset. Randalls challenge causes Michael to elaborate on his view - this is when it gets interesting:

> I fail to see how disconnecting user demand from developer
> obligation can improve the quality of software or accelerate
> the overall advances in the world of software and information
> technology.

Yes, I was surprised by this effect, and I wouldn't have expected it until I
started doing it. But having worked for 20 years in a more conventional
mainframe software world, I'm convinced its true. Of course, user demand
isn't disconnected from quality: users have a choice, they can use your
product or someone else's. It tends to mean that you're motivated by the
feedback from the most discerning users rather than those with the deepest
pockets, and that the feedback comes from real users undistorted by chains
of communication between their purchasing department and your marketing
department.

Users sometimes ask you to do good things and they sometimes ask you to do
bad things. You still get the feedback, but you can avoid doing the bad
things, such as deviating from standards in order to gain a short-term
performance advantage, or introducing a feature that's actually a workaround
to a bug in someone else's product, or forking the source code to make it
run in two different environments. Many times in my career as a mainframe
software designer we did bad things to the software because the people with
the money wanted things that were not in the best long-term interests of the
community as a whole (and very often not even in the best interests of that
particular user). Forking the source code is a classic example.

I remember a project where a bank had implemented a very early XML-based
system using a Microsoft parser that predated the XML spec and allowed </>
as an end tag. Microsoft updated their parser when the spec was finalised to
remove this feature. The bank complained and said it would cost them a
million pounds or so to change their application or delay its go-live date.
Microsoft stood firm. They were right to do so. An awful lot of suppliers
would have bent to that pressure (certainly my employers, who were the prime
contractor for the system, would have done so). So yes, I believe that the
ability to resist user pressure can often improve the overall quality of the
software.

So the communication between developer and user stays, but the demands and obligations are disconnected, giving the developer the freedom he needs to sometimes make painful decisions for the cause of the greater good and quality overall.

I wonder if this can somehow be applied in commercial and agile development methodologies. It would require that the developer is able to deny implementing some of the customers requirements - getting this clearance in the project contract will be extremely hard, and contradict both common logic and common practice in this area - actually, I don't think it will be possible to get it in the contract at all.

At my first Scrum project, my mentor told me that if we want something done, we should make a business case for it. He used the example that customers don't know, and don't want/need to know about what refactoring is. To them, refactoring code is spending money on code and gain nothing in return. If we wanted to refactor code, we should explain that this was a nescesary step we needed to take in order to make the development of future functionality (and maintenance) a lot cheaper - this way, the customer would learn that they don't spend money on refactoring, they save it.
I believe the same thing, at least in some situations, can be done with troublesome requirements, however, you'd have to present an alternative. The customer has a requirement because they need to do something special with it - that need does not go away even if the business case for not implementing the requirement is, from a code point of view, obvious.

No comments

When Cowboy Coding Works

Tuesday, 26. December 2006, 19:41:58

Introduction
Early this year (2006) I worked on a school-project with three other students from my class. The goal was to create an asynchronous messaging protocol, server, two client APIs (one for Java and one for .NET) and a small Windows.Forms monitoring app that would use the client API to connect to a maintenance topic on the server, so we could keep an eye on how it was doing. And it all had to be done in four weeks.

We went straight to work. Didn't bother setting up or deciding on a development methodology, and during the course of the project, we didn't write a single unit test. All we did, was getting together every day and code away.

The project was a great success. We finished ahead of schedule so we had a lot of time for the presentation (preperation for exam), and we all got marvelous grades. We even had time to do stress/load testing on the server, and it was incredibly robust and performant - 300.000 messages a second on regular workstation hardware, anyone?

But how did we do it? How did we manage to produce high quality software, on time, within the limits of a tight deadline, without using any actual methodology? Without unit testing?

I've thought about this particular project on several occasions, and why it was such a success. I believe it dosn't boil down to a single punch line, but is the result of several factors interacting.

  • Qualified developers
  • Simple, light-weight, architecture
  • Tightly knit team
  • Domain experts
  • Source revision control/SCM

Let me explain why I think these factors were vitally importaint to the success of this project.

Qualified developers
Though some more than others, we were all excellent coders.

Having qualified people is absolutely critical. The difference between a good coder and a bad can be huge, and sub-standard coders can slow the excellent ones down, because they, on top of their normal work, are now also burdened with having to fix the code produced by the sub-standard coders, and they are burdened with having to go to greater length when explaining design and implementation descitions.

In our project, we did have a drag in this area, denying it would be a lie. But the drag we had was manageable. That's a key point to note: in a project with more than one developer, there will almost always be a drag because some are better than others, or in other words; some are worse than others. The solution is not to fire everyone who dosn't live up to the super-star status (only fire those who are sincerely bad, and make sure the replacement is better), but to keep the drag manageable. And one of the ways to keep the drag manageable, is to have a simple, light-weight, architecture, as explained bellow.

Simple, light-weight, architecture
We wrote everything from the group up. The only APIs we used were the standard .NET and standard Java APIs. Our architecture was explained and described using patterns. It was possible than we didn't all understand the same patterns, but we all had the same pattern books, so it was no biggie to read up.

Because we were writing everything from the ground up, without any additional frameworks, we got precisely the frunctionality we needed, and we had complete knowledge of how the system did what. This meant that we got a rather simplistic architecture, that was easy to comprehend.

The thing about simple architectures is, that they are easy to code, and if something goes wrong, it's easy to figure out what it was.

The combination of the simple architecture and our shared, pattern based, terminology, meant that we could easily communicate design idears, and, as explained bellow, a teams ability to communicate is very important for its success.

Tightly knit team
We knew each other before we started the project; we hard worked together on other projects before and we knew each others strengths, weaknesses and personalities.

This meant that we knew how to communicate with each other, and with our shared terminology, we could easily express our ideas. Our communication was further enhanced by the fact, that we met face-to-face every day, and that we were no more than four on the project.

Communication through Skype, Messenger, e-mail or even phone, is often a poor substitude for talking face-to-face. Face-to-face communication has lower overhead, is more expressive, more responsive and reduces the risc of misunderstandings. Secondly, I'm not saying that four is a magic number for project sizes, but there's a limit to how many people you can put in a team, without getting too much coordination overhead.

Domain experts
We had knowledge of the problem domain. As a school-project, the intent was the practical application of the theories, patterns and technologies we had been taught the previous months.

This meant that we could all take on the role of domain experts, illiminating the communication overhead (are you sensing a pattern?) and the risc of misunderstandings involved in using domain experts that are external to the development team.

In short, the resources we needed to invest in research in the problem domain was considerably reduced by the fact that we already had thorough knowledge on the subject.

Source revision control/SCM
Don't ever start a project without it. I've found this to be an absolutely essential part of any development project - even single-person hobby projects.

My personal favorite SCM is Subversion. We also used Subversion for this particular project, and everyone in the the group had used this tool before in previous projects. This meant that we were able to all work on the same source code, and constantly integrate our changes. In fact, we often integrated our changes over a dusin times a day.

We had an additional advantage in working in the same room everyday: we could yell to the others every time we committed a change, and coordinate our merges to prevent conflicting changes.

In conclusion, using an SCM saved us from a lot of troubles with differing versions and conflicting changes, and ensured that we were able to keep our development velocity and not worry too much about integration and merging details.

Conclusion
So, when does cowboy coding work? Who knows, but it did work in this particular project, and a number of things struck me as having been hugely contributing to the success of that project: The project had a manageable size - small, tight team and manageable architecture. The developers knew what they were doing and we had a good insight in the requirements. Our internal communication was extreemly efficient, and we made heavy use of SCM and frequent integration.

One thing is certain; I wouldn't develop cowboy-style if the project had been much larger, or our communication hadn't been as efficient as it was.

PS: Merry christmas and happy new year!

3 comments

Have Courage, Men! We're going Agile!

Wednesday, 29. November 2006, 14:13:33

What does Courage have to do with agile development? Lots!

  • Management need to relinquish some control over the developers - that takes courage.
  • Customers need to accept that fixed-price, fixed-time contracts and budgets is something they cannot get - that takes courage.
  • Developers need to take responsibility for their estimates and their promises - that takes courage.
  • And we need the courage to point at a piece of software or code and say "That goes straigt to the garbage can!". - and that's exactly what I'm going to discuss right this instant!

Throwing out code takes a lot of courage. Think about it: this code has already been written - it has already been paid for by the customer. Who can we have the audacity to throw out something so expensive as code? Imagine if the code was a car, and the salesman calls with this message:

I'm sorry about this car you bought: it didn't quite live up to my standards so I had it scraped, however, I have this new model you can buy and I think you'll love it!


I know I'de be pissed if that happened to me!

But code isn't quite like cars in this respect, because code rot. As we write more of it, the entropy and chaos in the code base rises and it becomes gradually harder to get things done. In short, as the code entropy goes up, the performance of the team goes down.

So to keep the performance, and morale for that matter, of the team up, and the entropy down, the code needs to be refactored every once in a while - and this often involves removing old and ugly code.
To the non-developers: refactoring is what developers do, when they do a lot of work moving stuff around. Work that the customer have to pay for, without ever seing any tangible results of the work.

The point of this article is, that this happens more often during agile development than during, say, a (R)UP project. And why is that?

I like to make the analogy of agile methods and non-agile method as cabled ethernet and Wireless LAN, respectively. I like this analogy because of the way these two networking technologies handles collisions much the same way that the methodologies handles risc.

Wireless LAN employes a collision avoidance scheme. That is, try to prevent collisions from happening because fixing them in a wireless environment is very difficult/expensive/not possible/can't be done in a reasonable time frame, and so on...
Much the same way, (R)UP is risc driven and architecture centric. It's goal is to minimize the riscs and avoid future troubles. Much of the energy in sucha project is focused on getting the architecture just right, so it is able to either dodge or withstand the riscs and the problems that are going to attack it during the lifetime of the project. This basicly means that the architects must forsee all the troubles that the future will bring, and dream up a plan to deal with these future issues.
The rationale behind this is, that fixing bugs, or making any architectually significant change, is very expensive and should be avoided if at all possible. Of course it's expensive to alter an architecture, that took 8 people six months to build, in the second half of your Construction phases (or in Transition, for that matter) - especially when neither the developers nor the architecture are used to this kind of refactoring.
Also please factor in, that the architects in a (R)UP project tend to be the expensive kind of talent.

On the other hand, cabled ethernet is collision correcting. This means, that if we get a packet collision in a cable, we currect the mishap by resending the two colliding packages - each after a randomly chosen amount of milliseconds have passed. If the packages collide again (both senders waited an almost equal amount of time), we wait again.
This way, we don't waste any time trying to avoid making any mistakes - if something bad happens, we fix it and move on.
This fits nicely with the way agile methodologies handles risc. We solve the problems and errors as they turn up. Why do we do it this way? Because we know we can't forsee all and steer around all riscs and issues. In other words: we cannot avoid all errors - errors will happen no matter what we do. So instead of doing our best to avoid errors, we do our best at fixing errors when they occur.

The punch line is, that because code entropy tends to steadily go up, we need to refactor the code every now and then. But making the descision to refactor a piece of code takes courage, because it means throwing out something that was expensive to create in the first place. The customer also needs to be convinced that this is a good idea to avoid the car salesman example mentioned earlier, and doing that may not be easy since it dosn't make imediate business sence to a lot of people.

So that's (one of the reasons) why agile developers in particular need courage: It takes balls to get stuff done - collision correcting style.

No comments

Dinosaurs on Scrum

Friday, 10. November 2006, 15:48:00

,

No comments