Questioning YAGNI
Thursday, 13. November 2008, 22:39:42
There is another popular concept in Object Oriented Programming that really bothers me. The concept of encapsulating the internals of base classes from derived classes. I'll state it straight up that I
I can already see the vast majority of programmers profusely opposing what I've just said. I can foresee people giving anecdotal evidence to the contrary. However, we must look at where the advantages are. Why do people want the base class to be encapsulated from its derived classes?
The answer to that question lies in how developers write software. The volatility of your code is directly proportional to how much its developers will want to use base class encapsulation. If you end up changing your code a lot, then you'll likely end up changing base classes and if the internals are accessible by derived classes, you're in a world of hurt. And therein lies the problem. These developers are using incomplete base classes. I understand that requirements change all the time, but if you end up changing base classes, then you're doing it wrong.
Your base classes and data structures should be the foundation of your overall software system for the current application. These are tools made available to the rest of your software. As such, they should have a consistent expected behavior. Changing the behavior of your base classes and tools after the fact is a surefire way to cause havoc within your software.
I've often heard of an example where the internals of base classes are hidden away is the Windows GUI system. I had to laugh. You can access windows properties from anywhere in your code. You can even inspect another applications window properties. There is nothing hidden about the Windows GUI. With Borland compilers, you can use something called the VCL (Visual Component Library). It is an OOP wrapper for the Windows GUI. It works extremely well, but it does have limitations. Often, you can derive these classes and create your own. But VCL uses the ideology of encapsulating the internals of base classes. As such, derived classes have no access to its base classes' internals. Visual components are essentially state objects. They hold properties. So if you try and create a different kind of component that has a slightly modified behavior, you are out of luck because the base class will be out of sync with its derived class. This happens a LOT. So you're left with only one option which is to create your own component from scratch when all you needed was access to one internal property.
I've said it before and I'll say it again, the 'private' keyword should not exist. The main issue is this. If you have a list of private internal states and only declare public virtual functions, you are out of luck. Even if you override those virtual functions, you have no access to the internal state and cannot update it. That means you have to duplicate and keep track of all internal states, many of which you have no clue about because it is all hidden away. In short, it means you're writing from scratch even though you may be deriving from a base class. I've seen this happen a LOT.
Let me say it again, if you use the 'private' keyword, you are sacrificing extensibility. If you end up changing base classes, then may I suggest you stop doing that and first try to figure out what exactly you are supposed to be doing.
The use of the 'private' keyword ties in with the YAGNI principle. I've heard this a lot, especially by library writers. On the surface, it makes sense. Once you include a feature, you can never take it out. And the more features you have, the more that can go wrong. While this is true, you can't sacrifice a complete feature set for the sake of YAGNI. And this is where I veer off from accepted OOP practices yet again. I believe in feature sets. This is more commonly known as structured programming. Or separation of concerns.
Let me rewind and talk about classifications. Take a something as simple as a rock. Do we categorize it as a solid? As an inert object? As a tool? As a projectile? As a weapon? As a building material? The list is endless. When we write software, we have the tendency to globally classify objects. Classes are global and many languages don't have proper features for classification. The truth is that the classification of an object should not be determined by its base class, but rather by its user (or how it is used, i.e. how it is interacting with other objects).
To see this more clearly, consider a human being. Someone can be a government official, a tennis player (or any of the countless sports), a church member, a neighborhood watch member, etc. It'd be insane to use these classifications as base classes. Classifications can change depending on the location of the entity and its interactions. As such, classifications are better represented as members of a set. Base classes are used to retain and extend global properties of the entity itself. And if you understand this concept, then you can start to see exactly why the base class encapsulation principle is flawed.
If you use base classes as a classification, and we must as there are no other tools available, then we run into a problem. Classifications MUST change. Since these things change depending on its use, we must change the base classes if the requirements of the software change. This is why people choose base class encapsulation so that any changes are reflected through virtual functions and not its internals which would be unmaintainable.
The solution is to remove classification from being a base class. Instead, use the separation of concerns principle. This dates from the 70s, but it is used everywhere, mostly unknowingly. What you do is have another object that can view the primary object in a manner of its choosing. IOW, you use a helper object to represent another class as if it were in a different classification. A common issue with this is how to tag extra information. You need either an extension API or simply add it as a secondary base class if it's common enough. That way, you keep different properties grouped separately and can be used with different types of objects.
The concept of having different views for the same object is very powerful. It is something that is lacking in most programming languages. In fact, all accepted programming practices frown upon this. And even with all the issues that keep coming up, no one is looking at WHY these issues exist. And it doesn't surprise me that they do not look more closely. Helper objects (or view objects) MUST be able to look at the internals of the primary object upon which they are acting. Helper objects should also be used for enabling interactions between two primary objects. That way, you can still keep encapsulation, but at the helper object level, not at the primary object level. Primary objects should only declare functions where it can accomplish the task without the assistance of another object. Otherwise, you get into the problem of who should commit an action.
This last issue is so pervasive that I'm amazed programmers live with it. When two objects must interact, which one should perform the action? The answer is neither. If one of them performs the action, then you are including excessive and unnecessary coupling. If neither one performs the action, then the two primary objects remain reusable in other applications or in other parts of the same application. The helper object will handle the action and as such is specific to your application. And that's the way it should be. Your application should be about acting upon independent objects and manipulating them in a way that produces the expected result. But once you start including those specific actions within the primary objects themselves, you lose any concept of reusability and extensibility. Don't even think about those two features. They're gone.
Once you understand how these things affect each other, you start to see better ways to organize your software. You know you're on the right track when you can keep adding things to your software LONG after critical decisions have been made. So when you hear mention of YAGNI, be wary of how they are developing software. Check if they are putting all functionality within a single class. Java is especially worrisome as they do not allow multiple inheritance. There are ways around it, but it isn't very much fun. Also check if they include methods that call methods in other objects. This is 100% likely. The real world has solved this problem time and again. You have primary objects and then you have other objects that act upon those primary objects. If you keep this principle in mind, you will find your applications so much easier to build and maintain. You'll also notice that programming is a lot more fun.


Anonymous # 15. November 2008, 18:42
"The logic behind this is that the less you have, the less that can go wrong."
The logic behind YAGNI is that by ignoring unknown future requirements (and sometimes even known future requirements) you can achieve the most simple and effective software design.
Making your software extensible in some way before that extensibility is actually needed is both wasting time and adding complexity to your code.
You mention extensibility and reusability often but when it comes to maintaining application code, maintainability - the ability to change existing code - is paramount. Making code reusable is worthless unless it is actually reused. That is why the time to make such code reusable is when the opportunity for reuse presents itself.
I won't get into our old debate of how to design objects, but YAGNI is not an OO concept so we can debate that in separate... :-)
Vorlath # 15. November 2008, 19:25
I'm sorry, but I don't understand how what I said different than what you said. They seem identical. "less that can go wrong" = "most simple", no? Otherwise, why would you need simple if you're not concerned about things going wrong?
I disagree 100%. Programming languages have zero concepts of extensibility. If the programmer doesn't include it, who will? So you end up with ridiculous notions like changing existing code that can only end up in a world of hurt.
My entire blog entry was dedicated to trying to debunk this erroneous conclusion that is so pervasive. If you change existing code, you're doing it wrong. Other than bug fixes, NEVER EVER change existing code. That's poor man's programming. There are better ways to code than this. If ANYONE reading this is changing existing code, then I beg you to relearn how to code. You're essentially pounding nails with your fist.
So instead you're making your code un-reusable from the start. Ok, but understand that I don't want to suffer that fate and do not have the same limitations as you do because we use different styles.
Anonymous # 15. November 2008, 21:52
If you do not change existing code, how do you change the application's behavior?
Version 1 of my application had the phrase "Version 1" written in bold letters at the top left corner of the screen, version 2 needs to have the phrase "Version 2" written in italic at the bottom right corner. How do you do that without changing existing code?
Sean Conner # 15. November 2008, 22:02
One limitation? You can only edit files in the same directory. That's not an oversight on the original programmer's part, it's a limitation of MS-DOS 1.x, which it was written for. MS-DOS 1.x didn't support directories, so the program has no concept of directories and can't deal with them. Even if I had the source code (and I don't) how would I go about fixing this problem?
Another limitation---it requires MS-DOS 1.x. Sure, I can still run the editor (actually tested through Windows 98; I assume it would still run under 2k and XP), but my primary platform isn't MS-DOS these days. Nor is it Windows. Again, how would I fix this without touching the source code?
And for heaven's sake, why aren't we still using Mosaic?
Vorlath # 16. November 2008, 02:15
By adding to it. Not by changing it. But you can only do this if you have an extension system in place.
It's data, not code. The behaviour is the same in both cases.
If there was a module system in place for handling multiple filesystems, you wouldn't need to change anything. You could write a new module and be done with it. For an example of this, look at the IJG jpeg code for reading images from memory instead of a file. You write a new module and tell the jpeg code to use that for reading in the image data. In fact, I use Borland compilers and the default file handling code doesn't work because it's written with Microsoft's stdio library. So I wrote my own module to use Borland's I/O routines and now I can load jpeg images from Borland using IJG. But I can only do this because IJG has an extension system in place.
Also, you're using legacy software. You often don't have a choice but to change existing code. There are better ways to do things these days. Note that IJG was written in 1991 and hasn't been updated since 1998. Unfortunately (and unlike IJG), people today tend to write software for one time usage, not for future use. Bad libraries tend to show their deficiencies rather quickly because people will try to use them. With applications, those deficiencies are still there, but there is an active resistance to actually do it correctly even when people have a difficult time adding things to it.
------
Changing code is where you change the behaviour of an existing module. Taking the MS-DOS example, that would be like changing the existing module to now work with Windows. Don't do that. Write a new module and use that instead. But now you need a way to use that new module. Most people will not want to write an extension system or an interface even at this point. So they end up changing existing code. I've done this plenty of time with other people's code. I'm not a big fan of it. But at least I try to keep the changes to a minimum.
What I'm saying isn't rocket science. In the real world, whenever you need to change something, you need to find the boundaries and then rip out everything inside. This is often the case with renovations. But with software, you can implement pre-existing boundary points where you can replace sections of code at will. There is no need to rip anything out unless you no longer need it. Unfortunately, people don't understand this concept. They use the pyramid scheme where updating any part will risk making the rest of the application fall down.
If you don't have points in your software where you can completely remove a module and put in a new one, you are programming in a monolithic way. There is NOTHING in any programming language that will automatically provide extensability. And I'm still shocked that people willingly refuse to build software with extensability in mind.
Also, I'm not saying to build extension systems everywhere. But if you can foresee people changing a part of your application, then make it easier for them. The problem is that today's programming languages make this difficult to do. But once you grow accustomed to this kind of programming, you'll never go back. The headaches simply aren't there anymore and you'll find that even if you eventually DO need to change existing code, it's rather simple to do because everything works on helper objects. The area of code that will be affected will be restricted to an absolute minimum.
In fact, the only time I ever change existing code is to include a module system that wasn't there originally.
Vorlath # 16. November 2008, 02:40
With games, you can decide to take input from the keyboard, mouse, gamepad or whatever else. So writing your code to directly read (with OS routines) from the keyboard or mouse is not too smart. Eventually, you'll want to use another form of input or even remap what keys activate what actions. So what do you do? Change the code? Good luck with that.
No, you need to have a separate section for reading input. This section will translate inputs into actions. The actions remain static, but you can map different inputs to whatever actions you want. And you can only do this if you have input reading as a separate module. This is what makes input configuration possible in EVERY game you play. It is also what allows the programmers to add new devices with relative ease. The existing code doesn't change.
Sean Conner # 17. November 2008, 01:40
InputStream HttpGet(string location)
InputStream HttpPost(string location,string data) /* added shortly thereafter */
Several years go by and this works fine. Then HTTP v1.0 is defined, so that client programs no longer have to sniff the resulting datastream to figure out what it is actually getting. There's also a means by which the client can send along additional information with the request, and for the server to send back additional information about the resource. Oh, and a few additional methods were defined. This isn't a complete disaster though, because servers are backwards compatible with HTTP v0.9 so your existing code can still work. The new protocol though, is different enough that you might need to update the code.
Oh, but you can't. You can only add existing code. Okay, new API then:
HTTP HttpNewRequest(string location,METHOD method);
int HttpNewStatus(HTTP request);
InputStream HttpNewData(HTTP request);
MIMEType HttpNewDataType(HTTP request);
Original code can still use the original API since servers are backwards compatible. That is, until HTTP v1.1 is codified and the use of virtual sites (those that share an IP address) become very common. Then HTTP v0.9 breaks down. Well, it still work if the original site is the primary default site for a given IP address, but if it isn't, then what you get back is something you aren't expecting.
But you can't change code. You can only add code, right? Now what? You can't fix HttpGet() because that's changing code. You can't fix the "legacy" code to use the new API because that's ... changing code.
Oh, and HTTP 1.1 added a very significant change---persistent connections. No longer do you need to make multiple connections to a server to get multiple resources; you can now connect once, and make multiple requests over a single connection. So that requires some new APIs:
HTTP HttpNewerConnection(string host);
int HttpNewerRequest(HTTP connection,string resource,METHOD method);
int HttpNewerClose(HTTP);
(and so on) or a modification of the older ... sorry! That's right! No changing code, only adding code! Sorry about that.
You might counter that people implementing a protocol should know better and expect the protocol to expand, but how long had SMTP been around before a change? The better part of 20 years or so? (AUTH added). IPv4 practically unchanged since it was originally developed 30 years ago or so (changes in QoS bits, and some minor tweaks for congestion control and what not, mostly stuff that isn't even reflected in the userland API). Proposing the 2008 HTTP API back in 1991 and most people would have either bitched about it being way too complex for such a simple protocol. And besides, HTTP didn't have status codes back in 1991 (so it most likely would have been either a boolean, or some other code that no longer matches what is actually used today, so you still would have code to change, um, add).
Sean Conner # 17. November 2008, 01:43
Vorlath # 17. November 2008, 03:35
1. I never said you can't delete code. I said you can't change code.
2. Your example is based on using YAGNI.
About point #1, I actually encourage deleting code. I want people to be able to add or swap modules with relative ease. What I don't want to see is using an OS or library API calls all over your code where you'd have to hunt them all down if the API ever changes. Too often, I see people use other people's API's instead of defining their own for the application's use only.
That last sentence is why Java became popular. People could use a single API that came with the Java libraries. But those libraries DO change. And then people curse... A LOT! That's YAGNI for ya.
I gave an example of proper programming with IJG and with input processing. I'll give you another. ProjectV uses the same data structure for all data types. I have helper objects for each type so that I can manipulate the same data in a different way. Adding new types is especially easy. And there is a common set of functionality that is available to all helper objects. I have a long list of using this technique. I use it everywhere I can. But I will admit that I don't use it as cleanly as I should have in certain places, but am trying to get better at it. All the problems I have always happens when I'm don't follow my own advice. Like when I use YAGNI or other archaic techniques such as encapsulation.
About point #2, you're using legacy software. Like I said, sometimes you don't have choice but to change the way it works. And hopefully, you're adding an extension system so that next time, you won't have to change existing code again.
Even so, HTTP having a version number should have been an indication that things could change, but there is something wrong with your example. Unless you accessed sites by IP, the interface need not change. Since the beggining of the web, you always accessed a site by URL. Whether you were using version 0.9 or 1.1, you can always decompose the URL to produce the proper information for whatever HTTP version. Have a module for each one and start with 1.1 (or whatever is most recent). The server should return an error code if it cannot handle it, and you'll know which module to try next.
But you can't do that if you use YAGNI. Instead, you'd hack something away until next time. If you have an extension system in place, you can handle a fallback mechanism with relative ease.
And the other features are bogus as far as the example is concerned because the original app would have no concept of those things anyhow. So you can't change something that never existed in the first place.
Please note that I'm not saying you can't change legacy software (or badly written software). You don't have a choice in these cases. I'm saying there are better techniques available today that won't require people to change your software years from now.
Unfortunately, all these examples will likely fall on deaf ears until the people reading this actually has first hand experience using extension modules. I can't tell you how much fun they are to use and work with. My software actually gets easier to maintain over time. Of all the techniques out there, this is the only one where I've seen this happens.
Sean Conner # 17. November 2008, 04:09
As far as HTTP and IP addresses go, you do realize that the following three URLs:
http://www.flummux.org/
http://boston.conman.org/
http://66.252.224.242/
are in fact different sites, even though they all resolve to the same IP address? Back when HTTP was first developed, each website HAD its own IP address, so the notion of hosting multple sites on a single box wasn't even asked (remember, HTTP was developed in 1990). And really, it wasn't until the late 90s could you even think of commercially hosting multiple sites on a single IP address (HTTP protocol was changed to include hostname information).
My argument there was, you can't fix the original HttpGet() method because you don't believe in changing code.
Also, the version numbers in HTTP didn't get added until 1.0, sometime in the mid to late 90s. You also don't know what type of server you are talking to until you connect. HTTP 0.9 servers (dont' know if any exist in the wild anymore) just ignore pretty much everything in the request, while HTTP 1.x servers will typically ignore anything they don't recognize (and inform the client what version they do support, so both the client and the server have to fall back during the conversation).
I digress. A question: how often have you changed code in Project V? And why? According to you, you should only be adding code, not changing it.
Vorlath # 17. November 2008, 05:08
And about new features, the existing application knows nothing about them. So by definition, you cannot change anything that doesn't exist. You will have to add new functionality. By doing so, I hope you will add an extension mechanism to make this easier in the future and also to make the current task as simple as possible.
About Project V, I'm still developing it. I'm free to change whatever I want though I can usually avoid it. However, most of my changes involve including an extension system. I've done it for Direct3D (and OpenGL) although some parts still remain to be updated. I have one data structure for all data types represented by Project V and I use helper objects for viewing different types (though they do have a common set of functionality as well). The GUI system has an extension system where you can create your own custom GUI components. For the type system, I added the concept of flattened type hierarchies where all the data is directly accessible (no need to walk the tree). This was added onto the existing data network handling mechanism (and was easy to do because of the helper object extension system). The Project V IDE has a plugin mechanism for displaying custom data. For example, when you have a new type, values for those types need to be displayed. So there is a way to plugin your own built-in display functions or custom Project V components to handle this task. Communication between Project V components is also done through an extension system. Communication can be done via memory or socket. I may add more later. Anyways, I could go on and on about how I build on top of what's there.
Right now, everything I do on Project V is adding and using what's already there. This is how everything in the real world is done. I can use the extensive knowledge of the past 10 millennia for what I'm am doing, as far as putting my software together is concerned. I have a set of tools (helper objects), raw data (manipulated by those tools), and extension systems where I can create more tools to be added in (or where I can create more systems that can interact with existing ones with the use of those same helper objects).
As far as I can tell, the only thing I can see for your argument is not being able to predict the future. Certainly, this happens and there is nothing you can do about it. As Agassi, the best returner in Tennis once said, "Don't worry about the ones you can't see. Worry about the ones you can." But often, people end up using external APIs without defining their own. They use it all throughout their software and then they notice that this API can't handle the last 1% which is critical to the customer's requirements. What do they do? The entire application is garbage. You can't go through the entire application and change every usage of the API. That's just nuts. The entire application was built around this API. This is exactly the same problem as the game input filtering example I gave earlier. It's a colossal nightmare.
However, if they had noticed that this was an external API (something they can readily SEE), then they would have created their own internal API that fit their requirements. The implementation would have used the external API. Even if you change the external API, it's not that big a deal. You just need to write a NEW implementation that uses the new API and NONE of the existing code will have changed (other than perhaps the object factory which is why you need an extension mechanism in the first place).