Software Development

I support the notion of messages being able to write and read its contents. But I don't support more functionality than that. However, it must be noted that sometimes you need to move fully functional objects around. This is not so much to pass a message, but rather to move certain functionality to a new environment. I find it disheartning that there is no such distinction in OO languages that I've seen so far. (The old everything is an object thing again.)

About your tight coupling, what do you say about interfaces? With a new format, you need to decide if you will replace it or enhance the existing format. I've almost never seen a case where you want to remove an old format unless you're basically rewriting the application. Usually, you want to add things to the existing format. What better way to just add it to the existing data structure and still have the old code work the way it used to. For objects to use this enhanced format, use helper classes. That way, each system can have only the helper classes it needs without affecting any other part of the message. This is a real boon when you're trying to separate different functionality affecting the same message.

I disagree with the notion that an object isn't meant to understand or process the message. This is EXACTLY what it's supposed to do. Otherwise, it is nothing more than a filter that directs messages to one place or another. While this does have uses, it falls under the "moving" category I mentioned above. It's a message only in the sense that a delivery person sees a message. The real message comes into effect when it arrives at its destination to be used by the receiver. Unless it's to be placed in a list of some sort, the receiver MUST understand the message.

The idea behind encapsulating the message is a tempting one because we think that hiding the details will help us, but I've NEVER, not once, found this to be true. I use helper classes. For example, my backgammon server uses global helper classes for message passing between the server and clients. It's great because anyone that needs to read or write messages can use the helper class. If you want to add functionality, you can extend the helper class and then everyone has access to it, or you can have parts of the message that only certain objects can use by using two separate helper classes. It has much better "safety" than this private, protected BS, and it's impossible with methods alone without a lot of setup with access schemes.

I kept wondering why messages where the data was public was so much easier to program for. It's because encapsulation is a lie when it comes to message passing. If I am to use the message, I must understand it. I can't send methods over the Internet. I can only send data. This is the way it works everywhere. When we start to take advantage that on a single computer, we can provide functionality together with the data, we get caught when it isn't possible to send both. This will become more of a factor when multi-core and distributed programming starts to take off. In these scenarios, you don't want to group methods or functionality with the data. Because then you start shipping methods that you may not need to the destination, if at all possible. Helper classes has the advantage that they act like independant modules. You can request only the functionality you need and is not coupled to anything else, not even the data it acts on, other than its format.

About your last paragraph, how is an object meant to do some "logic" if it can't understand the message (other than routing)? I understand the cleverness of using polymorphism so that you can retrieve information about the message no matter what it is, but I'd rather see polymorphism with the helper classes. Unfortunately, most (all?) static OO languages have no support for this. I also REALLY dislike the fact that messages that are ojects can themselves act on other objects. I've seen this all to often where messages end up registering themselves for receiving events and then this very message would initiate the message passing mechanism. This is very dangerous. But it's done everywhere.

I can find nothing good about passing off encapsulated messages. In fact, I've seen situation where using them makes the application impossible to complete.

"Understanding the message in the form of extracting data from received objects should be avoided." I take the opposite view. In fact, I say that not extracting the data cannot lead to functioning software unless you have a monolithic framework.

I already explained why this encapsulation MUST fail in the case of messages. Both the sender and the receiver MUST understand the message. Having get and setters is not the issue. But most people go beyond this and add functionality. If there is an interaction to be done between the receiver and the object, then the receiver must know the internals of the message (or with get/set), OR the message must know the internals of the receiver. There can be no interaction of data unless one of these objects has access to both at the same time and understands both. I think this is generally accepted in the real world. I don't know why we think programming works differently. And my experience has been that this precise problem is the root of most software headaches.

That's why helper classes are cool, because you can extend them in whatever system need to interact with the actual message (or as you correctly point out are the arguments to the command/method). In any case, helper objects usually take the place of get/setters. The information extracted is raw data, or in equivalent form taht the receiver knows how to manipulate. That's my main point.

I'd just like to mention something about your network message passing because it's especially common in the programming field. The distribution of the class on both machines is indeed possible beforehand (or even during execution by shipping the code), but this only goes to show that both the source and the destination must understand the raw data. In this case, it is a self-contained solution. But more to the point would be data from two or more sources that need to interact. When it's one to one, you are correct that you can produce some "nice" or "well designed" code for your architecture. However, encapsulation is not there as both the sender and receiver understand the data. Take C++ for example. It is notorious for being ambiguous about making two different objects interact, especially when it comes to operators. Java has taken this out almost completly except in special cases.

There is no scenario that can achieve encapsulation when it comes to making different types of data interact. One object must understand how to manipulate all of this data together. Just knowing the methods that you can use to manipulate this data is breaking encapsulation because you are only deferring the action by one level. What this level of indirection gets you is the inability to expand on the functionality available to this data without physically changing the object and possibly causing adverse effects in other parts of your application. That's why encapsulation often makes applications impossible to complete. There is too much coupling.

BTW, helper classes avoid making _specific_ changes everywhere and also helps avoid coupling in areas where it's not needed leading to much easier extensibility. I understand your point of view as it is the common view (from what I can tell). But I think there should be more discussion in the programming community about the adverse effects of encapsulating messages. There seems to be some general avoidance of the reality of this problem.

What you say here is pretty much the same as what I'm saying. I think we just see it differently, and as such I've got some different design strategies which answering your question is a good followup.

Let's say I have a multi-user game room. I need a list of users and lists of rooms. Each room has its own list of users and its own rules for the game being played. For a simple example, take the socket mechanism. It needs the socket number, its state, input and output buffers. But many things can use sockets, not just users. There are database objects and relay objects that use sockets. Should I make the socket engine recognise them all? Of course not.

So I wrote a helper class that can extract commands as well as insert them (from the buffers). Remember that the socket engine cares not how the information is formatted. I cannot put this is the socket engine. Yet, I cannot put it in the socket info class either because this would lock the buffer into a specific format for the data stored. I can't subclasss it because then I can't place the socket engine in a DLL. I want to make each system as independant as possible.

When you open a file, you get a handle. When you open a window, you get a handle. When you allocate memory, you get a pointer. The same should happen with any module in your application. The socket engine mentioned here should provide a handle or data structure that any object can use. What I did was provide helper classes that could read input buffers and create message info (command + name/value pairs). If I want to use a different format for messages (say for the database only), I am free to do so, yet still use the same socket system.

At this point, you can add delegation to the objects (via subclassing) that use these helper classes if you wish so that is looks more OO like. For example, instead of
"SocketSystem->SendMessage(user->socket_info,msg);"
you can use the more traditional
"user->SendMessage(msg);"

This is but one very simple example. But the uses are everywhere. Game rooms can have different games for example. Each game requires to store some info about each player. Do I make each game recognise my custom user class? Of course not. I store in the user class the data the game needs (usually allocated by the game itself) and the game logic is stored in a helper class.

One thing I found frustrating was that C++ has no mechanism for creating true virtual objects. I would like to be able to associate methods from a helper class as if it were included directly in the object itself. For example, if I have user object A and helper class B, I'd like to be able to use methods of B as if it was in A. This would remove the need for delegation. As the example above shows, I could do something like
"C = MakeVirtual(A,B,A->B_data)"
Now if B has a method
"SendMessage(B_data bdat, char *msg)"
you could just go
"C->SendMessage(mymsg);"

A problem is that if you send a handle, there is usually no backlink to the overall object that uses this object. For example, a user class that contains a socket handle. By passing the socket handle, it is difficult to obtain the user class again unless you do a search or store it in some fashion for retrieval.

class socket_info
{
  int socket;
  char *in_buf;
  char *out_buf;
};

class user
{
  String username;
  socket_info sinfo;
  timer_info tinfo; // etc..
};

class SocketHelper
{
  SocketHelper(socket_info);
  ~SocketHelper();
  void SendMessage(char *msg);
};

// A plain way to use this.
user A;
SocketHelper *sh = new SocketHelper(&A.sinfo);
sh->SendMessage("Hello!");
delete sh;

// Above is obviously tedious and reference to original user is lost.
// This may be better.
user A;
Virtual<A,SocketHelper,&A.sinfo> vA;
// Now all three are linked and is much easier to use.
vA->SendMessage("Hello!");

// For dB object
class DB
{
  String DBname;
  socket_info sinfo;
  ... // other DB info.
};

DB db;
// Note that I can use a different helper class if 
// I'm using a different format or functionality.
Virtual<db,SocketHelper,&db.sinfo> vdb;
vdb->SendMessage("SELECT EVERYTHING FROM MYWORLD WHERE EVERYTHING = 'COOL'");

But I find these kinds of templates a little odd because it doesn't help when you need more than one piece of data or multiple helper classes. I usually end up subclassing the user class and delegating to the helper object(s) so that it looks like standard OO and can add more specialised usage. What I like is the ability to swap helper classes without affecting anything else.

Anyways, this is where most of my ideas come from. This problem plagues most large projects I've seen. Things get so coupled that they can't change anything. With streaming, I don't have any of these problems. But I think a written language could benefit from something that alleviates these problems.

Note that when I use message, I usually mean sending data, not the OO kind of message unless specifically mentioned as an "OO message". I'll use method most likely.

socket_info is what the socket system uses and the socket system has no use for SocketHelper. SocketHelper enables outside objects to store data such as the "Hello" text message and then get the socket system to send it out. If another object wants to use different formatting for the message, this would be impossible. Well, you could subclass it, but then this class would be locked to one kind of functionality. I often use different sets of functionality on the same data. In fact, this HAS to happen. All computers do is transform data.

I do use delegation for the DB and user objects. They are both delegated to the SocketHelper classes along with other more specialised functionality before and after using the helper class. For example, getting rows from the DB.

I have specialised methods like user->SendStats() depending on high frequency usage. But there's too many client/server messages to each have their own method. You can build your own message (like chat and server commands) and send it. That's where helper classes really shine.

By the DLL example, I only meant that I wanted a certain group of classes (what I call a system like the socket handling mechanism) to be completely independant. I should be able to completely separate it and still keep working. It should not care what is using it or what objects need to use sockets. It does requires a handle to a certain data structure and that's what it uses. Wherever you put this data structure is up to you. The unfortunate thing is that sometimes the objects that contain these handles need to be passed along from system to system. If the socket system only cares about its own handles, how can you pass along the user (or other object) that was using the handle if the reference to the user is no longer available? Having backlinks is nice, but requiring it on all handles is a waste of memory.

I tried using templates and it does alleviate the problem somewhat. But it's a hack more than anything.

Here's the deal. I have a user class that needs functionality that can make X and Y interact as well as Y and Z. My DB class needs (X,Y) functionality too, but not (Y,Z). It needs (Y,K) instead. It also needs (L,K,Y) for output. What I like about helper classes is that you can mix and match functionality. I can't do that if the functionality is locked in with the data structure itself. If I separate functionality from the data it acts on, I can mix and match what I need.

You need to use input/output buffers with name/value pair lists from messages, we have that. You need to use name/value pairs for dispatching which may lead to timer events or to the DB, we have helper classes for timers as well as for DB handling. These systems in turn (like DB) will need to use input/output buffers with name/value pairs, but from the database. We can use the same helper class as before. Game system needs to be able to send messages to users, but custom game related messages. Ok, we have that helper class already that lets you build your own messages. Already, this helper class is used three times for three completely unrelated purposes.

Now, let's say I use someone else's database and it doesn't work with my messages anymore. Well, if I were using encapsulation, I'd be stuck or I'd have to manually replace methods in the DB class. With helper classes, you simply build your new functionality for these new messages and replace the old one. The boundary for this functionality is very well defined. Since these are new messages, you're more than likely to have special handling, so you'll have different (and more) methods available to you. Yet, the same socket mechanism is used. Only the format and handling of the messages has changed. And you don't affect objects using the old message formatting.

I couldn't have used interfaces because of new features added. I can't use subclassing because it needs to use multiple data structures (or objects) at once. I can't use encapsulation for the same reasons. But what I am doing is achieving homogeneous grouping of functionality.

Let's break it down to something really simple in OO fashion. A SocketInfo class an a Message class. The Message class is used for constructing or querying info from a client/server message.

class buffer
{
  int capacity;
  int max_capacity;
  int size;
  char *data;
}

class SocketInfo
{
private:
  int socket;
  int state;
  buffer in,out; // can use STL instead
public:
  
};

class StringPair
{
  String name;
  String value;
};

class Message
{
private:
  String command;
  vector<StringPair> pairs;
public:
  void SetCommand(String cmd);
  void SetValue(String name, String value);
  String GetCommand();
  String GetValue(String name);
}

Not very exciting and incomplete. Message and SocketInfo are the two main classes. Here's where my "problem" comes in. I want to be able to convert my Message class so that it can be sent out over a socket. To do this, I have to place the data in the out buffer (we can assume the SocketInfo knows how to send this data).

So how do I do that? In OOP, we usually send OO messages/methods to the object so that it may act on itself. This means that I should have a method in SocketInfo that takes a Message object as a parameter. The problem is that I can't do that for the object itself because it's already defined and the socket system has no need of this extra functionality. We don't want to contaminate a perfectly separable module by adding a new class that it doesn't need into this module. So let's try to subclass it instead. Well, what if I want to send and receive not only Messages, but other data as well over the Internet on this very same socket? A subclass locks me into one specific data format. What would be nice is if I could change the class at runtime to use a different set of methods. But I also don't like that SocketInfo knows about the Message class. It should care less about the Message class or any other class. It's not its job. It's real job is handling the buffers and nothing else. It should not know or care what the meaning of the data stored in its buffers is or how it's formatted.

So instead of using switchable vtables (not sure how that would be done anyhow), I use different helper classes along with some delegation when needed. Or I use the same helper classes within different composite objects. But to use this technique, I must not use encapsulation other than get/setters and even then, that gets in the way much of the time.

So for the above example, I would have a third (fifth?) class that is the helper class which understands both of the SocketInfo and Message classes and both their internals if necessary. This helper class can even add its own data structures to the internals of these two classes to keep state or other data. One such use for Messages is that you don't want to keep converting messages that go out to a few thousand client computers. So the helper class actually takes over the Message class' functionality by keeping track if the message data has changed. If so, it will rebuild a cached version of its output. Otherwise, it will just use the cached version ready to be stored directly into the output buffer. In this case, it's only about efficiency, but there could be other uses.

Note too that I don't create more than one or two instances of these helper classes per module. I reuse the same ones. The reason is simple. Other modules using the same data will have their own helper classes. So when these modules communicate, they send the raw data only and each module is free to use the functionality they want on this data. Like I said, there are minor complications to this (backlinks). But I rather like it. It's the cleanest and most extensible I've ever designed an architecture.

This also adds to what I said in the past about having each module see the data through its own eyes. To sum up, I need to be able to add functionality without injecting coupling to the existing modules or systems.

Interfaces could work if I were using Java I suppose. It would solve the linking of the class with the data structures. Meaning that I can pass any object to the socket system as long as the interface is implemented. I like this a LOT! C++ doesn't have that. But even so, these interfaces still don't hide that much. The parameters to their functions are pretty bare, which is what my talk about breaking encapsulation is about.

The way you solve the Message class by using an interface is exatly what I'm talking about. I've tried something similar to what you're doing and many other ways as can be seen above. I wish there was a way that I didn't have to link together Message and Socket that directly. Subclassing is tempting, but that limits the class to one particular expansion of functionality. I like the previous comments. It's very reflective of current programming trends. Not that it's bad. Quite the contrary. I simply think there is a limation to these languages. I haven't looked to much at others, but if they have a solution, I would likely start using it.

We need ways to introduce extra functionality to existing objects without modifydng them in such a way that they now require external objects to function. Yes, for this specific use, it will require another object for the extra functionality. But it should be optional. When I use the original object, the source should be exactly the same as it always was and be completely independant. Added "plugins" would allow it to achieve this extra functionality, but without having to change the source of the original object. I think that explains it better.

Because adding it into the class locks it into that one specific use. This is unacceptable for any large scale project where objects routinely get used in multiple locations for different purposes.

BTW, the dependancy you see is within one project. It is but an illusion. I'm thinking larger scale to multiple projects. If I have class A, I want to be able to use it in multiple projects as is. But if I want to add functionality to class A in ONE of my projects, I should be able to do that without touching class A (because I don't want to couple its definition to anything else). My problem is that this added functionality should be applicable to different classes. There are very real uses for this. I have a system for this and it's the most flexible and extensible system I've ever used. I only program this way now because it's SO much easier.

I'd like to pick at your external objecst and high level objects argument for a second. High level objects can't act on anything. They are simply wrappers. For the programmer, this may be fine to a certain extent, but it locks you into that one specific use. Even with interfaces, it locks you into a request mechanism that doesn't reflect many things in the real world. In the real world, most requests happen asynchronously and the receiver usually opens the package and deals with the low level parts him or herself. Even if you delegate the job, someone must eventually understand the information in the package as well as the information in another package. There's no getting around it. And that's why encapsulation is a lie.