Software Development

Correcting The Future

C++ Callbacks

Callbacks in C++ are one of those issues that seems to have been left behind by the C++ committee until recently. Apparently, closures are finally going to make it into the specs. But it's about three decades too late. It would have been nice to have back in the 80's, 90's and in the new millennium. Of course, I didn't use C++ in the 80's. I was using assembly (and BASIC). But that's besides the point. The other critical feature missing from C++ is having properties. Bjarne Stroustrup is dead set against them, so I doubt we'll see them anytime soon. Anyhow, this is why I use Borland C++ Builder even though it's not one of the most popular compilers out there. Borland has had both closures and properties for as long as I can remember.

Back to the topic at hand is callbacks and closures. Closures are usually found in functional languages. When imperative languages like Borland C++ Builder say they have closures, they usually mean something much simpler in that a pointer can also hold a reference to the object instance. So when you invoke the function reference, it knows what object to pass as the function's first parameter. FYI, all non-static member functions have an implicit "this" pointer as a first parameter. That is the parameter that is causing so much trouble with callbacks since we need both the function pointer and the "this" pointer to the object.

So what are the ways to use callbacks that point to methods? There are four ways that are most common. There are certainly other ways of doing this. For now, these are the simplest and quickest.

1. Hack your own closure.

I've done this when porting Borland C++ Builder code to GCC for example. What you do is build a class (or struct) that holds two pointers. One is the function pointer and the other is the object pointer. As to invoking the function, a simple way is to use variable arguments. You need at least one argument for the prototype, but that's no problem since you need to pass the object pointer. Unfortunately, this is prone to errors and you'll have issues with return values.

With the advent of templates, you have much more flexibility and could come up with a solution. In fact, the next two solutions are pre-built versions of this.

2. Use Boost

I don't use Boost, but a lot of people like it. I'll just repeat an example posted by Sc4Freak in this reddit thread.

using namespace boost;

function<int (int)> callback(bind(&TestClass::Foo, &TestObject, _1));

//Call TestObject.Foo(5).
callback(5);

//Change callback to a free function.
callback = SomeRandomFunction;

//Call SomeRandomFunction(8).
callback(8);


You need specific functionality for callbacks to work.

  • Creation
  • Assignment of method
  • Assignment of function
  • Invocation


Creation is done by using a template. You specify the return type and the argument types within the angle brackets. Then you use bind to have the object and its method together. Not sure about the _1. Might be the number of arguments.

Invocation is straightforward as is assignment of a function. I haven't seen assignment of a method though. Perhaps it works the same way, but use bind again, like this.

callback = bind(&TestClass::Foo, &TestObject, _1);

callback(5);


Can someone verify?


3. Use a library such as PlusCallback

I really like this one. It's simply a header file that you include. I may actually use it if I port Project V to other compilers.

Here is the link to the examples.

From the example, suppose we have the following code.

#include <callback.hpp>

struct Dog
{
    void Bark(int volume);
};
 
struct Cat
{
    void Meow(int volume);
};
 
Dog spot, rover; //We have two dogs
Cat felix; //and one cat.
 
//Define a normal function.
void Func(int a);


Creation

//Define a callback to a function returning void and taking
//one int parameter.
cb::Callback1<void, int> speak;


Assignment of method

//Point this callback at spot's Bark method.
speak.Reset(&spot, &Dog::Bark);


Assignment of function

//Callbacks can be set to free functions.
speak = Func;


Invocation

speak(50);


It also supports creation of closure instances with cb::Make1(&felix, &Cat::Meow). The examples show this used within STL containers. I'm not sure how it knows what the argument types are though. Maybe there's some template magic going on since the insert function would require a specific type.

4. Use Borland C++ Builder

This is my favourite option for my own projects because C++ Builder has closures built-in. But when at work or on other projects, you may already be using another compiler. So while this option isn't always possible, it's great when it is.

Creating a closure.

// It's a regular typedef.
// Only difference is you add the keyword __closure
typedef bool (__closure __fastcall *NCompareValue)(NEntry *A, NEntry *B);

NCompareValue myfunc;


This is code from my type system in Project V (only the typedef). You can add custom functions for comparing values of specific types in the NEntry's. What's fantastic about this is that there is no new syntax. The __closure keyword just says to also store object instance.

Assignment of method.

class NTypeModule
{
protected:
  // A built in value comparison.
  bool __fastcall DefaultNCompareValues(NEntry *A, NEntry *B);
};

NTypeModule mymodule;

// This is how you assign a closure.
myfunc = mymodule->DefaultNCompareValues;


Assignment of function.

You cannot do this. A closure is a pointer to a method only in Borland. This isn't really a big deal since you can always write a method to call the static function.

(edit: There is actually an undocumented way to do this, but I can't remember the details. I have used it. From what I recall, there is a function that will create a closure from a static function pointer and you can then use that.)

Invocation

if (myfunc(node1,node2))
{
  // Do stuff here...
}


That's it. There is ZERO new syntax. The only difference is using the __closure keyword when defining the typedef. There is ZERO change anywhere else. Closures are used everywhere in VCL for events (VCL is a C++ library for Windows GUI and structures). It's great being able to use C++ for event handlers and then simply assigning those methods to the proper event. No need for checking what event was triggered with switch statements or any need to use message maps.

But hey, it seems like a pointer that can also hold the object instance in standard C++ has been too much to ask for in these past three decades.

Obviously, the need is there. Of all the things that OOP is supposed to be about, specifically message passing, the simplest of closures should have been available from the beginning. Hopefully, the next revision of C++ will have closures. Unfortunately, it's a little late to the game. Now, if properties could eventually make it in there... well, by that time, Toronto will have won the Stanley Cup.

Out Of Memory Checks (in DataFlow?)Project V Execution Model

Comments

Unregistered user Thursday, March 18, 2010 2:20:09 AM

Hal9001 writes: My friend Vorlath, the C++ syntax is really deprecated, I'd prefer use the pure "C" to get no more troubles in my life ;-)

Unregistered user Friday, March 19, 2010 10:06:50 AM

Dan writes: Hal9001, I don't really agree - C is just as much, if not more, hassle. At least in C++ you have the option (you dont have to!) to use templates and classes and so forth - in C you have to cook all of this yourself or use some complicated libraries to do it. Yet, underneath it all, C and C++ are quite similar, so you don't really win much. Personally, I'd avoid using C and C++ altogether for a more modern language wherever I can. C and C++ are pretty dated now. Sadly, I still find myself using C++ regularly :-/ though less and less as time passes, thankfully.

Unregistered user Friday, March 19, 2010 10:20:10 AM

Dan writes: RE: OOP - the more I use other paradigms, the less I like OOP. OOP seems to make sense when you first start using it, but then you find that you put a lot of effort into forcing something to be OOP, when it would be much easier to use another abstraction. For example, I like to put my data into lists and treat them as streams (which, conveniently, makes it easier to switch to a dataflow computational model). Sometimes its simpler to treat everything as a tree or a graph. Other times the OO style actually does make sense, but full blown OOP is probably still overdoing it - a table of closures is often a good solution for me. The point is that I a lot of the time I don't see the need for complex object hierarchies or layers of abstraction, when we have abstractions available that fit the problem much closer. This is probably why I like Clojure - it makes it easy for me to use whatever abstraction is most suitable at the time. If only it had some proper dataflow support... maybe I'll try and build such a thing some day (theres already a simple dataflow-esque library in clojure-contrib, perhaps it can be adapted and extended...) PS: damn you, Vorlath, for getting me interested in dataflow. You've tainted my "love" for other languages ;-)

Vorlath Friday, March 19, 2010 11:08:47 PM

@Dan:

PS: damn you, Vorlath, for getting me interested in dataflow. You've tainted my "love" for other languages ;-)



I know exactly what you mean. I wish I had someone to blame. I can't even blame JPM since I came up with it on my own. Since discovering dataflow, I can't look at other languages the same way either. It opens up a whole different way of seeing things. What I was fighting against, and what a lot of people are fighting against, is that a lot of problems are meant to be solved using dataflow, but the language isn't quite suited for it. However, I feel you can still use OOP within the framework in certain areas. Even in Project V, I can't completely eliminate global state required by OOP. The code or network itself is a state. Its existence I mean. So sequential programming still has a place.


Dan Pologeadpologea Wednesday, April 21, 2010 3:33:05 PM

I just want to add few things related to what a callback function really is into a FBP (Flow Based Programming) environment, from my perspective. In my view into a FBP system you have only components whether you are talking about data or pure functions. Each component has input and output ports. I will not go deeper than this for now (you can read J.P. Morrison's book and excellent Vorlath's posts here).

The callback from a classical system is a function (executable code) “injected” into the system (usually into another object) that could be “called back” later on.
What a “callback function” could be into a FBP system? In this case the callback is a component that has the input and output ports connected at runtime to another component that is using its services in the same manner a client is using a server. The “callback component” is the server in this case. The big difference is that in a FBP system the client component does not call the server component (function). The components are evaluated automatically by the execution system and IPs (information packets / parameters) are transported automatically from one component to another as they are produced and consumed by these.
In a FBP system a “callback” could be a component that can be wired with a high flexibility. For instance a component injected into a FBP network could connect its ports to other different / various components. This has strong similarities with actor or entity systems. This potential to wire the ports in any configuration possible is very powerful comparing with what a classical «callback» can do because the classical callback has only one point of entry (and exit). It has to return into the same place like any other “classical function”. Into a FBP network a component could be injected and connected in as many ways as the types of its ports allow. The injected component (“callback function”)
does not has to “return” into the only one place where it was called from because there is no call in the first place.
This is a huge advantage that usual callbacks can not accomplish.

The closures appeared because of the necessity to construct new functions during the execution of a program and its free variables to be bounded in the lexical environment. But these “bounds” in FBP are connections created at runtime and the “free variables” are in fact “state variables” of the processing component. By the way, in a “classical system” you can have “state variables” (of an object) and into a FBP system you can have transitory variables stored into connections.

Now, at first glance it appears curious why we need two pointers for the callback: the function pointer and the "this" pointer to the object. Why the life is more complicated than necessary? In my view regarding FBP, we don't need two pointers and also the connection point is not one but multiple (through several input and output ports connected in any configuration possible). It is possible to group all these input and output ports together into a single “plug” (composite port) in order to simplify connecting.

The “method” from OOP is a subcomponent in FBP. This subcomponent is an instance whatever it really is (pure function, data or a mixture). That means if we have a pure functional component in FBP, this is “instantiated” in a component environment. How a function could be instantiated you might ask. What are its state variables? Well, its state variables are the connections that are specific to that instantiation of that functional component model. The correspondence in a classical environment is a specific function call. This “specificity” means the calling environment that is usually saved on a stack (parameters).

And last but not least, in my view, into a FBP environment you don't have global or static variables anymore. Each variable is a component that is defined into another parent component. All connections with outer world have to pass through component's membrane (i.e. ports). It means you can “cut” the functional object (i.e. component) from its place and reuse it into another place (as a model or as an instance).

The problem with two pointers specified above comes from the fact that the “method” need “this” pointer. That means more than the method in general really uses. In FBP is like saying that a subcomponent depends on all the content of the parent component instead of its input ports only. By the way, this is another reason why you have encountered situations where you can not reuse a method from a class / object just because it is bound to the whole object through “this” pointer. This is a dependency that complicates the logic but simplifies the implementation.

Vorlath Wednesday, April 21, 2010 4:54:48 PM

Nice review of the problem. In Project V, I made certain decisions concerning this. There are three kinds of "callbacks" if you will. One kind is as you mention where you define an interface and you can substitute the implementation. This is actually the core of Project V and how it works. The default behaviour is to look on your machine and see what's the "best" possible implementation to use. Also, different machines will be able to talk to each other and share implementations if need be.

The second kind of callback is similar to the above, but where you can pass around the implementation itself. So instead of creating a connection to the inputs and outputs of a component, you would use meta-inputs and meta-outputs which sets and retrieves the implementation at runtime. This is similar to what we commonly use in contemporary programming languages where we manually select what interface implementations we use. It's like setting or retrieving a reference to the implementation.

The third kind of callback is much more powerful and much more difficult to use. In another layer of your network, you can develop a controller program. It's usually done in small parts. But this layer allows you to modify and build anything that is possible in Project V. For example, you can develop a meta-layer that will add a new connection to your ADDITION component when all inputs are filled up and generate the appropriate internal components.

That's a simple example. But you can do other things such as rewire a network at runtime. So you can indeed move components around and connect all inputs and outputs to as many different components, even if they are on other machines.

Most programming languages mix these two layers together. I think that's the wrong way to go. For example, the Actor's model is a primary culprit of this. The actor itself can produce other actors. That's wrong. Languages like Erlang work this way and IMO is a terrible, terrible flaw in the design.

More than that, anything that creates or controls how different parts of your system interacts should not be part of the main program. It should be separate. So if you're using OOP, 'new' (and delete if you have it) should never be part of your main program (if the language was designed properly). Unfortunately, OOP has no mechanism for controlling creation and deletion of objects on a higher level. No thought has gone into this and so the program itself MUST control the objects themselves.

To be clear, I'm not at all suggesting you not use 'new' and 'delete'. I'm saying it could have been better designed.

This is a problem that has already been solved in real life and see no reason to re-invent a broken wheel such as the Actor's model. In the real world, management is a hierarchy. And the workers always work directly on the end result. Not to say that management doesn't do any work. Quite the contrary. What I'm saying is that there are separate layers of computing. The top layers are valid programs just like the main layer. It's just that the top layers are management of the lower layers. It works well in the real world and I see absolutely no reason at all why I should counter thousands of years of evidence and knowledge.

Think of it this way. As a programmer, you are on a different level than the code. You enter the code and the compiler is the tool that is on a higher level that can construct the objects and functions that you entered. But you lose this functionality at runtime. Even if you have things such as eval(), it's not the same. The higher level is gone. eval() or any such construct is WITHIN the same code as everything else. You can no longer have a view of everything and make changes the way you could at the development/compiling stage.

Project V changes all that. All the power you have at the development stage will be available to you at runtime (and I'm not talking only about the ability to compile programs either which is almost besides the point). As I said, having a separate layer is a core principle that has existed for thousands of years whenever you need to produce something requiring large amounts of resources. Think about it. Even Java has a separate layer for garbage collection. That's this principle at work. It is separate. Think of JIT. Again the same principle. Unfortunately, the rest (like bytecode and the language itself) is contrary to this principle.

Vorlath Wednesday, April 21, 2010 5:12:41 PM

I just had a cool idea. In the beginning, I have no choice but to have a built-in runtime engine. But I like the idea of the runtime engine being one of the higher level layers. The higher levels would actually be directly responsible for the "execution" of the lower levels. So eventually, I will rewrite the runtime engine within Project V itself. This way, implementation selection, execution priority, etc. will all be customizable at design time (and runtime if need be). The very top layer won't be customizable at runtime though because only lower layers can be dynamically updated. I may revise this later on.

Unregistered user Thursday, April 29, 2010 7:12:35 PM

Dan writes: "So sequential programming still has a place." Oh, I wholeheartedly agree. For me its all about the freedom to choose whichever abstractions are best suited for the job at hand, be it OOP, streams, trees, sequential, dataflow.. whatever. Some problems are inherently well suited to a dataflow model, some problems are most easily and naturally modelled as objects, yet others are inherently sequential - a language shouldn't limit you by forcing you to use a small subset of possible abstractions, instead it should provide you with the tools necessary to implement whatever abstractions you may need. So, what I want is a Lisp-like language, with homoiconic syntax and a powerful macro system, which can function in both true-dataflow (by which I mean that it parallelizes the execution, instead of how most existing dataflow-esque libraries provide a limited set of dataflow-like constructs which actually execute sequentially) and sequential styles, perhaps preferring dataflow as the default, built in immutable data structures (Clojures sequences are a great start: lists, vectors, maps, sets, regexes...) and a standard library which uses macros to implement other popular abstractions: constraint programming, OOP, functional constructs and so forth.

Dan Pologeadpologea Thursday, April 29, 2010 8:45:54 PM

Originally posted by anonymous:

For me its all about the freedom to choose whichever abstractions are best suited for the job at hand, be it OOP, streams, trees, sequential, dataflow.. whatever. Some problems are inherently well suited to a dataflow model, some problems are most easily and naturally modeled as objects, yet others are inherently sequential - a language shouldn't limit you by forcing you to use a small subset of possible abstractions, instead it should provide you with the tools necessary to implement whatever abstractions you may need.


I am hearing this quite often. My fellow programmers did say the same thing: "Well, I don't care what library or language to use as long as I get the job done". The problem was just that the job got done ALWAYS after the deadline and process was painful. It was so hard to demonstrate that there were simpler ways to do it but radically different. It seems this "radically different" is very disturbing. If you look carefully to all concepts spread in the programming world you will realize that most of them are bullshit and what is good takes into account physical laws that nobody can escape.
Personally I have noticed that when all this "knowledge" is getting larger my believes are oriented toward the same set of intertwined concepts: "Matter, Energy, Laws" (the Holy Trinity). Each one of these aspects has a correspondence in software. The software is just a simulation of the real world and it has to obey to the physical laws.
My conclusion is: there is a tendency, if you are really good you are good to do more other things, if you are really bad you are bad to all. If you have a concept in programming that is really good than it has the tendency to be good in many other programming subdomains. I am not saying there is a language that fits all but there are for sure powerful principles that fits all. Among them is also DATA FLOW.
Remember the holy trinity: Matter, Energy, Laws. You can make a kind of association: Matter -> Data, Energy -> Hardware, Laws -> Functions (restrictions). A software component is a mixture of Matter (data) and Laws (functions) but it lacks the energy that pushes the matter through the sieve (laws). The energy is given from the real physical component that is the hardware (microprocessor). But the hardware is more than that. It represent also data and laws. The software is just it's extension but it lacks a very important aspect: the energy. If the data could move "by itself" trough the pipes it would be great but it can't. The matter moves "by itself". In fact it is about forces, pressure, energy distribution or whatever else. In software we need hardware for this.
To conclude: Flow Based Programming seems to take into account the natural physical laws. It starts from the knowledge that software is a simulation of the real physical world and takes it into account instead of fight it against.

Vorlath Thursday, April 29, 2010 11:40:01 PM

From Anonymous:

So, what I want is a Lisp-like language, with homoiconic syntax and a powerful macro system, which can function in both true-dataflow (by which I mean that it parallelizes the execution, instead of how most existing dataflow-esque libraries provide a limited set of dataflow-like constructs which actually execute sequentially) and sequential styles, perhaps preferring dataflow as the default, built in immutable data structures (Clojures sequences are a great start: lists, vectors, maps, sets, regexes...) and a standard library which uses macros to implement other popular abstractions: constraint programming, OOP, functional constructs and so forth.



I'm not certain this is the best way to go. C++ is very imperative and yet people are saying it's trying to do too much. That it tries to be everything to everyone. I think you need to compartmentalize and stick with one paradigm within each compartment. And make sure that the interactions between compartments are fluid and dynamic. This is harder than it seems and is where most of the problem come from as Dan correctly points out.

I do agree with this:

instead of how most existing dataflow-esque libraries provide a limited set of dataflow-like constructs which actually execute sequentially



If you limit a dataflow network to be linear (the outputs of a component can only link to a single following component) and you limit the network to one single data packet, then yes indeed, you will simulate imperative sequential programming.

Write a comment

New comments have been disabled for this post.

June 2012
S M T W T F S
May 2012July 2012
1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30