Re: Callbacks Again

From: William Kempf (sirwillard_at_[hidden])
Date: 2000-11-20 10:14:32

--- In boost_at_[hidden], Karl Nelson <kenelson_at_e...> wrote:
> > > 
> > > Okay 3 issues here...
> > > 
> > > 1) Should a library have a callback and later build slots?
> > > I have had that arguement a number of times and what always 
gets 
> > pointed
> > > out is that building just callbacks causes some many problems
> > > it isn't worth it.  Libsigc++ provided the class Callback# which
> > > was exactly that.  It was a union of pointers, function 
pointers,
> > > methods with objects etc.  However, as I discovered you
> > > would have to build your own set of factories for those 
callbacks
> > > in effect doubling the size of the library.
> > 
> > That's what libraries like Lambda are for.
> 
> Well, the only difference between a Callback and a slot is
> the Slot can track both forward and reverse dependencies.
> 
> Forward means when the sender doesn't need the callback the
> callback gets destroyed, if chaining is allowed then adaptors
> are possible.  SigC supports cahining which is I can build
> a slot and then make another slot point to it and so forth.

Valuable implementation details, though it makes for some interesting 
overhead for uses like that needed for thread callbacks (in case you 
don't understand what I mean here, as *might* have been indicated by 
something you said in another post, the thread callback is what is 
invoked as the "work" of the thread when it's created).  I'd need to 
evaluate how much overhead exists before really commenting here.

> Reverse dependencies includes the ablity for the target to
> force the clean up.  That is if the target falls out of scope 
> it takes out the chain. 
> 
> Example.
> 
>    Slot0<void> s;
>      {
>        Foo f;
>        s=slot(f,&Foo::some_function);
>        s(); // calls f.some_function();
>      }
>    s();  // calls nothing
> 
> This is the only functionality provided by a slot beyond
> a callback.  

I'm not sure how you achieve this magic.  Unless Foo derives from 
some base class (not acceptable for generalized callbacks) I'm not 
sure how you'd know when it goes out of scope.  I definately need to 
go through libsigc++ with a fine tooth comb.  I'm in danger here of 
arguing with you about which concept is more appropriate with out 
having a clear idea about the particulars.
 
> > >  Second, you can't
> > > really build adaptors with the callbacks because they would
> > > require chaining of resources.   In other words, slots
> > > in the clean up capablity allow cleaning up the objects which 
> > > allow adaptors.  So Callbacks are of limited use.
> > 
> > I *really* don't follow this.  Care to post some specific 
examples of 
> > what you mean?
> 
> SigC provides for adaptors which in turn form cascades...
> 
>   int foo(int, float);
>   Slot0<void> s;
>   s=slot(rettype<void>(bind(slot(&foo),1,1.5)));
>   s();  // calls foo(1,1.5);
> 
> Also note the chaining adds some complexity to the 
> cleanup.  In other words if you get both a chain and 
> a target removal it can get quite messy.  

The Lambda library already allows for the "cascades" you show in the 
example above.  The only thing I'm unsure of is the references you 
make to "cleanup".

> > >  Further,
> > > they aren't all that useful to build the Slots themselves as
> > > the slot just ends up carrying arround the bulk of
> > > a general callback where it could be trimmed down.
> > 
> > I'm not suggesting that Slots have to be built off of a general 
> > callback, just that it would be worth considering once done.  As 
> > for "bulk"... we're talking a virtual table, a pointer and the 
size 
> > of the functor.  A Slot must carry at least this any way.  I 
don't 
> > see problems with bulk.
> 
> Okay, we are thinking in different terms here after looking at the 
> Callback library someone submitted.  SigC does not do Functors
> like that it does something more like Closures.  SigC does not
> carry the Functor with it only the address of a target who will
> receive the call.  Thus it targets pure functions and methods 
> not function objects.

Not precisely, since it does what I know as "closures", which are 
functors that bind an object instance and a member function.  Such 
closures are "functors" in every sense of the word.

No, what I think you really mean is that "slots" are the 
only "functors" and must be built using only function pointers 
and "closures".  *If* I'm guessing correctly (still need to really 
look at the code closely) this makes it easier for you to control 
the "cleanup" magic, but it sacrifices a lot of functionality, 
especially with Lambda Library and other expression template 
libraries.  Sounds like the tradeoffs need to be carefully weighed.
 
> (Side note, SigC was designed not to carry virtual tables in
> templatized types.  Many compilers at the time in which 
> SigC developed generate a huge bulk of vtables if each 
templatization
> was a virtual.  Thus it has no virtuals in those classes but rather
> a function pointer.  Early testing found that using vtable like
> that expanded the gtk-- library by several megs in pointless 
vtables 
> typeinfo and exception info largely because the linkers didn't
> combine the info when building libraries.  Essentially, every copy
> of a template with virtuals would implement the vtable in every
> object file and thus make a huge library with many repeated 
> symbols.  Compiler tech has improved since and this may not be 
> an issue since no one is using the compiler sigc originally
> targeted gcc 2.7.2.3)

I assume you used techniques similar to those in the Hickey paper on 
callbacks?  I'm a little concerned about whether or not this "tricks" 
are standards compliant, even if they work on known compilers.
 
> > > 2) Can a Slot be a Callback?  
> > > The answer to this is yes.  That is that if I make a Slot like
> > > concept with the ablity to disconnect and clean up the 
> > > traces between objects, I don't have to use that path.  
> > > Thus it is a very common misunderstanding the to use SigC
> > > you must use SigC::Object.  You don't, you simply must use a 
> > different
> > > factory to produce pure callbacks rather than slots.  Thus
> > > nothing in SigC requires SigC::Object other than SigC internals.
> > 
> > But Signals/Slots are multi-casting concepts, which is *not* the 
same 
> > as a callback.  Or are you suggesting that the Slot by itself is 
a 
> > pure callback?  If so, the documentation in libsigc++ doesn't 
address 
> > this at all, which is a huge disservice.  Callbacks are 
frequently 
> > needed in areas in which a multi-casting Signal/Slot is either 
> > overkill or even problematic.
> 
> Signals are the multicast.  A Slot is just a generic callback with
> the target tracking.  Escentially, if you make a good callback 
library
> all you need is a class called Signal which contains a list of
> callbacks and one specialized method which unifies the calling
> of the list.   
> 
> Someone released a SigC companion library which added Event.  So
> certainly Slot is more general that multicast.

This makes it sound like we should have a Callback, which is similar 
to a Signal but with out multicast.  You connect a Slot to a Callback 
the same as you do to a Signal.
 
> > > 3) Can a Signal/Slot be used in Threads?
> > > There are issues but they aren't as bad as they first appear.  
> > > Ther real problem is that only one thread can be destroying or
> > > adding things to a list at one time.  Some of this 
> > > can be done properly with make before break and atomic flags on
> > > the lists.  Further, the reference counters have to be 
protected.
> > > Really advanced adaptors and signals are out unless the signal
> > > queue is protected internally.  But for the most part it
> > > is doable. 
> > 
> > I never suggested it wasn't ;).  The Boost thread library (which 
is 
> > far from ready for prime time) could be quite useful for insuring 
> > thread safety here.
> 
> Well, sigc is going to be switching away from providing the 
> thread capablity.  That is I intend to make the library compile
> itself twice, once with locks and once without.  Most people won't
> need locking so there is no point in placing if except in the 
> multithread environment.

It may make sense to templatize on a "mutex type" with a null_mutex 
that gets optimized away at compile time.  This allows you to mix and 
match the locking requirements within a single program, which can be 
very beneficial.

In any event, I was only suggesting the Boost Threads library as a 
means of implementing the locks in a portable way.
 
> > > Okay so you quickly think why can't I just add a list in the 
> > > the object the callback is pointed to to kill the callback and
> > > make a slot?  But that creates its own problem, who cleans up
> > > the list if the callback goes away?  Basically you end up with
> > > dependency loops, which require a unified method to handle the 
> > > internal cleanups.  Adaptors to callbacks also cause this
> > > dependency loop because the data in the adaptor can become 
stale.
> > > Thus for efficency building a really good Signal/Slot library
> > > which has the ablitity to do the event and threading is 
> > > the right approach.
> > 
> > I still have problems with the multi-cast issue.  If you can 
address 
> > that to my satisfaction I'll have a better idea of where you're 
> > coming from.
> 
> Okay here are the issues which SigC was seeking to address.
> (NOte, that you will find much of this
> in the documentation, but it has been so abstracted in the 
> code to save memory it is practically indeciperable.)
> Source is the Slot which points to adaptors and callbacks.
> 
> 1) Target clean up of a Callback
> 
>    Source  -> Callback -> Target
> 
> If  Target goes away Callback should go away and free any
> extra resouces it was holding.  Source should revert to
> uninitialized.  If Source goes away Callback must go. 

Removal if Target goes away sounds like a noble goal.  I *really* 
have to figure out how you do this, and if it can be applied to 
generalized functors.  As for removal when Source goes away... that 
seems a non-issue to me.  In the versions of callbacks we've been 
discussing a callback is a first class object.  Its lifetime is 
controlled the same as any other first class object.  In other words, 
when it's no longer being used (Source goes away) it's destroyed.  
The only thing that would prevent this would be if the callback 
object were placed on the heap and not managed by Source, but I don't 
think you can, or should, control this behavior.
 
> 2) Chaining
> 
>    Source -> Adaptor0 -> Adaptor1 -> Callback -> Target
> 
> If Target goes way, Adaptor# and Callback should be
> destroyed.  Source should return to uninitialized.
> If Source goes away, same.

This already happens with Lambda Library style expressions.
 
> 3) Multicast
> 
> A Signal is a container of Slots.  It connects to a Slot
> with a list element.
> 
>   Signal *-> SignalNode -> Callback -> Target
> 
> This has the same issue as before.  If the target goes
> away we must remove everything up to the signal node. 
> 
> When emitted the signal must trigger all the Slots
> in the callback list.  

Very important functionality for Signal, but we need a Callback in 
that case.  Multicasting would be bad for the use I have.
 
> 4) Multicast return marshalling
> 
> Given a list of Slots how do we resolve the return values.
> A class called Marshal handles this.  The Marshal can do
> anything with the returns from stacking them up in lists 
> to ignoring them completely.
> 
> peusdo C++
>   
>   template <R,P...,Marshal>
>   R Signal::emit(P...)
>     {
>       if (empty()) 
>         return Marshal::default_value;
>       Marshal m;
>       for (iterator i=begin();i!=end();++i)
>         m.marshal(i->call(p));
>       return m.value();
>     }

Also a good idea.
 
> 5) Generate readable errors.  
> 
> Okay this was the tall order.  Gtk-- is not supposed to
> be hyper advanced C++.  It was meant for people constructing 
> GUIs who used Qt.  They may be experts but more likely 
> they are a bit green.  In the design of the library, I
> wanted to isolate the errors.  Thus rather the spitting out...
> 
>   on line  (From boost/Callback)
>      s=mul_to_int();  // where mul_to_int doesn't take right 
arguments
> 
> boost/callback.hpp: In method `int 
boost::detail::functor_callback_impl<2>::base<mul_to_int,int,float,flo
at,boost::detail::Unused>::operator ()(float, float)':
> boost/callback.hpp:461:   instantiated from here
> boost/callback.hpp:256: no match for call to `(mul_to_int) (float 
&, float &)'
> callback1.cpp:13: candidates are: int mul_to_int::operator ()
(float, float, bool
> 
> 
> SigC spits out 
> 
>   on line
>     s=slot(&mul_to_int);
> 
> test.cc:8: no match for `SigC::Slot2<int,float,float> & = 
SigC::Slot3<int,float,float,bool>'
> /usr/local/include/sigc++/slot.h:489: candidates are: class 
SigC::Slot2<int,float,float> & SigC::Slot2<int,float,float>::operator 
=(const SigC::Slot2<int,float,float> &)
> 
> Likely this last one isn't an issue to you.  But to my users it
> really is important. 

I still *think* this can be resolved quite nicely using Concept 
Checking.
 
> > > Well, signal/slots are always going to require their own 
generators
> > > as to get the unregistering capablity you must form both the 
> > > bound pointer and make the dtor mechanism to clean up.
> > 
> > But such a factory could be simplified by using Lambda or other 
> > binding libraries.  If the factory needed only a simple functor 
to 
> > generate the "slot" things would be greatly simplified, no?
> 
> I don't believe so.  The Slot can not extract to target from the
> Functor, thus I am not sure how you would construct to reverse 
> tree give a functor which holds a method pointer to another 
> object.  I think we lack some level of generality, because 
> I don't see how to expand SigC to use functors as the boost/Callback
> does and at the same time I don't see how boost/Callback could
> be expended to work like Slots.

Again, this sound like your Target must derive from some base class 
(or at least follow some interface standards to be used by a template 
function).  It's a noble goal, but results in some serious tradeoffs 
that must be considered carefully.
 
> > > I won't look at libsigc++ TOO closely, because there was a lot
> > > of crap there which I quite frankly and throwing out.  
> > > 
> > > Things that are going...  Handles, Scopes, Callback#.  These
> > > were attempts to make a unified type ie a Slot was a Object
> > > etc.  These bad attempts are going away.  The reduced library is
> > > a fraction of the size.  Further I am paring out the 
> > > use of some memory elements.  Thus the cost of an object slot
> > > is going from 64 bytes to 28 and so on.  
> > > 
> > > With luck the cost of a good slot system will be almost the 
> > > same as the cost of a good callback system.
> > 
> > Any work in progress we can look at?  I realize you're time is 
> > limited, but the need for callbacks in Boost is great enough to 
> > generate some urgency on my part.  If you plan to submit the slot 
> > system any way I'd really like to not step on your toes by 
pushing 
> > for a (possibly) competing callback system.
> 
> I can send you the code I have.  It is missing multicast and
> thread locks but has most of the rest there.  Can you send me a 
> email address privately?  (The new code is well outside the level I 
> want to release for boost.)

My e-mail address should be available through the eGroups 
postings :).  Just to make it clear here, it's sirwillard_at_my-deja.com.
 
> As for whether SigC is approprate for boost or could serve in
> that function, that is fairly questionable.  THere is a fairly wide
> gap between your functors and sigc slots, though both are
> clearly attempts at the same general class of problem.

General class only.  I'm beginning to see very different goals 
between the two beyond the obvious (multicast and non-multicast).  
The obvious goals of multicast and non-multicast I think are both 
appropriate for Boost, and would go so far as to say that both are 
needed.  The trick now becomes addressing the subtler differences.

Bill Kempf