RE: Re[2]: [boost] Boost.Thread : IO multiplexing

From: Darryl Green (Darryl.Green_at_[hidden])
Date: 2004-01-11 23:49:16

> From: Roland [mailto:roland.schwarz_at_[hidden]] 
>
> (Darryl Green <Darryl.Green_at_[hidden]>) wrote:
> > Specifically, I don't think that boost.thread condvars should be
> > extended to deal with IO any more than I think pthreads 
> condvars should
> > be. I do think that portable IO multiplexing would be great and
> > shouldn't be in a socket specific lib. However it looks 
> like the latest
> > proposal for the boost.socket lib should be generic enough 
> (maybe needs
> > a name change?).
> 
> Thank you for the pointer. Btw.: where can I find it? Is this 
> in the sandbox cvs?
> 

Sorry - too many libs to pick from - Hugo Duncan's net031107.tar.gz from
the files section was the last one I looked at. It is definitely a work
in progress but it appears to be very generic - it can use select or
other reactor concepts, sockets or named pipes etc. I'm not sure where
it is up to at present.

> 
> I have come to this opinion because cond.wait() and WFMO or select are
> the two mechanisms where one deliberately puts a thread into 
> sleep. (These
> are also automatic cancelation points in pthread.) So when I 
> need to wait
> for either one of a condition beeing notified or an IO event 
> occuring there
> is no way of doing this efficiently (altough the operating 
> systems would allow
> for this). I can imagine a helper thread which converts  IO 
> events to notifies
> or notifies to events that can be recognized by the IO 
> multiplexer. But this
> gets (unnecessary?) complex, and shurely inefficient. Also this thread
> likely will be a bottleneck for throughput. How many helper 
> threads should be
> there? The other solutions unfortunately need to alter some 
> code within
> the library _or_ disclose some of the internal structs. I 
> think the latter is
> to be avoided at all circumstances, while adding a defined primitive
> that gives the desired access would be preferable.

I agree that there is some appeal in a unified interface for various
"event" or "condition" types.
I also agree that inefficiency must be avoided if the library is to be
genuinely useful.
I don't know of a portable way of unifying IO notification (be it
select/poll style or async io) and condition variables that is efficient
- it seems to me to be a kernel dependent issue that can't be addressed
purely in library code.

By providing only a restricted set of primitives that is portable, the
developer will have to consider how to deal with synchronisation issues
in a way that is efficient and portable - this may not be possible and
platform specific code will need to be written. Its unfortunate, but I
don't see how a lib can help with this and remain portable.

> 
> 
> > Any IO multiplexing system should have some way to allow 
> non-IO events
> > to wake it up.
[snip]
> Thank you for pointing this out, but I already was aware of 
> this option. What
> I dislike on this is the inhomogenous nature of signalling. 
> Say you have
> a thread that sits waiting for IO, and then you want to send 
> it a command.
> Say it is waiting for IO inside an object that is made thread 
> safe using
> the monitor pattern. Now you want to send a command. The command
> functions are secured by the monitor. How do you secure the function
> that is waiting for IO? You would need to manually unlock and lock
> the monitor somehow, while still not beeing able to honour a general
> notify that would be able to force a state stange of the 
> object irrespective
> how many thread are waiting on it in any combination of wait types.

You want to block on IO ready inside an object protected by the monitor
pattern? Why/how would you ever find yourself doing that? You mean that
you want a "threadsafe" object with an interface protect by a monitor,
but somehow you end up invoking a reactor within it? I don't think I
understand your example.

> 
> I must admit that I am a little biased now by the fact that I 
> was able to
> write a small application that contains two classes that are derived
> from thread and condition namely cancelable_thread and 
> cancelable_condition
> that have been possible simply by _using_ boost.thread. For me this
> was a strong indication of how complete the boost.thread primitives
> already are. I wouldn't dare to tamper with them at all!
> 
> But, my classes still suffer from the problem that I cannot 
> sucessfully
> cancel when a thread is inside the IO wait. Sure cancelling a 
> thread can
> be tackled by other means (e.g like pthread) but cancelling 
> an object on
> which you have a mix of different threads waiting?

I'm not sure I follow just what you mean by "cancelling an object...".
It sounds to me like you have some conceptual framework in mind for
active objects and you are having trouble implementing it using the
existing thread library when the object is able to do IO? I don't have a
good enough picture of your model to comment.

> 
> Recently I skimmed over a tutorial of the ACE library. They 
> also point out
> that they have unified these two types of wait.

Really? On all platforms? Not last time I used it. As I recall it does
have some level of support for waking up a thread blocked in a reactor.
As I recall the posix implementation used a pipe in the select set to do
this. But maybe you/the tutorial was using a proactor? Or something
else? ACE is kinda big, and doesn't (didn't?) always work (completely)
on quite as many platforms as one might hope - quite prossibly because
the platform lacks the needed support, it restricted to particular
"event" types or it is buggy.

> 
> But please, don't get me wrong. I am not in a position to 
> criticize the
> boost.threads at all. I want to learn how to use it 
> efficiently and solve
> some basic problems with it.
> Perhaps you could tell me how you would tackle the following problem:
> 
> You have an object that implements the doubly bound buffer as to be
> seen in one of the examples. You have an IO port (tty, 
> sockets, parallel, ....)
> Now you want a thread that multiplexes on the buffer or the IO.
> 
> I would like to expose the condition that signals data vailability
> and then beeing able to wait on either this condition or the IO.
> I would be glad if you could give me a sketch of how this could be
> done with boost.thread  (efficiently) without altering it.

I'd refactor :-) I would have to do something different to run it on
most systems (ie not windows - no wfmo)? What exactly I would do would
be a design decision based on more than can be conveyed in a simple
example.

> 
> > > Unfortunately I do not see a way, how this could be done without
> > > altering the code of condition.hpp so before giving it a try 
> > > and start hacking
> > > away I would like to hear some opinions. Perhaps there still 
> > > is a way to emulate
> > > this on top of boost.thread somehow?
> > 
> > Not on top - along side. I must admit I haven't looked 
> closely at your
> > use case. Maybe if I have missed something you can point it out.
> > 
> 
> Along side implies, in my reading, that the problems are orthogonal.
Ideally yes.

> I hope I was able to point out that they are almost, but not 
> completely orthogonal.

I appreciate and agree with your point of view here.

> I try to be more precize: Altough the operating system makes no
> difference on threads whether they are waiting for IO to complete
> or they are waiting on a synchronization primitive (condvar) the
> application program has to deal with these cases in a fundamental
> different manner. (And I think this makes things unnecessary 
> complicated.)

This is where we disagree. It depends on the OS - some at least do treat
these cases quite differently.

> Put it yet another way: Completing an IO request causes some kernel
> module to issue a signal. In an ideal world this would be a condvar
> beeing notified, wouldn't it?

In some world (not sure if it really would be ideal) it would be.
Unfortunately we aren't in that world.

My point is simply that I don't believe there is an efficient way of
unifying this in a library while remaining sufficiently portable. I'd be
happy to be proved wrong! My suggestion is that a very basic/minimal
"wake up" facility is

i) portable
ii) sufficient for most uses

The wake-up facility can be implemented on top of various select-like
interface and of course wfmo.

If you are interested in how various forms of io multiplexing work (or
don't) on a variety of platforms a good place to start is
http://www.kegel.com/c10k.html.

Efficiency, portablity, elegance - pick any 2 (if you are lucky) :-)

Regards
Darryl Green.

##########################################################################
This e-mail is for the use of the intended recipient(s) only.  If you have 
received this e-mail in error, please notify the sender immediately and 
then delete it.  If you are not the intended recipient, you must not use, 
disclose or distribute this e-mail without the author's prior permission.  
We have taken precautions to minimise the risk of transmitting software 
viruses, but we advise you to carry out your own virus checks on any 
attachment to this message.  We cannot accept liability for any loss or 
damage caused by software viruses.

##########################################################################