[boost] Secret handles and descriptors: Enemies of reuse,

From: Simon Meiklejohn (Simon.Meiklejohn_at_[hidden])
Date: 2004-09-13 16:36:24

>From my perspective developing interactive telephony applications this
problem is an example of a more general one. Heres the outline of a
solution that I've been using

In general a program may need to integrate a variety of libraries which
are sources of information/events. Often it's the nature of these
libraries that they either own a thread which calls back into
application code, or require the user to call a potentially blocking
function that returns the data/event.  Then again there are the
libraries which explicity make available o/s objects that can be waited
on.

Its frequently the case that no single 'raw' demultiplexor system can
interface to all of these at once. (I guess that's the reason for this
thread). To me the underlying issue is one of threading. 

Heres a list of typical issues that arise in my apps.

- my networking library wants to call me back in its own thread 
- ditto for my multi-media timer
- my telephony library wants to signal me by setting an event - so I'd
better dedicate a thread to it. That thread will also call me back.
- my application objects are having events fired at them from all sides
from multiple threads and therefore need to use Mutex's, but events from
below (telephony) keep on deadlocking with events from above (network).
- my COM interface wants to issue connection point callbacks to me in a
the app main thread via the app message pump.
- I cant call out on my COM interface on my Network library's callback
thread because its not marshaled for that kind of stuff.
- So I put the network messages in a queue - now I have to send a
windows message to the main thread, where its ok to call COM. This isn't
necessary on my unix version, so I have ugly platform specific code in
the middle of my app.
- My smart network library parses my messages into complex data types
before I get to see them. So getting them out of the queue in the main
thread involves some kind of dispatch on type. This is expensive and
hard to maintain.

STEPPING BACK A BIT
While it may seem that many of my problems stem from COM, I think it
stems from an unfortunate need to implement the application's threading
policy at all levels of the code - library and application code. 

Also, the hoops I have to jump through to get COM working wont be at all
applicable if I shift to unix or move to a non-COM interface. What I
really need is the ability to define a threading policy at some high
level point in my code (e.g. main), have library code that defers to
that policy. I've called this a Thread Redirector policy

Looking at the Network lib to COM path as an example:
- the network lib receives bytes from the network in its own thread.
When it has a complete message (but before it starts decoding the
message) it calls the Thread Redirector asking for a callback in the
appropriate thread.
- in main() we've set things up so that when Networking asks for a
callback it will be called back in the main thread via a windows
message. (On Unix I substitute a pthreads based callback.)
- On windows the callback occurs in the main thread. The Network library
starts decoding the message. It determines it's a message type XXX and
calls back into the App code MyAppObj.DeliverMessageXXX( MsgXXXType& XXX
).
- the app can now call into its COM API  like - myCOMIF->DoSomething(
XXX.param) - without first queueing the message.

I can define it so that all my libs deliver their events to the
application objects on the same thread. This relieves the thread
deadlock issues.

I can go into more detail if required - but that's the flavour of it.

Simon Meiklejohn