Re: [boost] Multi-processing - Boost future libraries?

From: David Abrahams (dave_at_[hidden])
Date: 2005-08-18 10:01:51

Ion Gaztañaga <ion_g_m_at_[hidden]> writes:

>> 1) Boost Thread library work seems to have slowed considerably and yet now 
>> more than ever this is one area where C++ is sadly lacking. Recent 

Yes.

> It's a shame since there were some comments about reworking some 
> components like rw_locks. There was also a discussion about the interface.

Yes.

>> commentaries such as Kevlin's on this list have not necessarily
>> flattered the Boost thread design, but do we have a champion(s) who
>> are/could help to address this, or are alternative efforts
>> happening instead behind the scenes in C++ working groups?

I'm not an expert, but I think such experts as Alexander Terekhov will
be pretty quick to stand up and tell you that something at about the
same level of abstraction as Boost's design needs to be available.
>
> I find Kevlin's "More C++ Threading" paper is really good. Maybe we 
> should adapt Boost.Thread's implementation to this interface and check
> if this is better than current one. But it's a quite big task.
>
>> I've seen lots of excellent advise from Peter Dimov, Dave Abrahams and many 
>> others but is it the case that its just seen as too thorny a problem for 
>> individual's to commit to, or is Boost thread considered sufficient?
>
> I'm not sure but some people have said they don't see boost-thread 
> interface as the best interface. But I don't know how many think that. I 
> personally prefer the "futures" approach described in Kevlin's paper.

I haven't seen Kevlin's paper.  All I can find is
http://www.two-sdg.demon.co.uk/curbralan/papers/accu/MoreC++Threading.pdf.
IIUC, futures are a very useful high-level abstraction but you can't
solve all the same problems with them as you can with the facilities
of Boost.Threads.  'Course, I could be wrong, but that paper doesn't
mention condition variables anywhere so I can't tell.

That is to say, IIUC, futures are a good idea *in addition to* having
some lower-level primitives.

>> 2) Is Shmem being considered for the full-scale Boost library treatment as 
>> it appears to have a great deal of useful functionality to support existing 
>> OS threading and shared memory techniques.
>
> Well, I'm developing Shmem and I'm a biased, but I do think that C++ 
> world needs some serious inter-process library apart from threads. 

Yes.  Especially considering that threads and processes are, um,
different ;) However, it would be very useful to have some common
abstractions that allow one to move between the two with little or no
pain.

> The fact is that Shmem tries to offer a framework to put object in
> shared memory, including STL container, but for this, I needed to
> develop basic operating system wrappers. Not being a OS expert
> myself, I've took as much as I could from Boost.Thread interface
> (locks, conditions...).
>
> I have no idea about memory models so the current discussion for C++0x 
> is somewhat far from my knowledge, 

It sounds more complicated than it actually is.  It's simply that when
you take the language used in the standard to describe what a program
sees when it reads and writes memory, and you toss in the idea of
concurrency on top of it, it's impossible to describe the behavior of
synchronization primitives so that they give us the results we expect
from them.

The formal description of how a program sees its memory is very loose
-- just tight enough to work for single threaded programs, but not
forced to be enough like the memory model of any particular real
system supporting multithreading to make it possible to specify
synchronization.  The challenge is to tighten it up just enough
without overspecifying it so that we rule out existing MT systems or
force avoidable inefficiencies.

> but in boost, we could try to define some things about the
> interface:
>
> -> Is boost threads interface sufficient or do we prefer kevlin's 
> approach? We can try to develop the second interface based on
> existing Boost Thread's code.
>
> -> We must define the synchronization objects. If we like boost thread's 
> approach, that's fine, but I would suggest that the use of lockers (like 
> scoped_lock) should be optional and not mandatory.

?? How would you like to lock a mutex, then?

> -> The same for thread local storage.

I'm pretty sure nothing in Boost.Threads mandates the *use* of TLS.

> Apart from this, I would define more concepts:
>
> -> A semaphore: Some say it's dangerous, but sometimes, a semaphore is 
> tha only way to notify events from drivers and interruptions. 
> For me, a semaphore is a must.

Is it possible to abstract away any of the "unsafety" without losing
applicability to the crucial use cases?

> -> Define read-write locks interface definitively. Some have expressed 
> boost thread's approach is inefficient. Same with barriers.

Good plan.

> -> Define a message queue. A message queue should be a must for a 
> serious C++ standard thread library. Should it use posix-like 
> priorities? System V approach?

> -> We can even define a pipe, that can be used in a iostream-like 
> interface to pass data between threads/processes.

Nifty.
>
> I would then adapt all Shmem stuff to the defined thread interface, so 
> we can use the same lockers with thread and process-wide mutexes. The 
> same with the inter-process message-queue. I would also separate 
> Shmem's low-level wrappers to a ipc library, so that Shmem just uses 
> them to offer STL-like shared memory containers, allocators and some 
> framework to create named objects. This way, we could split Shmem and 
> make reviews shorter.
>
> Obviously, I would love to have also a unix socket-like ipc, but it 
> would be a serious development to emulate it in windows without using 
> TCP sockets.
>
> I appreciate a lot boost libraries, but sometimes I think the libraries 
> are "expert level ideas", and that C++ is lacking basic blocks like 
> network, threads, processes, data-bases, and a GUI. 

We're all aware of the issue; the question is, who's going to do
something about it? ;-)

> Not that I have much time, but I'm ready to help a bit in
> thread/process ideas.

What we really need is someone/people to step up to the plate and take
responsibility for development, maintenance, and documentation of
relevant libraries.  We need more than ideas; we need leadership.
That isn't to say it has to be done by one person alone, but at least
someone needs to rally the energy of Boosters with the interest and
expertise to tackle the problems.  Maybe you and Paul can do something
together?

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com