Re: [boost] Re: Mutex & lock unification [was: Boost.Threads: Do we need all those mutexes?]

From: Matt Hurd (matt.hurd_at_[hidden])
Date: 2004-07-17 02:26:21

On Fri, 16 Jul 2004 23:38:43 -0400, Michael Glassford
<glassfordm_at_[hidden]> wrote:
> Matt Hurd wrote:
> 
> > On Fri, 16 Jul 2004 13:54:32 -0400, Michael Glassford
> > <glassfordm_at_[hidden]> wrote:
> >
> >>In an attempt to reconcile the different preferences and ideas expressed
> >>in this thread, here's a thought experiment for mutex & lock
> >>unification. What if the mutex and lock classes were policy-based,
> >>allowing users to choose the interface they want and supplying a typedef
> >>or two for the "most common" or "approved" variations?
> >>
> >>Example below (note that I haven't tried to implement multiple policies
> >>elegantly). Hopefully it could be expanded to include recursive,
> >>checked, read/write, etc.
> >>
> >
> >
> > <snip>
> >
> >>//Example typedefs:
> >>
> >>typedef mutex_type<
> >>    blocking_mutex_policy,
> >>    try_mutex_policy,
> >>    fast_timed_mutex_policy
> >>    >
> >>    mutex;
> >>
> >>typedef blocking_mutex_type<
> >>    blocking_mutex_policy,
> >>    null_policy,
> >>    null_policy
> >>    >
> >>    mutex;
> >>
> >>typedef try_mutex_type<
> >>    try_mutex_policy,
> >>    null_policy,
> >>    null_policy,
> >>    >
> >>    mutex;
> >>
> >>typedef timed_mutex_type<
> >>    timed_mutex_policy,
> >>    null_policy,
> >>    null_policy,
> >>    >
> >>    mutex;
> >>
> >
> > <snip>
> >
> >>//Example typedefs:
> >>
> >>typedef lock_type<
> >>    blocking_lock_policy,
> >>    try_lock_policy,
> >>    timed_lock_policy
> >>    >
> >>    lock;
> >>
> >>typedef lock_type<
> >>    blocking_lock_policy,
> >>    null_policy,
> >>    null_policy
> >>    >
> >>    blocking_lock;
> >>
> >>typedef lock_type<
> >>    try_lock_policy,
> >>    null_policy,
> >>    null_policy>
> >>    try_lock;
> >>
> >>typedef lock_type<
> >>    timed_lock_policy,
> >>    null_policy,
> >>    null_policy
> >>    >
> >>    timed_lock;
> >>
> >
> >
> > I think this is thinking along the right track though I'm not fond of
> > your interface above, but it is a good start to the thinking.
> 
> Do you mean having three template parameters? I mentioned that that was
> just for the sake of example. If you mean something else, I'm not sure what.

Not sure about the way you've done the template parameters.  I haven't
given it enough thought to comment much, I'll try to find some time in
the next couple of daze to give it some diligence.
 
> > I've been reading over some of the old boost thread threads and I
> > think I now see why Bill Kempf is now away from boost::thread.  The
> > best spin is, I think, he gave up on consensus.
> 
> This occurred to me during this massive threading thread. It seems that
> no two people agree on what a threading library should look like (e.g.
> some people want one mutex class and one lock class; others want three
> mutex classes and three lock classes with no overlap; others want three
> of each, with each being a superset of the last; others like things the
> way they are now; etc.). I thought perhaps that well designed
> policy-based classes could give most people what they want without
> making everyone else have it too.

Yes, I think it is going to be the only way to provide basic locking
functionality that is not easily abused and the full low level access
that others strive for.

There are a bunch of clever people with strong views on needing
different things and a policy based approach maybe the only way to
deliver this.  Basic typedefs for scoped locks and the standard
mutexes will do a 95% job I'd think.

> > It it interesting the old ground we are covering, some of the thinking
> > back then was perhaps a little better then.  Some of the current ideas
> > seem a little better too.
> 
> Yes, many of the same discussions seem to get rehashed, don't they? Much
> like they do with smart pointers, oddly enough.

Sometimes this is good, I might understand what is going on the fourth
time around ;-)
Documents and wiki's are often better than newsgroups at this as the
sense of history is better.

> > I do like David Held's thinking, perhaps because it is similar to mine
> > ;-), in these mails:
> >     http://aspn.activestate.com/ASPN/Mail/Message/boost/1213138
> >     http://aspn.activestate.com/ASPN/Mail/Message/boost/1207304
> >
> > Bill Kempf was concerned about providing as safe as possible semantics
> > for locking.
> > He saw the need for lock transfers especially for Bjarne's auto
> > locking proxy idiom.
> > One way of doing this was to expose lock and unlock on a lock, but he
> > felt exposing the same on the mutex was safer
> 
> I read things the opposite way. At the very beginning he wanted no lock
> an unlock methods on the lock, but changed his mind or at least conceded
> the argument; however, he thought (and I agree) that lock and unlock
> methods on the mutex were dangerous. Did you mis-type, or could you give
> references?

I think we are agreeing in a round about way... I'm not the clearest typer.
Yes, he does concede the need for the equivalent of lock and unlock
but saw the transfer of ownership as the better way rather than
explicit lock and unlock as you indicate.  I think.

He covered the mutex locking thing here, saying he considered it but
thought it was a bad idea.

http://aspn.activestate.com/ASPN/Mail/Message/boost/1207226 
<quote>
It's the second problem that's more difficult.  I originally thought I'd =
solve this solution by creating some sort of lock_operations<>  template =
that would expose a Mutex's lock operations externally, thus reducing =
the likelihood that a user would abuse the use of the lock operations =
solely because they were readily available in the Mutex's interface.  =
However, this isn't really a great solution.  For one thing it just =
feels like a hack, and the end result is still an interface that's =
easily misused.
</quote>

>  >  yet still troubling.
> > Bill related transfer semantics to auto_ptr and saw the shared_ptr
> > approach as problematic.
> >
> > Others are of the opinion that lock and unlock is fine and catering
> > threading newbies should not prevent them from the low level tools
> > that makes their life fun ;-)
> >
> > I side with Bill, on balance, I think on this.  I see public a lock
> > and unlock as the equivalent of a threading "goto".
> >
> > However, I'm not so sure about his shared pointer reluctance at one
> > level.  His argument was that a shared_pointer interface requirements
> > are a lot different to an interface for a lock with move/transfer
> > semantics, which is true.  He also argued that sharing a lock made no
> > sense and was dangerous.
> 
> I agree that sharing a lock makes no sense; explicit transfer (not
> sharing) of a locks seems to make sense in some circumstances, though.
> 
> I think Bill came to the conclusion that transferable locks were
> something he should do, and that most/all of the people participating in
> the discussion agreed.

Yes.  However, thinking about what a read lock is, it is really just a
reference counted lock in some respects...  In this light is it not
just the same as sharing a lock?

Kind of anyway, it is really sharing the ownership of the mutex, which
has the colloquial form "lock sharing".

>  > This is kind of true, but we do share a
> > mutex with a shareable mutex (the one i keep calling a shareable mutex
> > and the world calls a read write mutex).
> >
> > A non-shareable mutex in effect would have a count of zero or one
> > (true or false) and ownership transfer would be the correct interface.
> >  Careful consideration must go to thinking about such situations as :
> >     do_something( lock_object );
> > w.r.t. to transfer.  Perhaps
> >     lock_object = do_something( lock_object );
> > covers the case, but I can see the danger already...
> >
> > There are grounds for concern over the implementation delivery of
> > shared / transfer implementation as it should either rely on
> > no-locking, atomic primitives or locking primitives below the normal
> > locking to avoid the chicken and the egg of locking a count.
> 
> I'm not sure I understand what you're getting at. To make sure we're
> talking about the same thing:
> 
> 1) I'm not talking about transfer of locks between threads.

Agreed, but with the read/shared lock exception to the rule...

This is a bad idea.  The case I gave previously still stands where is
stuffs up the thinking with respect to sharing a mutex with multiple
objects.  Which is at least one example where  transferring locks
across threads can cause grief if allowed.  This could be overcome by
a policy that takes care of the mutex / thread_id housekeeping to make
this work.

However, semantically a read lock is sharing a lock amongst many
threads which confuses the issue.

> 2) I am talking about transferring locks from one scope to another scope
> (e.g. out of a function) within the context of a single thread.

Yes, this should be done with the transfer / move semantics.

> 3) This is essentially move construction or move assignment (that is,
> the internal state of one lock--e.g. the mutex reference and lock
> state--is copied into another lock, then the first lock clears its state
> so that it doesn't try to unlock the mutex or whatever). It's a transfer
> of ownership.

Agreed.

> 4) I'm not talking about transfer = lock promotion or demotion. Though
> the syntax is the same, it's a different operation (ownership is
> transferred, but the lock type is changed as well). That's not to say
> that I disagree with that syntax for promotion or demotion, just that
> that's not what I mean when I talk about lock transfer.

Agreed.

Yes, I can see the case for an upgradeable read lock but the
circumstance strikes me as unusually rare and I would like to see it
as a different mutex and not pollute a normal shareable (rw) mutex.
 
> > I would like to see David Held's RIIA mechanism, mixed with Michael's
> > policy approach to deliver a solution that could also be consistent
> > with the resource acquisition requirements of socket and file i/o.
> > Timer interaction, especially because or try_lock, where the timer is
> > a resource should perhaps be covered too.
> >
> > The mutex taxonomy I have in mind would be:
> >     1.  null_mutex
> >             elided ops, atomic ops with interlocking
> 
> This is a good idea.

Uh oh, I mistyped that in my rush to go riding with my kids and you
are agreeing with me....

I think the null mutex should have inc, dec, exchange and whatever as
static non-interlocked ops and the other mutexes should have them as
static interlocked ops.
 
That keeps a null mutex to the single threaded case.

Perhaps interlocked ops should live in a multi_thread:: or
single_threaded:: struct and the appropriate mutexes should just
inherit these interfaces.

That way seems neater as you can still reference the
multi_threaded::add(...) and the Mutex::add(...) for the appropriate
context sensitive operations.

> >     2.  simple_mutex
> >             basic os mutex / critical section
> >             recursive on windows, non-recursive on posix
> >     3.  recursive_mutex
> >             recursive always
> >     4.  shareable_mutex
> >             supports exclusive and shared access
> >             reflects os primitives where available
> >             ( what does this mean with recursiveness???)
> >     5.  upgradeable_shareable_mutex
> >             adds upgradeable read lock equivalent
> >
> > yeah, maybe not the best names
> >
> > - a lock is a resource acquisition on a mutex.
> > - avoid lock and unlock methods if possible via move/transfer semantics
> > - consider a read lock as a shared_lock, lots of owners.
> >
> > run out of time, gotta go bike riding with the kiddies...
> >
> > $AUD 0.02
> >
> > Matt Hurd.
> 
> Mike

I'll have a look at your interface and put some of my own thoughts in
a code form in the next few daze.

Nice chatting, regards,

Matt Hurd.