Re: Threads: Other Primitives

From: danl_miller (danl_miller_at_[hidden])
Date: 2002-03-14 11:32:38

--- In boost_at_y..., "dmoore99atwork" <dmoore_at_a...> wrote:
> This makes sense to me, except that when we are talking about a 
> concrete realization of a semaphore which is incapable of tracking 
> more than "N" units of work, once the N+1 is produced, you have a 
> situation where there are N+1 units of work existing, presumably 
> consuming some type of resources such as memory, etc.

  If a unit of work overflows the message-queue, the message-queue
does not increment its semaphore.

> The downstream processing can only possibly know about "N" of them, 
> because they are relying on the semaphore to represent the count of 
> work items.

  Yes, that is the lossy semantics as intended.  The would-have-been
(n+1)th unit of work in the message-queue is forever silently lost
(unless the producer resubmits a duplicate at a later time),
unbeknownst to the downstream processing.

> So, as you write, the burden of rememberance falls to the producer
of 
> the units of work.  The ability to resubmit and duplicates being ok 
> is fine, but once we reach "N+1" units of work, I can't see how the 
> system ever recovers in this case, because even if every consumer
was 
> waiting on the semaphore with a count of 0, there would still be
that 
> extra 1 unit of work bouncing around in a producer, who is waiting 
> for a completion notification that's not coming....

  The producer whose idempotent units of work must be resubmitted if
they are lost retains a rememberance of those units of work until
their successful completion is somehow acknowledged (e.g., a message
is sent to the producing thread explicitly stating successful
completion of
such-and-such unit of work; or the rememberance data-structure is
shared among threads in an MT-safe manner and is updated by some
downstream consuming thread).  After some deadline (e.g., timer
expires) the units of work which have gone too long without a
successful-completion acknowledgement are resubmitted in the hope that
they will not be lost due to message-queue overflow this subsequent
time.  This repeats until all units of work are successfully completed
(i.e., until the submitted-unit-of-work rememberance data-structure is
drawn down to empty, such as during periods of system quiescence or as
a prerequisite to a orderly system shutdown).

> Thanks for the detailed info.
> Dave

  You are welcome.  The embedded real-time system viewpoint has been
substantially absent in MT topics in Boost.  I hope that I & others
can correct that absence, because embedded real-time systems have been
a community using MT (i.e., threads in a shared address-space) for a
long time.  The UNIX interprocess concurrency (in the days/paradigm
before multiple threads per process) must live with
separate-address-space limitations (or shared memory limitations)
which can cramp certain MT-architectures' style.  The real-time
community has been working with multiple threads in a shared
address-space for over 2 decades.  Thus a large industrial body of
shared-address-space MT "existing practice" regarding multithread
applications is in the world real-time embedded systems.  We must not
ignore or sideline that large body of "existing practice".  If an MT
concurrency library (e.g., Boost.Threads) can successfully be the
solution-space for the entire real-time MT problem-space, then that MT
library fulfills its core mission on which greater & greater
abstraction can be built.  If an MT concurrency library cannot be the
solution-space for the entire real-time MT problem-space (e.g., due to
lack of expressivity/features or lack of efficiency), then that MT
library fails a nontrivial portion of its core mission of modeling the
best portions of "existing practice".  For the C++0x standard to not
address the real-time MT space would be sad missed opportunity.