[boost] Re: Functions as Filters (was Program Reuse...)

From: Jonathan Turkanis (technews_at_[hidden])
Date: 2005-01-13 18:12:44

Jonathan Turkanis wrote:
> Daniel James wrote:

>> Simon Tatham provides a nice motivation for this kind of thing at:

> Nice example! That's the type of evidence I was hoping Christopher
> would produce. Refering to one of my old messages, which doesn't seem
> to be archived yet, you have written a filter in form [B] (using
> streams as function arguments). I think [C] (using Sources and Sinks)
> would be sufficient here. So I'm leaning toward allowing filters
> along the lines of [C].

>> Not necessarily, he could use threads or fibres with pipes, although
>> that's quite expensive. That's why I was playing around with using a
>> Duff's Device style switch statement for implementing coroutines.
>
> I think I already answered this. Standard input streams don't have a
> way to indicate that input is temporarily unavailable. Therefore if
> the producer and consumer are operating in separate threads and the
> consumer gets ahead of the producer, a false EOF will be detected.
>
> Version [C] will not suffer from this problem, because the final
> version of the filter and device concepts will provide a way to
> distinguish EOF from EAGAIN. Unfortunately, there's no way to
> retrofit this onto the standard streams or stream buffers, since they
> do not recognize this distinction.

I may have spoken too soon (as I have several times in this thread ;-). Once the
run-length encoding example is rewritten to handle the possibility that get()
can return an indication that input is temporarily unavailable, I'm not sure the
simplfication will survive.

If we want to preserve the simplcity of the example without requiring that an
enitre stream of data be processed at once, we could switch to thread-aware
concepts. E.g.,

   template<typename AsyncSource, typename AsynSink>
   void decompress(AsyncSource& in, AsynSink& out)
   {
     char c;
     while((c = boost::io::blocking_get(in)) != EOF)  {
       if(c == 0xFF) {
         int len = boost::io::blocking_get(in);
         if(!in || !boost::io::blocking_get(in)) // Return an error.
         while(len--) boost::io::blocking_put(out, c);
       }
       else {
         boost::io::blocking_put(out, c);
       }
     }
   }

(I've been planning to introduce such concepts eventually, but definitely not in
the initial release.)

With some sacrifice of efficiency, we could even use your example unchanged, and
stipulate that the provided istream and ostream wait on some synchronization
object until input is available or EOF is reached (for the istream) or until the
output buffers have free space (for the ostream).

In short, I 'm not sure the co_filter idea can be implemented without some cost
in efficiency or simplicity.

Jonathan