[boost] Re: IOStreams formal review start

From: Jonathan Turkanis (technews_at_[hidden])
Date: 2004-08-30 12:32:28

"Carlo Wood" <carlo_at_[hidden]> wrote in message
news:20040830123305.GA23389_at_alinoe.com...
> Apart from some broken links and typos in the documentation/comments,

Would you please point them out?

> I am very impressed by the amount of work and profesional setup
> of the documentation and the over all design.

Thanks.

> This looks like
> it is a graduation project for a university or something (and
> perhaps it is).

I wish my 'graduation project' (i.e., dissertation) were going so well. :-)

> The only thing that really bothers me is that it seems not
> possible to replace std::streambuf being used by the library
> (ie streambuf_facade) with a custom streambuf implementation
> (also derived from std::streambuf of course).  I'd have use
> for that because very likely I'd want to use my own, optimized
> streambuf (one that adds an interface to be able to read/write
> to it in a way that there is hardly ever need to actually
> _copy_ data, see http://libcw.sourceforge.net/io/dbstreambuf.html)

This looks interesting. I'd like to look at it carefully before I comment on it
(which I'll try to do soon).

For now, let me just make these points:

1. There is already a mechanism to avoid copying data in certain cases: by
implementing resources which model the concept Direct.
2. There are several optimizations which I have held in reserve which would also
minimize copying:
    a) Allowing resources to advertise that they are streambuf-based, so that
i/o is performed directly to the underlying streambuf, with no additional layer
of buffering
    b) Giving special treatment to symmetric filters (used to implement
compression/decompression) to allow them to have direct access to the buffers of
adjacent streambufs in a chain.
    c) allowing for a category of 'transparent filters' which simply observe
character sequences, forwarding them unchanged. This would allow many useful
filters (such as the offsetbuf suggested by David Abrahams) to have essentially
zero overhead.
3. Instead of the current one-size-fits-all buffering policy, hardcoded in
<boost/io/detail/streambufs/indirect_streambuf.hpp>, it's possible to factor a
buffering policy out of the current implementation of streambuf_facade, so that
streambuf_facade would look like this:

         template< typename Resource,
                         typenmae Tr = ...,
                         typename Buffering = basic_buffering<Resource>,
                         ... >
         class streambuf_facade;

This would allow essentially any buffering policy to be employed. The main
application I have in mind is cases where the underlying resource should be
accessed in mulitples of a certain block size.

In fact, I have already (mostly) implemented such an approach, but I have not
incorporated it into the library for several reasons:
- The buffering policy has a rather bulky interface which I think I may be able
to simplify
- I'm not convinced yet that it's a performance win -- only tests will tell. If
it makes only a small difference in a few cases, it may not be worth
complicating the library.

To summarize, I'd like to make streambuf_facade flexible enough so that you
don't have to substitute you own home-brewed version. This is *not* a criticism
of your library: if you have good ideas about how to make streambufs more
efficient, I'd like to incorporate them directly into streambuf_facade -- 
possibly as buffering policies -- with your permission.

> Another thing that is bothering me is that the whole
> presence of anyting 'stream-like' (ostream/istream) seems
> not in the right place here.  This is not only because
> the std::ostream/std::istream class are merely 'hooks' to
> hook into the operator<< and operator>> functions which
> are primarily intended for text (human readable representations)
> while this library is about binary data - but more importantly
> because everything this library does is related to and at the
> level of streambuf's (which DO have a binary interface)
> This fact is most apparent by considering the fact that this
> code should work:
>
> filtered_ostream fout;
> fout.push(filter);
> fout.push(cout);

It works (with 'filtering_ostream'). What's wrong with it?

> std::ostream& out(fout); // Only have/use the std::ostream base class.
> out << "Hello World"; // This must use the filter.
>
> A much more logical API would therefore be:
>
> filtered_streambuf fbuf;
> fbuf.push(filter);
>
> std::streambuf& buf(fbuf);
>
> And then using 'buf' as streambuf for some ostream of operator<< inserters
> are desirable.

Could you rephrase this whole argument? I don't think I follow it.

If you mean that the best way to perform formatted i/o with custom stream
buffers is via plain istreams, ostreams and iostreams, then I'm familiar with
this argument. My opinion is that it's a matter of taste; personally, I like
having matching streams for my stream buffers. stream_facade and
filtering_stream are just thin wrapper, so they don't contribute much to the
size of the library. If you want to use plain standard library streams, you can
easily do so:

    filtering_istreambuf buf;
    in.push(regex_filter(regex("damn"), "darn"));
    in.push(zlib_decompressor());
    in.push(file_source("essay.z"));

   istream in(&buf);
   // read from in.

> To summarize:
>
> - I think that the stream interface should be ripped out and replaced
>   by one that is an equivalent streambuf.  Providing a stream interface
>   should be merely a 'convenience' interface and not the main API.

The stream interface already *is* just a convenience, as explained above.
Perhaps the misunderstanding stems from the fact that in the examples I tend to
use streams, since they are more familiar to most users than stream buffers.
'Ripping out' the stream interface would simply mean omitting the two files
stream_facade.hpp and filtering_stream.hpp, for a combined total of about 11k
;-)

> - This streambuf interface should use a 'Streambuf' template parameter
>   for its base class that only defaults to std::streambuf (and may demand
>   that it is derived from std::streambuf if that is really necessary) but
>   allows the base class to be replaced with a custom implementation.

I think a buffering policy is the way to go.

Thanks for your review!

Jonathan