[boost] Re: Re: IOStreams formal review start

From: Jonathan Turkanis (technews_at_[hidden])
Date: 2004-08-30 18:00:46

"Carlo Wood" <carlo_at_[hidden]> wrote in message
news:20040830190632.GB1116_at_alinoe.com...
> On Mon, Aug 30, 2004 at 11:32:28AM -0600, Jonathan Turkanis wrote:
> > > Apart from some broken links and typos in the documentation/comments,
> >
> > Would you please point them out?
>
> On
http://home.comcast.net/~jturkanis/iostreams/libs/io/doc/classes/alphabetically.html
>
> converting_stream and converting_streambuf do not have hyperlinks.

They're not implemented. This is explained here -- http://tinyurl.com/4hkut -- 
but I probably shouldn't have included them in the index. Thanks.

> stream_facade and streambuf_facade link to non-existing pages.

Which page contains the bad links?

> http://home.comcast.net/~jturkanis/iostreams/boost/io/filtering_stream.hpp
>
> contains a comment:
>
> // Macro: BOOST_IO_DEFINE_FILTER_STERAM(name_, mode_)
>
> while the macro signature is actually:
>
> #define BOOST_IO_DEFINE_FILTER_STREAM(name_, chain_type_, default_char_)

Thanks -- I'm pleased to see someone is actually looking at the source ;-)

> [..snip..]
> > For now, let me just make these points:
> >
> > 1. There is already a mechanism to avoid copying data in certain cases: by
> > implementing resources which model the concept Direct.
>

<snip good discussion>

Thank you for the detailed explanation. I don't think I understand all of it
yet -- I'll have to look at your implementation. Perhaps you could elaborate on
the concept of a 'message'. Are you thinking of something which is specific to
network programming or is more general?

BTW, this sounds vaguely like the descriptions I have read of the Apache 'bucket
brigade' filtering framework. Any connections?

At any rate, you are right about the limitations of the Direct concept. I
introduced it originally to handle memory mapped files. Later I realized that it
is really a degenerate case of a more general concept. In general, for output,
when you run out of room in the array provided by the Direct resource you would
be able to request a new array to write to. Similarly for input you could
request a new array when you finish reading to the end of the current array. For
random access, you might request an array of a certain length containing a given
offset -- you might not get exactly what you requested, but you'd always get an
explanation of how the returned array relates to the requested array. (All this
would be handled internally by streambuf_facade, of course.)

I didn't implement this for four reasons:

- I had limited time
- I wasn't sure there was a real need for it
- It would make the library harder to document and learn, and
- There are cases where  resources have to be handled directly rather than
indirectly through streambuf_facades (see, e.g., <boost/io/copy.hpp>);
generalizing the Direct concept would lead to nightmares in those situations.

> [...]
> > streambuf_facade would look like this:
> >
> >          template< typename Resource,
> >                          typenmae Tr = ...,
> >                          typename Buffering = basic_buffering<Resource>,
> >                          ... >
> >          class streambuf_facade;
> >
> > This would allow essentially any buffering policy to be employed.
>
> Including the one I described above?  Having a linked list of
> allocated memory blocks and reference counting 'message' objects
> that reserve parts of it and communicate with the buffer about
> those parts really being free for reuse?

I hope so. If not, I can change it so it does ;-)

> > The main
> > application I have in mind is cases where the underlying resource should be
> > accessed in mulitples of a certain block size.
>
> That block size (my message size I think) does not have to be fixed.
> There are many protocols put there that have variable sized messages! ;)

Fixed block size is just one possible buffering policy.

> > In fact, I have already (mostly) implemented such an approach, but I have
not
> > incorporated it into the library for several reasons:
> > - The buffering policy has a rather bulky interface which I think I may be
able
> > to simplify
> > - I'm not convinced yet that it's a performance win -- only tests will tell.
If
> > it makes only a small difference in a few cases, it may not be worth
> > complicating the library.
>
> Well, this only makes sense for large servers with thousands of connections
> that all burst data in huge quantities... exactly the kind of applications I
like
> to write ;).

Just to clarify, what do you mean by 'this' here? Your lists of dynamically
allocated memoery blocks, or my idea of a buffering policy?

> > To summarize, I'd like to make streambuf_facade flexible enough so that you
> > don't have to substitute you own home-brewed version. This is *not* a
criticism
> > of your library: if you have good ideas about how to make streambufs more
> > efficient, I'd like to incorporate them directly into streambuf_facade -- 
> > possibly as buffering policies -- with your permission.
>
> My ideas are free :p.  You won't be able to use libcw ('s code) anyway
> because it wasn't written with a friendly interface in mind - I designed
> it with two goals: 1) Speed, 2) The ability to adapt to yet-unknown demands,
> in other words 'flexibility' at the user level (or 'one size fits all',
> but that really sounds too bad :p).  As a result, the interface so
> complex that someone who doesn't understand it (and that is everyone
> else besides me) will call it bloated ;)

I'll use it just for inspiration, then :-)

> > > Another thing that is bothering me is that the whole
> > > presence of anyting 'stream-like' (ostream/istream) seems
> > > not in the right place here.  This is not only because

<snip argument I couldn't follow>

> > Could you rephrase this whole argument? I don't think I follow it.
>
> I am afraid I cannot explain it ... it's experience :/.
> By providing an interface foobar_stream while you really
> only need to provide foobar_streambuf you do something
> that makes my alarm bells go off. The word "inflexible"
> comes to mind.  This will lead to problems of the kind
> that a user wants to do something but can't.  You are
> limiting yourself too much this way.

How can you be limited by additional functionality? All the streambufs are there
right along side the streams. You can use them raw, with standard streams, or
with the thin wrappers provided by the library.

> Another thing, and I can explain that better, is that
> users only write serializers for std::ostream (and please
> don't ask them to do that again for filtering_ostream!).
> Therefore, if there has to a filtering_ostream then it
> MUST be derived from std::ostream

It is.

> AND still work
> (the same) if all you have - at any moment after construction
> and initialization - is a pointer to the std::ostream
> base class.

It should.

> >From that follows that if you don't NEED initialization
> for it - then you don't need the whole ostream class.
> However, what convinced me more is the notition of what
> an ostream really is: a hook to the operator<< classes.
> If you only need a hook and users will only write
> operator<<(std::ostream& ... functions, then why would
> you ever need something else then an ostream?  It is
> just too unlogical.  When I look at the interface of
> this library and see filtering_ostream then that strikes
> me as "impossible", you just CANNOT need that. So, why
> is it there?  You already answered that yourself
> later by the way: to make it easier for the user.
> You provide a filtering_ostream as wrapper around
> std::ostream so that the initialization functions for
> the *streambuf* look nicer (and surely, yes, this cleans
> up code that uses ostreams).

It's a convenience for people who like it. A lot of people wouldn't like a
network library that provided a socketbuf but no socket streams.

> Therefore, my only objection against filtering_ostream
> is that it HIDES the real interface: filtering_streambuf...
> which is doesn't hide as you told me now ;).
>
> So, you can consider this objection to be void.  But
> I still think you should make it a bit more clear that
> filtering_ostream is just candy, convenience - and not
> hide the real thing (filtering_streambuf) behind it in
> all your examples and documentation.  I completely missed it!

I think that's the real point. I need to stress in more places that
streambuf_facade is the main component. (But see http://tinyurl.com/6k86s: "The
fundamental component provided by the Iostreams Library is the class template
streambuf_facade.")

> > I think a buffering policy is the way to go.
>
> If you can add the functionality of libcw's dbstream into this
> library - then it will become my favourite boost lib ;)

Great!

> Err... probably not.

Darn, should have read ahead further :)

> There is another thing that I'd missing.
> But I can't ask to add that too; its too... ugly (I wanted to say
> complex).  When I use iostream classes I need TWO streambufs
> (one buffer for the input and another for the output).

Why can't you use a streambuf_facade with mode inout, which buffers input and
output separately? (http://tinyurl.com/6bjbl)

Thanks for your detailed comments!

Jonathan