[boost] Re: Iostreams filter buffering policy leads to extremelysmallreads.

From: George M. Garner Jr. (gmgarner_at_[hidden])
Date: 2005-04-18 14:09:17

Johnathan,

> I'd appreciate it if you would comment out the declaration and 
> implementation of
> xsgetn and see if you still experience problems.

It makes the problem worse.  Then I get reads for a single character from
std::streambuf<>::uflow():

virtual int_type uflow()
{ // get a character from stream, point past it
    return (_Traits::eq_int_type(_Traits::eof(), underflow())
    ? _Traits::eof() : _Traits::to_int_type(*_Gninc()));
}

>> Fortunately, it is relatively trivial to disable Iostreams buffering
>> altogether and write a buffering_shim_filter.
>
> You can simply set the buffer size to zero when you add a filter to a 
> chain.

Actually not!  I was much chagrinned to find that you cannot disable input 
buffering.  Setting the input buffer and pback buffers to 0 actually results 
in an input buffer size of either 4 or 1, depending on whether or not I 
disable your STLPort workaround.  :-(  I do think it should be possible for 
someone not using STLPort to disable input buffering.

The following modified code appears to work:

template<typename T, typename Tr, typename Alloc, typename Mode>
void indirect_streambuf<T, Tr, Alloc, Mode>::open
(const T& t, std::streamsize buffer_size, std::streamsize pback_size)
{
using namespace std;
// Normalize buffer sizes.
buffer_size =
(buffer_size != -1) ?
buffer_size :
is_filter<T>::value ?
default_filter_buffer_size :
default_buffer_size;
pback_size =
(pback_size != -1) ?
pback_size :
default_pback_buffer_size;
// Construct input buffer.
if (can_read()) {
// Begin gmg modifications here
// It should be possible to disable this workaround for people
// not using STLPort.
#ifndef FORGET_STLPORT
pback_size_ =
std::max( static_cast<streamsize>(2), // STLPort needs 2. //illegal token on 
right side of
pback_size );
#endif //FORGET_STLPORT
if(streamsize size = pback_size_ + buffer_size)
{
in().realloc(size);
if (!shared_buffer())
init_get_area();
}
}
// End gmg modifications here
// Construct output buffer.
if (can_write() && !shared_buffer()) {
if (buffer_size != 0)
out().realloc(buffer_size);
init_put_area();
}
storage_ = wrapper(t);
flags_ |= f_open;
if (can_write() && buffer_size)
flags_ |= f_output_buffered;
}

template<typename T, typename Tr, typename Alloc, typename Mode>
typename indirect_streambuf<T, Tr, Alloc, Mode>::int_type
indirect_streambuf<T, Tr, Alloc, Mode>::underflow()
{
using namespace std;
// Begin gmg modifications here
buffer_type& buf = in();

// If buffering is disabled return eof().
if(buf.size() == 0)
return traits_type::eof(); //gmg
if (!gptr()) init_get_area();
//buffer_type& buf = in();
// End gmg modifications here
if (gptr() < egptr()) return traits_type::to_int_type(*gptr());
// Fill putback buffer.
streamsize keep = std::min( static_cast<streamsize>(gptr() - eback()),
pback_size_ );
if (keep)
traits_type::move( buf.data() + (pback_size_ - keep),
gptr() - keep, keep );
// Set pointers to reasonable values in case read throws.
setg( buf.data() + pback_size_ - keep,
buf.data() + pback_size_,
buf.data() + pback_size_ );
// Read from source.
streamsize chars =
obj().read(buf.data() + pback_size_, buf.size() - pback_size_, next_);
setg(eback(), gptr(), buf.data() + pback_size_ + chars);
return chars != 0 ?
traits_type::to_int_type(*gptr()) :
traits_type::eof();
}
template<typename T, typename Tr, typename Alloc, typename Mode>
std::streamsize indirect_streambuf<T, Tr, Alloc, Mode>::xsgetn
(char_type* s, std::streamsize n)
{
using namespace std;
buffer_type& buf = in();
if (!gptr() && buf.size() > 0) init_get_area();
streamsize total = 0;
do {
// Fill request from buffer if anything is available.
if (streamsize avail =
std::min(n, static_cast<streamsize>(egptr() - gptr())))
{
traits_type::copy(s, gptr(), avail);
gbump((int) avail);
total += avail;
s += avail;
n -= avail;
}
else
{
//Either buffering is disabled or the buffer has been exhausted.
// If the buffer size is less than or equal to n fill the request
// directly from the source.  This includes the case where input
// buffering has been disabled.

if(buf.size() <= n)
{
streamsize to_read = buf.size();
if(to_read == 0)
to_read = n;
if(streamsize amt = obj().read(s, to_read, next_))
{
s += amt;
n -= amt;
total += amt;
}
else
{
// Something is screwed. Get out of dodge.
break;
}
}
else
{
//The number of bytes requested is less than the buffer size.
// Call overflow to refill the input buffer.
// s and n will be updated from the (now full) input buffer on
// the next iteration of the do-loop.
if(traits_type::eq_int_type(traits_type::eof(), underflow()))
break;
}
}
} while(n > 0);
return total;
}

This appears to work so far.

Regards,

George.