[boost] Iostreams filter buffering policy leads to extremely small reads.

From: George M. Garner Jr. (gmgarner_at_[hidden])
Date: 2005-04-09 16:51:40

Johnathan,

The Iostreams library provides for built in buffering of filters which in 
principle should permit efficient use of filters even with streams that do 
not support filtering.  However, in practice it does not appear to live up 
to expectations.  More specifically, the following line in 
indirect_streambuf.hpp can lead to extremely fragmented reads to lower 
filters or devices:

std::streamsize indirect_streambuf<T, Tr, Alloc, Mode>::xsgetn
(char_type* s, std::streamsize n)
{
    [...]
     streamsize amt = obj().read(s + avail, n - avail, next_);  // n - avail 
may equal 10 or less

    [...]
}

This is true even if the filter buffer size is set to 1MiB or more.

Large buffer sizes (> 2 MiB) greatly enhances read and write performance on 
many modern operating systems.  This is particularly felt when reading and 
writing large files (> 4 GiB).  Ideally, if you set a filter buffer policy 
(e.g. to 1 MiB) you would like all reads to lower filters and devices to 
request that value, except possibly for the last read.

Fortunately, it is relatively trivial to disable Iostreams buffering 
altogether and write a buffering_shim_filter.  But then I do not understand 
what purpose Iostreams buffering serves.  I Googled through the online 
documentation and I didn't find a detailed discussion of its objectives, 
though there were some comments that touched on the subject matter during 
the review process.  Perhaps you can further elaborate this subject matter.

Regards,

George.