[Boost-users] [iostreams] buffer flushing problem

From: Darren Kessner (darren.kessner_at_[hidden])
Date: 2008-06-19 17:25:10

Hi all,

I have encountered some strange behavior while using the Boost  
iostreams zlib compression filter.

The problem is tricky to reproduce -- I have posted a tarball that  
includes an example program (pasted below) together with a 72k binary  
file that will be read in and compressed:
http://proteowizard.sourceforge.net/temp/zlib_error.tgz

If you're curious, the 72k data array comes from an array of doubles,  
which were intensities from a single scan on a mass spectrometer.  The  
problem occurs on platforms darwin, gcc, msvc.

In general, the functions run_filter_1 and run_filter_2 produce  
identical output.  However, when the input is the binary array read in  
from the 72k file, run_filter_1 appears not to flush some buffer  
properly. This may be due to my ignorance of how to force the buffers  
to flush, though it's not clear to me that this should be necessary  
with the use of boost::iostreams::copy().

Thank you in advance for any help!

Darren

#include "boost/iostreams/filtering_streambuf.hpp"
#include "boost/iostreams/filtering_stream.hpp"
#include "boost/iostreams/copy.hpp"
#include "boost/iostreams/filter/zlib.hpp"
#include "boost/iostreams/device/array.hpp"
#include <iostream>
#include <fstream>
#include <sstream>
#include <vector>

using namespace std;
using namespace boost::iostreams;

template <typename filter_type>
string run_filter_1(const string& buffer)
{
     ostringstream result(ios::binary);
     array_source source(&buffer[0], buffer.size());
     filtering_streambuf<input> in;
     in.push(filter_type());
     in.push(source);
     boost::iostreams::copy(in, result); // not flushing properly in  
all cases?
     return result.str();
}

template <typename filter_type>
string run_filter_2(const string& buffer)
{
     ostringstream result(ios::binary);
     filtering_ostream fos;
     fos.push(filter_type());
     fos.push(result);
     fos.write(&buffer[0], buffer.size());
     fos.pop();
     fos.pop(); // force flush buffer
     return result.str();
}

int main()
{
     // fill buffer with something trivial

     string buffer(72000, '\0');
     for (size_t i=0; i<buffer.size(); i++)
         buffer[i] = i;

     string compressed1 = run_filter_1<zlib_compressor>(buffer);
     string compressed2 = run_filter_2<zlib_compressor>(buffer);
     cout << "sizes: " << compressed1.size() << " " <<  
compressed2.size() << endl; // same
     assert(compressed1.size() == compressed2.size());

     // read in "bad.bin" into buffer

     ifstream is("bad.bin", ios::binary);
     if (!is)
     {
         cerr << "bad.bin not found\n";
         return 1;
     }

     is.read(&buffer[0], buffer.size());
     compressed1 = run_filter_1<zlib_compressor>(buffer);
     compressed2 = run_filter_2<zlib_compressor>(buffer);
     cout << "sizes: " << compressed1.size() << " " <<  
compressed2.size() << endl;
     assert(compressed1.size() != compressed2.size()); // different!

     // decompressing

     try
     {
         // compressed2 is correct
         string decompressed2 =  
run_filter_2<zlib_decompressor>(compressed2);
         cout << "decompressed2.size(): " << decompressed2.size() <<  
endl << flush;
         assert(decompressed2.size() == buffer.size());
         for (size_t i=0; i<buffer.size(); i++)
             assert(decompressed2[i] == buffer[i]);

         // throws zlib_error
         string decompressed1 =  
run_filter_1<zlib_decompressor>(compressed1);
         cout << "decompressed1.size(): " << decompressed1.size() <<  
endl << flush;
     }
     catch (zlib_error&)
     {
         cerr << "caught zlib_error\n";
         return 1;
     }

     return 0;
}

IMPORTANT WARNING: This message is intended for the use of the person or entity to which it is addressed and may contain information that is privileged and confidential, the disclosure of which is governed by
applicable law.  If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this information is STRICTLY PROHIBITED.

If you have received this message in error, please notify us immediately
by calling [phone] and destroy the related message.  Thank You for your cooperation.