[Boost-users] [Iostreams] Zlib compression sometimes creates corrupt data on OSX

Subject: [Boost-users] [Iostreams] Zlib compression sometimes creates corrupt data on OSX
From: Michael Jung (boost-users_at_[hidden])
Date: 2013-10-02 11:07:00

Hello,

I’m having the following problem with my program: If I save data on OSX
using a boost::iostreams::filtering_ostream with a
boost::iostreams::zlib_compressor() the resulting file is sometimes
corrupted.

The following example code reproduces the problem:

#include <fstream>
#include <iostream>
#include <vector>
#include <boost/iostreams/filtering_stream.hpp>
#include <boost/iostreams/filter/zlib.hpp>

int main(int argc, char** argv)
{
  if (argc != 2) return 1;
  std::ifstream file(argv[1]);
  if (file)
  {
    file.seekg(0, std::ios::end);
    std::streampos length = file.tellg();
    file.seekg(0, std::ios::beg);

    std::vector<char> buffer1(length);
    std::vector<char> buffer2(length);

    file.read(&buffer1[0], length);

    {
      std::ofstream ofs("test.dat", std::ios_base::binary);
      boost::iostreams::filtering_ostream fos;
      fos.push(boost::iostreams::zlib_compressor());
      fos.push(ofs);

      fos.write(&buffer1[0], length);
    }
    {
      std::ifstream ifs("test.dat", std::ios_base::binary);
      boost::iostreams::filtering_istream fis;
      fis.push(boost::iostreams::zlib_decompressor());
      fis.push(ifs);

      fis.read(&buffer2[0], length);
    }

    std::cout << (buffer1 == buffer2 ? "equal" : "not equal")
            << std::endl;
  }
}

This piece of code reads a file that is specified as command line
argument, compresses and saves it to “test.dat”, reloads and
decompresses it and then compares the result with the content of the
original file.

Using it with two files linked below leads to the following output:
$ ./bugtest example1_working.txt 
equal
$ ./bugtest example1_not_working.txt 
not equal

i.e. in the second case the compression creates corrupt data. I verified
that the problem is in the compression part by trying to decompress
test.dat on Linux using zlib-flate which gives an error (incorrect data
check). If I compress the file manually on Linux using zlib-flate and
decompress it on OSX with code similar to the above, the input file is
reproduced correctly.

The two files used above are save files of my software.
You can download them from:
http://neptun.iup.uni-heidelberg.de/files/example1_working.txt
http://neptun.iup.uni-heidelberg.de/files/example1_not_working.txt

Another pair of files with the problem is:
http://neptun.iup.uni-heidelberg.de/files/example2_working.txt
http://neptun.iup.uni-heidelberg.de/files/example2_not_working.txt

I haven’t been able to find out what is special about the ones causing
the problem, so I just put them up as they are. The files are quite
similar, but show a few differences on every (or almost every?) line. If
needed, I can create other pairs of files showing the same problem.

I compiled the above code in the following way:
c++ -stdlib=libc++ -std=c++11 test.cpp -I
/Users/michael/projects/boost-svn-build/include/ -c
c++ -stdlib=libc++ -std=c++11 -o test test.o
/Users/michael/projects/boost-svn-build/lib/libboost_iostreams.a -lz

The boost libraries were built from svn as follows:
./bootstrap.sh --prefix=../boost-svn-build --with-libraries=iostreams
--with-toolset=clang
./b2 -j6 link=static toolset=clang cxxflags="-std=c++11 -stdlib=libc++"
linkflags="-stdlib=libc++" install

Clang version is: “Apple LLVM version 4.2 (clang-425.0.28) (based on
LLVM 3.2svn)”
I am running OS X 10.8.2.

The same files are compressed correctly with the above code on Linux,
compiled with GCC 4.6.3.

I think this problem might be a bug in boost or maybe also in the llvm
libraries (c.f. https://svn.boost.org/trac/boost/ticket/8150 , which
looks like a similar problem). Compiling boost with zlib 1.2.8 instead
of using zlib 1.2.5, which is shipped with OSX, leads to the same
problem. Both Boost 1.54.0 and the svn version are affected. Should I
create a ticket for this on the bug tracker?

Or is this just me using the library incorrectly??

Best regards,
Michael
-- 
  Michael Jung
  michaeljung_at_[hidden]