[boost] Compression usinig IOStreams library

From: George M. Garner Jr. (gmgarner_at_[hidden])
Date: 2004-09-11 09:47:39

Johnathan,

I got zlib, gzip and bzip2 compression working with the IOStreams library 
using zlib-1.2.1 and bzip2-1.0.1.  There were just a few minor glitches as 
follows:

1. The zip file format output by zlib_compressor does not appear to work 
with either Winzip 9.0 SR-1 or Windows XP native zip support. 
zlib_decompressor is able to decompress files previously compressed using 
zlib_compressor.

2. Compiling IOStreams bzip2 support, there is a conflict with the macro 
"small" that is defined somewhere in the header files that you include.  I 
had to insert the following at the beginning of bzip2.hpp to get it to 
compile:

#include <boost/config/abi_prefix.hpp> // Must be the last header.
++#ifdef small
++#undef small
++#endif //small
namespace boost { namespace io {

3. bzip2_decompressor enters into an infinite loop inside of
symmetric_filter_adapter_impl<>::read() unless the following modification to 
symmetric_filter_adapter.hpp is made:

bool eof = (state_ & f_eof) != 0;
if (buf_.ptr() != buf_.end() || eof) {
bool done =
!filter_->filter(
const_cast<const char_type*&>(buf_.ptr()),
buf_.end(), next_s, end_s, eof
--) && eof;
++) || eof;

4. There doesn't appear to be any convenient way to copy decompressed data 
to a wide character stream.  Compressed data is a binary data stream which 
by definition is narrow.  However the case occurs where the decompressed 
data is wide.  So you would like to be able to do the following:

ifstream ifile("hello.zip", ios_base::in | ios_base::binary);
filtering_streambuf<input> in;
in.push(zlib_decompressor());
in.push(ifile);
boost::io::copy(in, wcout);

But this doesn't compile.  This would appear to be another application for 
your     converting_ostream discussed earlier.

5. Do you really want to include the external compression headers (e.g. 
zlib.h, bzip2.h) in your library?  This will create a problem keeping your 
library in sync with the external compression libraries.  Your zlib header 
is already out of date, for example.  Wouldn't it be better to require the 
user to obtain the external headers and include macros to contigently 
compile dependent sections?

My compression code follows at the end of this message.  Now for some real 
fun.  I am going to try and attach an overlapped filebuf to one of your 
streams. :-)

Regards,

George.

// compression_example.cpp
#include <windows.h>
#include <tchar.h>
#include <crtdbg.h>
#include <fstream>
#include <iostream>
#include <boost/io/filtering_streambuf.hpp>
#include <boost/io/filtering_stream.hpp>
#include <boost/io/copy.hpp>
#include <boost/io/zlib.hpp>
#include <boost\io\gzip.hpp>
#include <boost/io/bzip2.hpp>
#include <libs/io/src/zlib.cpp>
#include <libs/io/src/bzip2.cpp>
#ifdef _UNICODE
#define _tcout wcout
#define _tcerr wcerr
#else
#define _tcout cout
#define _tcerr cerr
#endif //_UNICODE
bool zip_compression()
{
using namespace std;
using namespace boost::io;
try
{
{
ofstream ofile("hello.zip", ios_base::out | ios_base::binary);
filtering_ostream out;
out.push(zlib_compressor());
out.push(ofile);
// write() is used as opposed to << so as to permit writing Unicode 
characters
// to binary compression streams. Is there a better way?
out.write((char*)__T("This gets compressed using one of the compression 
formats.\n"), _tcslen(__T("This gets compressed using one of the compression 
formats.\n")) * sizeof(_TCHAR));
}
ifstream ifile("hello.zip", ios_base::in | ios_base::binary);
filtering_streambuf<input> in;
in.push(zlib_decompressor());
in.push(ifile);
_tcout.clear();
boost::io::copy(in, _tcout);
return true;
}
catch(zlib_error& e)
{
_tcerr << __T("zlib error: ") << dec << e.error() << endl;
}
return false;
}
bool gzip_compression()
{
using namespace std;
using namespace boost::io;
try
{
{
ofstream ofile("hello.gz", ios_base::out | ios_base::binary);
filtering_ostream out;
out.push(gzip_compressor());
out.push(ofile);
_RPT1(_CRT_WARN, "The length of output string is %ld\n", _tcslen(__T("This 
gets compressed using one of the compression formats.\n")));
out.write((char*)__T("This gets compressed using one of the compression 
formats.\n"), _tcslen(__T("This gets compressed using one of the compression 
formats.\n")) * sizeof(_TCHAR));
}
ifstream ifile("hello.gz", ios_base::in | ios_base::binary);
filtering_streambuf<input> in;
in.push(gzip_decompressor());
in.push(ifile);
_tcout.clear();
size_t nRead = boost::io::copy(in, _tcout);
_RPT1(_CRT_WARN, "The length of input string is %ld\n", nRead);
return true;
}
catch(gzip_error& e)
{
_tcerr << __T("gzip error: ") << dec << e.error() << __T(" zlib error: 0x") 
<< e.zlib_error() << endl;
}
return false;
}
bool bzip2_compression()
{
using namespace std;
using namespace boost::io;
try
{
{
ofstream ofile("hello.bz2", ios_base::out | ios_base::binary);
filtering_ostream out;
out.push(bzip2_compressor());
out.push(ofile);
_RPT1(_CRT_WARN, "The length of output string is %ld\n", _tcslen(__T("This 
gets compressed using one of the compression formats.\n")));
out.write((char*)__T("This gets compressed using one of the compression 
formats.\n"), _tcslen(__T("This gets compressed using one of the compression 
formats.\n")) * sizeof(_TCHAR));
//_RPT1(_CRT_WARN, "The length of output string is %ld\n", nWritten);
}
ifstream ifile("hello.bz2", ios_base::in | ios_base::binary);
filtering_streambuf<input> in;
in.push(bzip2_decompressor());
in.push(ifile);
_tcout.clear();
size_t nRead = boost::io::copy(in, _tcout);
_RPT1(_CRT_WARN, "The length of input string is %ld\n", nRead);
return true;
}
catch(bzip2_error& e)
{
_tcerr << __T("bzlib error: ") << dec << e.error() << endl;
}
return false;
}
int _tmain()
{
zip_compression();
gzip_compression();
bzip2_compression();
return 0;
}