[boost] Re: Compression usinig IOStreams library

From: Jonathan Turkanis (technews_at_[hidden])
Date: 2004-09-11 15:04:15

"George M. Garner Jr." <gmgarner_at_[hidden]> wrote in message
news:chv36b$gg$1_at_sea.gmane.org...
> Johnathan,
>
> I got zlib, gzip and bzip2 compression working with the IOStreams library
> using zlib-1.2.1 and bzip2-1.0.1.

Good.

> There were just a few minor glitches as
> follows:
>
> 1. The zip file format output by zlib_compressor does not appear to work
> with either Winzip 9.0 SR-1 or Windows XP native zip support.
> zlib_decompressor is able to decompress files previously compressed using
> zlib_compressor.

Dumb question - is Winzip supposed to be able to read the zlib format? I usually
use it to decompress .zip or tar.gz archives.

> 2. Compiling IOStreams bzip2 support, there is a conflict with the macro
> "small" that is defined somewhere in the header files that you include.  I
> had to insert the following at the beginning of bzip2.hpp to get it to
> compile:

I hate these stupid lowercase macros! I've already had problems with the maco
unix in gzip.hpp and newline_filter.hpp.

I think I'll just change 'small' to slow everywhere -- unless there's a macro
'slow'.

> #include <boost/config/abi_prefix.hpp> // Must be the last header.
> ++#ifdef small
> ++#undef small
> ++#endif //small
> namespace boost { namespace io {

> 3. bzip2_decompressor enters into an infinite loop inside of
> symmetric_filter_adapter_impl<>::read() unless the following modification to
> symmetric_filter_adapter.hpp is made:

> bool eof = (state_ & f_eof) != 0;
> if (buf_.ptr() != buf_.end() || eof) {
> bool done =
> !filter_->filter(
> const_cast<const char_type*&>(buf_.ptr()),
> buf_.end(), next_s, end_s, eof
> --) && eof;
> ++) || eof;

Thanks. It took me a long time to get this to work with my sample data, but
since I tried only a few samples, I guess I'm not surprised it's still wrong.
Somehow, though, I seem to remember "|| eof" failing, too.

> 4. There doesn't appear to be any convenient way to copy decompressed data
> to a wide character stream.  Compressed data is a binary data stream which
> by definition is narrow.  However the case occurs where the decompressed
> data is wide.  So you would like to be able to do the following:
>
> ifstream ifile("hello.zip", ios_base::in | ios_base::binary);
> filtering_streambuf<input> in;
> in.push(zlib_decompressor());
> in.push(ifile);
> boost::io::copy(in, wcout);
>
> But this doesn't compile.  This would appear to be another application for
> your     converting_ostream discussed earlier.

Right. The usage would be the same as above, but it would actually work.

> 5. Do you really want to include the external compression headers (e.g.
> zlib.h, bzip2.h) in your library?  This will create a problem keeping your
> library in sync with the external compression libraries.  Your zlib header
> is already out of date, for example.  Wouldn't it be better to require the
> user to obtain the external headers and include macros to contigently
> compile dependent sections?

I guess the way to do this would be to have the user define variables which tell
bjam where to look for the headers. I think we should make packages containing
win32 binaries and the relevant headers available somewhere. I think Jeff
Garland once suggested making them available through the wiki. They could be
updated more frequently than the boost distribution.

> My compression code follows at the end of this message.  Now for some real
> fun.  I am going to try and attach an overlapped filebuf to one of your
> streams. :-)

Good luck!

BTW, I just now realized that you're the one I was discussing these issues with
over at microsoft.public.vc.stl ;-)

>
> Regards,
>
> George.

Jonathan