Re: [Boost-users] [zlib] Compression intended use

Subject: Re: [Boost-users] [zlib] Compression intended use
From: Anthony Foiani (tkil_at_[hidden])
Date: 2013-08-11 13:30:39

Michael Powell <mwpowellhtx_at_[hidden]> writes:

> I've got a package of output files that I want to bundle, tarball,
> gzip, or however I can manage to compress them.

You're also welcome to use some code I put together to create a Zip64
archive on the fly:

  http://ajf-prog.blogspot.com/2013/07/streaming-zip64-archive.html

Full implementation here (relies on zlib, obviously):

  https://github.com/tkil/ajf-prog/tree/master/Zip64Streamer

This is used on a data recorder, which is an "embedded" target (albeit
one that would have made a pretty nice desktop 10-15 years ago).  To
reduce complexity on the instrument, the code does not try to optimize
the output form at all -- it always uses the 64-bit extensions (which
are rarely needed by this instrument, but I can't guarantee that it
won't have to transfer a file larger than 4GiB).

As the name implies, it is also designed for streaming -- it does no
seeking, and can write directly out to a socket if you like.  In
particular, it allows me to allocate a buffer once, fill it with
compressed data, then pass that buffer off to the transmission
machinery; no copying necessary.

Output has been tested with Info-Zip on Linux; Windows Explorer and 7z
on Windows 7.  Apparently even the 10.8 Finder doesn't support Zip64
files, so it doesn't work there (but you can build/get Info-Zip for
that platform.)

It turns out that info-zip's "zip" code also has a library mode, and
also has a streaming mode, although neither is particularly well-
documented.  Unfortunately, while the authors were sympathetic and
helpful, they could not make any promises about thread safety.

Hope this helps.

Good luck,
Tony