Higher than expected memory usage by boost::iostreams::gzip_compressor

From: Alex Petty (pettyalex_at_[hidden])
Date: 2022-07-13 18:57:23

Hello,

I'm looking at adopting boost::iostreams for reading/writing compressed
streams. I'm testing with textual files commonly on the order of a couple
of GB uncompressed, and a couple hundred MB compressed. I noticed during
some initial testing that using a gzip_compressor to write a compressed
stream through boost::iostreams results in significant memory usage, I'm
seeing hundreds of MB per compressed stream. It looks like internally the
gzip_compressor must be doing some GC-type behavior, because I see its
memory usage increase to just under 1GB per stream with a gzip_compressor,
and then decrease again.

Is there any way to control the memory that gzip_compressor is using? I've
tried both increasing and decreasing the buffer size when gzip_compressor
is constructed, but that buffer is tiny and cannot be what's driving the
increased memory usage.

Thanks,

Alex