Re: C++ Deflate (zlib-like) library

From: Vinnie Falco (vinnie.falco_at_[hidden])
Date: 2020-03-10 03:14:49

On Mon, Mar 9, 2020 at 7:30 PM Gavin Lambert via Boost
<boost_at_[hidden]> wrote:
> That answer may be different depending on your context; for example
> Boost.Asio provides a number of buffer concepts and methods for
> manipulating them.
>
> Which are good, in that they're templates and support all kinds of
> different buffer concepts (from basic vectors and strings through to
> dynamic multi-buffer streams), and you only pay for the complexity that
> you actually use.
>
> They're also bad, in that they're templates and they're harder to
> consume generically without forcing your own code to also be a template.
>
> But if you're in the context of Boost.Beast or some other app that uses
> Asio, it makes sense to use these buffer concepts rather than
> reinventing the wheel, especially with these things
> probably-maybe-eventually landing in the standard.

Yes now we're getting to the meat of it! You are proposing: Use Asio's
BufferSequence concepts (or something substantially similar) in the
high level ZLib interface (or a generic codec interface).

On the face of it this is not a bad idea, but the reality is that all
codecs work a single buffer at a time. This is different from Asio or
OS-level I/O, which support a scatter/gather interface where the
buffers are all sent together. Thus, it seems to be that at the lowest
level of a codec interface, it will be single-buffer oriented.

On top of that single-buffer interface, algorithms which work on a
buffer sequence or range of buffers can be built. Asio's buffer
sequence concepts are tragically connected to the asio::const_buffer
and asio::mutable_buffer concrete types, which makes it necessary to
have all of Asio as a dependency. Using these types out of the box for
a low-level ZLib / codecc library doesn't seem like a great idea.

I did propose that Asio split out its buffer sequence concepts and
types into a separate Boost library. This would allow Beast's HTTP
parser to be in its own library, without requiring Asio. The
experience with the large Beast library makes me now more in favor of
smaller, more numerous libraries with fewer dependencies.

Perhaps something like this could be an improvement on ZLib's C API:

    struct input_buffer
    {
        input_buffer(
            void const* data,
            std::size_t size);

        bool is_empty() const noexcept;
        void const* data() const noexcept;
        std::size_t used() const noexcept;
        std::size_t remaining() const noexcept;
        void consume(std::size_t) noexcept;
    };

    struct output_buffer; // similar to input_buffer

Then we can express a compression function using these types, and get
better safety:

    // returns the number of bytes written to the output buffer
    std::size_t compress( output_buffer& out, input_buffer& in );

A function to compress a ConstBufferSequence into a
MutableBufferSequence could be built from this one primitive, and a
dependency on including <boost/asio/buffer.hpp>

Thanks