Re: C++ Deflate (zlib-like) library

From: Domen Vrankar (domen.vrankar_at_[hidden])
Date: 2020-03-10 08:31:07

On Tue, Mar 10, 2020, 4:15 AM Vinnie Falco via Boost <boost_at_[hidden]>
wrote:

> On Mon, Mar 9, 2020 at 7:30 PM Gavin Lambert via Boost
> <boost_at_[hidden]> wrote:
> > That answer may be different depending on your context; for example
> > Boost.Asio provides a number of buffer concepts and methods for
> > manipulating them.
> >
> > Which are good, in that they're templates and support all kinds of
> > different buffer concepts (from basic vectors and strings through to
> > dynamic multi-buffer streams), and you only pay for the complexity that
> > you actually use.
> >
> > They're also bad, in that they're templates and they're harder to
> > consume generically without forcing your own code to also be a template.
> >
> > But if you're in the context of Boost.Beast or some other app that uses
> > Asio, it makes sense to use these buffer concepts rather than
> > reinventing the wheel, especially with these things
> > probably-maybe-eventually landing in the standard.
>
> Yes now we're getting to the meat of it! You are proposing: Use Asio's
> BufferSequence concepts (or something substantially similar) in the
> high level ZLib interface (or a generic codec interface).
>
> On the face of it this is not a bad idea, but the reality is that all
> codecs work a single buffer at a time. This is different from Asio or
> OS-level I/O, which support a scatter/gather interface where the
> buffers are all sent together. Thus, it seems to be that at the lowest
> level of a codec interface, it will be single-buffer oriented.
>
> On top of that single-buffer interface, algorithms which work on a
> buffer sequence or range of buffers can be built. Asio's buffer
> sequence concepts are tragically connected to the asio::const_buffer
> and asio::mutable_buffer concrete types, which makes it necessary to
> have all of Asio as a dependency. Using these types out of the box for
> a low-level ZLib / codecc library doesn't seem like a great idea.
>
> I did propose that Asio split out its buffer sequence concepts and
> types into a separate Boost library. This would allow Beast's HTTP
> parser to be in its own library, without requiring Asio. The
> experience with the large Beast library makes me now more in favor of
> smaller, more numerous libraries with fewer dependencies.
>
> Perhaps something like this could be an improvement on ZLib's C API:
>
>     struct input_buffer
>     {
>         input_buffer(
>             void const* data,
>             std::size_t size);
>
>         bool is_empty() const noexcept;
>         void const* data() const noexcept;
>         std::size_t used() const noexcept;
>         std::size_t remaining() const noexcept;
>         void consume(std::size_t) noexcept;
>     };
>

First questions regarding your API - why do we need all those functions?

You compress/decompress in chunks so all you need is a const std::span for
input and std::span for output and call decompress on a for loop.

Since compression/decompression is not random access but streaming you can
alternatively just accept an input iterator pair/range for input/output and
cover tue streaming case without a for loop. With std::istream_iterator you
can then handle std::stream (file, memory etc.) support. You can also chain
for e.g. boost.spirit x3 parser and lazy read->decompress->parse/process

Can't existing asio buffers be written to through span/iterators?

    struct output_buffer; // similar to input_buffer
>
> Then we can express a compression function using these types, and get
> better safety:
>
>     // returns the number of bytes written to the output buffer
>     std::size_t compress( output_buffer& out, input_buffer& in );
>

Whit iterators/ranges described above you wouldn't need to return
std::size_t so return can be used for error codes instead.

Regards,
Domen

A function to compress a ConstBufferSequence into a
> MutableBufferSequence could be built from this one primitive, and a
> dependency on including <boost/asio/buffer.hpp>
>
> Thanks
>
> _______________________________________________
> Unsubscribe & other changes:
> http://lists.boost.org/mailman/listinfo.cgi/boost
>