Re: C++ Deflate (zlib-like) library

From: Domen Vrankar (domen.vrankar_at_[hidden])
Date: 2020-03-09 20:25:56

On Mon, Mar 9, 2020 at 2:26 PM Vinnie Falco via Boost <boost_at_[hidden]>
wrote:

> The "Shitty API" is not having constructors and destructors, and
> relying on the user to call a function to "free" some resources
> instead of having their lifetime managed automatically via RAII.
> Anyway... this has veered off-topic.
>
> You came close to discussing the topic when you linked this library
> wrapper:
>
> <https://github.com/do-m-en/libarchive_cpp_withwrapper
> <https://github.com/do-m-en/libarchive_cpp_wrapper>>
>
> But you didn't actually discuss its API and the suitability to zlib.
>

I did not intend to as I'm not a compression/decompression expert but I did
notice a pattern of "let's just use C" in the past week or so on this
mailing list and it struck my nerve a bit.

But here goes...

First the C vs C++ API that I've already touched upon a bit. I quite often
(most of the time) use Clang experimental forks on my Ubuntu computer which
means that from time to time I have to do the "recompile the world" due to
boost library X ABI compatibility not playing nicely between gcc/g++ and
Clang (the last one that I've had to bother with a couple of months ago was
using Boost::DLL as it depends on Boost::filesystem even with C++17
compiler... and Boost::filesystem was causing compatibility issues - not to
mention that I'm now using two different filesystem implementations std for
my code and boost for Boost::DLL). This is the in favor of C API part - and
since for me all libraries below my code are low level as far as I'm
concerned OpenSSL and ZLib are as much low level as Boost::filesystem or
Boost Spirit X3 (you can always draw lines - I draw mine at the "when do I
have to recompile a library with the same version that my distro already
provides).

On the other hand I'm far too old to torture my memory by for e.g.
remembering which functions should be called in which order and belong to
the same "class" that accepts the handle next, remembering to call
free_this_resource or instead of using:

out1 = decompress(data, type::gzip); // type is in this case a tag
out2 = decompress(data, type::bzip2);

I have to write a switch statement because:

out1 = decompress_gzip(data);
out2 = decompress_bzip2(data);

so I need to write C++ wrappers (either just abusing std::unique_ptr with
default destructor or a wrapper on the level of the linked libarchive
wrapper). These are the wrappers that I'd like to have in Boost.

So bottom line is that I'd prefer a C API library that can be provided by
the distro (zlib, libarchive,...) and a C++ part (provided by Boost that is
also part of the distro) that would be compiled along side my code - but
for small libs header only the entire lib would also do the trick.

This are the reasons for me preferring C++ std lib only stuff (so migrating
Boost libs towards standard) - it comes with the experimental compiler that
I have to compile anyway - and why I'm ranting on Boost mailing list from
time to time that Boost libs should move to std dependencies once those are
in standard.

This would solve my recompile the world issues (or at least shrink them a
bit).

Looking over this library I see:
>
>     template<ns_writer::format FORMAT, ns_writer::filter FILTER>
>     static writer make_writer(std::ostream& stream, size_t block_size);
>
> Are you suggesting that ZLib should use ostream and istream as the
> interface for writing and reading data?
>

Now for the API of the library proposed by OP.

I've skimmed over the code and allot of it seems like just-a-bit-above-C
flavour - I could be wrong as I didn't find the examples and unit tests
also haven't shown me how to easily use the library.

The problem that I have with such a library is that it's not useful for my
work - it's too low level and is reinventing the wheel. The low level part
was already implemented by zlib, pigz and other C libraries and I find it
counter productive to try and play catch up in C++ since those libraries
are already compatible with C++.

What I'd like to see from a C++ library in general and Boost library in
particular is a high quality API that would cover capabilities of
libarchive (and use it as the underlying implementation at first). The APIs
of primitives (inflate/deflate) should come at a later stage - perhaps not
before C++ standardization of the APIs (and even then I'd expect that std
libs would use what is already available in distro X instead of reinventing
the wheel). My old lib has a poor API as it's not exactly configurable
(allocators etc.) but it's still a big simplification compared to
libarchive C API and that's what I expect from a high quality wrapper -
simplify allot but still let an advanced user squeeze the performance of
the underlying implementation if/when needed (complexity should be reserved
for those and hidden from us mere mortals).

For the capabilities of such API I'd expect it to work seamlessly with std
streams (I don't care if people believe that they are slow, until they are
part of the standard and there is no better std replacement I expect
libraries of such type to work with them out of the box) and support for
binary data not just std::string as in/out buffers in situations where you
have everything already in memory or are using memory mapped files (perhaps
also (Boost::)ASIO/Networking TS streams but that's your domain so you
probably have a better opinion on how ASIO network payload could be
compressed/decompressed on the fly).

For an example of why I believe that a libarchive level C++ API should be
preferred in Boost: a while back I needed TLS SNI support for Boost
Asio/Beast as I wanted to use multiple https domains on a single IP. Asio
and Beast are useful libraries but don't provide server side support, on
the other hand OpenSSL world is evolving at it's own pace so the library
already has the support and since Asio provides a high level SSL context
wrapper and Beast doesn't do any weird magic with it I just wrote a simple
context that just works:

https://github.com/do-m-en/boost_asio_tls_sni_context

OpenSSL is a community focused on network cryptography so rewriting a
subset as a Boost lib would've been counter productive as I'm quite certain
that such a lib would not have so much support. On the other hand
networking support is as far as I'm concerned on solid foundation in Boost
so a high level wrapper combination is IMO the best solution.

Regarding the OP's library as it currently stands I'd suggest integration
of Boost::Beast instead (if it's a benefit) and split it into a separate
library once it can serve as a high level Boost library's backbone (and get
that high level API into Boost first).

P.S. There used to be a zip stream in Boost (had a bug in at least one
Boost version on AIX OS so it was useless for me at that point but it was
there) - don't know what happened to it but I probably wouldn't use it as
for me libarchive level of functionality is what I mostly need and once you
already have a hammer in the codebase...

Regards,
Domen

> Thanks
>
> _______________________________________________
> Unsubscribe & other changes:
> http://lists.boost.org/mailman/listinfo.cgi/boost
>