Re: C++ Deflate (zlib-like) library

From: Domen Vrankar (domen.vrankar_at_[hidden])
Date: 2020-03-09 20:31:43

On Mon, Mar 9, 2020 at 9:25 PM Domen Vrankar <domen.vrankar_at_[hidden]>
wrote:

> On Mon, Mar 9, 2020 at 2:26 PM Vinnie Falco via Boost <
> boost_at_[hidden]> wrote:
>
>> The "Shitty API" is not having constructors and destructors, and
>> relying on the user to call a function to "free" some resources
>> instead of having their lifetime managed automatically via RAII.
>> Anyway... this has veered off-topic.
>>
>> You came close to discussing the topic when you linked this library
>> wrapper:
>>
>> <https://github.com/do-m-en/libarchive_cpp_withwrapper
>> <https://github.com/do-m-en/libarchive_cpp_wrapper>>
>>
>> But you didn't actually discuss its API and the suitability to zlib.
>>
>
> I did not intend to as I'm not a compression/decompression expert but I
> did notice a pattern of "let's just use C" in the past week or so on this
> mailing list and it struck my nerve a bit.
>
> But here goes...
>
> First the C vs C++ API that I've already touched upon a bit. I quite often
> (most of the time) use Clang experimental forks on my Ubuntu computer which
> means that from time to time I have to do the "recompile the world" due to
> boost library X ABI compatibility not playing nicely between gcc/g++ and
> Clang (the last one that I've had to bother with a couple of months ago was
> using Boost::DLL as it depends on Boost::filesystem even with C++17
> compiler... and Boost::filesystem was causing compatibility issues - not to
> mention that I'm now using two different filesystem implementations std for
> my code and boost for Boost::DLL). This is the in favor of C API part - and
> since for me all libraries below my code are low level as far as I'm
> concerned OpenSSL and ZLib are as much low level as Boost::filesystem or
> Boost Spirit X3 (you can always draw lines - I draw mine at the "when do I
> have to recompile a library with the same version that my distro already
> provides).
>
> On the other hand I'm far too old to torture my memory by for e.g.
> remembering which functions should be called in which order and belong to
> the same "class" that accepts the handle next, remembering to call
> free_this_resource or instead of using:
>
> out1 = decompress(data, type::gzip); // type is in this case a tag
> out2 = decompress(data, type::bzip2);
>
> I have to write a switch statement because:
>
> out1 = decompress_gzip(data);
> out2 = decompress_bzip2(data);
>
> so I need to write C++ wrappers (either just abusing std::unique_ptr with
> default destructor or a wrapper on the level of the linked libarchive
> wrapper). These are the wrappers that I'd like to have in Boost.
>
> So bottom line is that I'd prefer a C API library that can be provided by
> the distro (zlib, libarchive,...) and a C++ part (provided by Boost that is
> also part of the distro) that would be compiled along side my code - but
> for small libs header only the entire lib would also do the trick.
>
> This are the reasons for me preferring C++ std lib only stuff (so
> migrating Boost libs towards standard) - it comes with the experimental
> compiler that I have to compile anyway - and why I'm ranting on Boost
> mailing list from time to time that Boost libs should move to std
> dependencies once those are in standard.
>
> This would solve my recompile the world issues (or at least shrink them a
> bit).
>
> Looking over this library I see:
>>
>>     template<ns_writer::format FORMAT, ns_writer::filter FILTER>
>>     static writer make_writer(std::ostream& stream, size_t block_size);
>>
>> Are you suggesting that ZLib should use ostream and istream as the
>> interface for writing and reading data?
>>
>
> Now for the API of the library proposed by OP.
>
> I've skimmed over the code and allot of it seems like just-a-bit-above-C
> flavour - I could be wrong as I didn't find the examples and unit tests
> also haven't shown me how to easily use the library.
>
> The problem that I have with such a library is that it's not useful for my
> work - it's too low level and is reinventing the wheel. The low level part
> was already implemented by zlib, pigz and other C libraries and I find it
> counter productive to try and play catch up in C++ since those libraries
> are already compatible with C++.
>
> What I'd like to see from a C++ library in general and Boost library in
> particular is a high quality API that would cover capabilities of
> libarchive (and use it as the underlying implementation at first). The APIs
> of primitives (inflate/deflate) should come at a later stage - perhaps not
> before C++ standardization of the APIs (and even then I'd expect that std
> libs would use what is already available in distro X instead of reinventing
> the wheel).
>

Just to clarify this... From C++ std lib I'd expect that it would expose
the primitives from which libarchive level API would be built but I'd
expect the standardization of such low level APIs that could use zlib and
co. as the underlying implementation of those primitive APIs - and I
wouldn't expect the distros to re-implement them as one implementation of a
certain library for one distro is enough and effort should be made to
improve those instead of having two implementations where possible.

> My old lib has a poor API as it's not exactly configurable (allocators
> etc.) but it's still a big simplification compared to libarchive C API and
> that's what I expect from a high quality wrapper - simplify allot but still
> let an advanced user squeeze the performance of the underlying
> implementation if/when needed (complexity should be reserved for those and
> hidden from us mere mortals).
>
> For the capabilities of such API I'd expect it to work seamlessly with std
> streams (I don't care if people believe that they are slow, until they are
> part of the standard and there is no better std replacement I expect
> libraries of such type to work with them out of the box) and support for
> binary data not just std::string as in/out buffers in situations where you
> have everything already in memory or are using memory mapped files (perhaps
> also (Boost::)ASIO/Networking TS streams but that's your domain so you
> probably have a better opinion on how ASIO network payload could be
> compressed/decompressed on the fly).
>
> For an example of why I believe that a libarchive level C++ API should be
> preferred in Boost: a while back I needed TLS SNI support for Boost
> Asio/Beast as I wanted to use multiple https domains on a single IP. Asio
> and Beast are useful libraries but don't provide server side support, on
> the other hand OpenSSL world is evolving at it's own pace so the library
> already has the support and since Asio provides a high level SSL context
> wrapper and Beast doesn't do any weird magic with it I just wrote a simple
> context that just works:
>
> https://github.com/do-m-en/boost_asio_tls_sni_context
>
> OpenSSL is a community focused on network cryptography so rewriting a
> subset as a Boost lib would've been counter productive as I'm quite certain
> that such a lib would not have so much support. On the other hand
> networking support is as far as I'm concerned on solid foundation in Boost
> so a high level wrapper combination is IMO the best solution.
>
> Regarding the OP's library as it currently stands I'd suggest integration
> of Boost::Beast instead (if it's a benefit) and split it into a separate
> library once it can serve as a high level Boost library's backbone (and get
> that high level API into Boost first).
>
> P.S. There used to be a zip stream in Boost (had a bug in at least one
> Boost version on AIX OS so it was useless for me at that point but it was
> there) - don't know what happened to it but I probably wouldn't use it as
> for me libarchive level of functionality is what I mostly need and once you
> already have a hammer in the codebase...
>
> Regards,
> Domen
>
>
>> Thanks
>>
>> _______________________________________________
>> Unsubscribe & other changes:
>> http://lists.boost.org/mailman/listinfo.cgi/boost
>>
>