Re: C++ Deflate (zlib-like) library

From: Domen Vrankar (domen.vrankar_at_[hidden])
Date: 2020-03-09 21:24:45

On Mon, Mar 9, 2020 at 9:56 PM Janson R. via Boost <boost_at_[hidden]>
wrote:

> On 09/03/2020 21:25, Domen Vrankar via Boost wrote:
>
> > Now for the API of the library proposed by OP.
> >
> > I've skimmed over the code and allot of it seems like just-a-bit-above-C
> > flavour - I could be wrong as I didn't find the examples and unit tests
> > also haven't shown me how to easily use the library.
>
> Observation is mostly correct, and it is one of the main reasons I've
> come to ask for feedback here; Vinnie rightfully pointed out that the
> question being asked is "what [does] a clean, Boost-quality, modern C++
> API to codec that operates on memory buffers [look] like[?]".
>
> Since the emails discussions have begun I've integrated the following
> prototype API after suggestion by a few of the people here:
> ```
> optional<std::string> easy_compress(string_view in, wrap wrapping =
> wrap::none);
>
> optional<std::string> easy_uncompress(string_view in, wrap wrapping =
> wrap::none);
> ```
>
> While it is definitely something that was needed (a friend of mine got
> to use it and obviously found it much more trivial for his use than the
> more expert-mode preexisting API), it definitely does not cover all the
> possible use cases of such a library.
>

I've seen this, didn't know that it was later addition but was surprised by
the names.

The issue that I have with this is that it expects a large file to be
loaded into memory - I've mostly worked with large files and memory
constraints (for various reasons that were out of my control and were
policy and not technology related) so I'd expect at least streams support
so that you can write:

read -> process -> write

pattern where you only handle part of the data at a time.

> > What I'd like to see from a C++ library in general and Boost library in
> > particular is a high quality API that would cover capabilities of
> > libarchive (and use it as the underlying implementation at first). The
> APIs
> > of primitives (inflate/deflate) should come at a later stage - perhaps
> not
> > before C++ standardization of the APIs (and even then I'd expect that std
> > libs would use what is already available in distro X instead of
> reinventing
> > the wheel). My old lib has a poor API [...]   but it's still a big
> simplification compared to
> > libarchive C API and that's what I expect from a high quality wrapper -
> > simplify allot but still let an advanced user squeeze the performance of
> > the underlying implementation if/when needed (complexity should be
> reserved
> > for those and hidden from us mere mortals).
>
> Does that imply that you would welcome quality C++ wrappers for common C
> libraries in the Boost ecosystem?
>

I'm not a Boost author or maintainer so it's not my place to say but from
where I stand that's the place where Boost belongs.

> > For the capabilities of such API I'd expect it to work seamlessly with
> std
> > streams (I don't care if people believe that they are slow, until they
> are
> > part of the standard and there is no better std replacement I expect
> > libraries of such type to work with them out of the box) and support for
> > binary data not just std::string as in/out buffers in situations where
> you
> > have everything already in memory or are using memory mapped files
> (perhaps
> > also (Boost::)ASIO/Networking TS streams but that's your domain so you
> > probably have a better opinion on how ASIO network payload could be
> > compressed/decompressed on the fly).
> >
>
> That's exactly the kind of feedback and information I'm looking for; so
> thanks! Possible quick function declarations are more than welcome to add!
>

You're tackling inflate/deflate - libarchive is handling archives (e.g.
.tar.gz not just gz part). The way I see it you're competing in the wrong
league and I can't give constructive feedback on that. Your library would
be a building block for writing libraries such as libarchive but that is a
C lib and would not bother with your lib. On the other hand I wouldn't use
it as I usually had to handle archives not a single compressed file.

That's the reason I'm recommending going from the other side - high level
API that handles the (from my point of view) usual use cases and give users
a reason to use your library instead of just falling back to C libraries by
default.

> > Regarding the OP's library as it currently stands I'd suggest integration
> > of Boost::Beast instead (if it's a benefit) and split it into a separate
> > library once it can serve as a high level Boost library's backbone (and
> get
> > that high level API into Boost first).
> >
>
> Boost.Beast will need at some point at least some code being transferred
> back into it to allow some HTTP features to be implemented (which are
> the very first reason why I got on that adventure to begin with).
>

Yeah... That's the problem that I see. Library is handling an internal
implementation detail of Boost.Beast and that's a small subset of what you
need for a proper general purpose compression/decompression lib. Like I
said I'm not certain that this is the right angle to begin from as it's
competing against instead of enhancing higher level C libs.

Regards,
Domen

> Regards
>
> Janson
>
> _______________________________________________
> Unsubscribe & other changes:
> http://lists.boost.org/mailman/listinfo.cgi/boost
>