Re: Boost.Buffers: network-agnostic buffer types and algorithms

From: Vinnie Falco (vinnie.falco_at_[hidden])
Date: 2023-02-03 01:02:06

On Thu, Feb 2, 2023 at 1:58 PM Andrzej Krzemienski <akrzemi1_at_[hidden]> wrote:
> Today the basics for networking seems like good candidates for
> vocabulary types. Is Boost.Buffer a candidate for such a vocabulary type?

In theory, yes. The intent is for Buffers to be more general than just
networking.

> First question, is this library to be consumed directly by user programs,
> or only by other libraries, such as Boost.Beast and Boost.Http.Proto?

User programs could use it when they want to express algorithms in
terms of ranges of buffers. My personal use-case is Beast, HTTP,
Websocket, and JSON but this could be extended to say, base64 encoding
and decoding, calculating SHA-2 digest over a range of buffers,
simplified front end to things like zlib. Example:

    template<
        class DynamicBufferOut,
        class DynamicBufferIn >
    result< void >
    deflate(
        z_params&,
        DynamicBufferOut&
        DynamicBufferIn&,
        bool end_of_data);

> Is it possible that the final program that uses Boost.Buffer will not use Boost.ASIO?

That is possible, yes.

> For me, as a consumer of networking libraries, who can afford to
> download and build the entire set of Boost libraries, the biggest
> advantage would be that Buffers would be well documented and tested.

That is an advantage yes but in my opinion the advantage is that you
get access to a rich and useful set of algorithms, containers, and
concepts, without also depending on a particular network
implementation.

> I am not convinced by the argument with Boost.JSON not depending on ASIO.

Well, users definitely do not want to drag in Asio with their JSON.

> We have (I think a still unresolved issue) of Boost-serializing Boost.Optional.
> Should the code responsible for the serialization of Boost.Optional be part of
> Boost.Serialization or Boost.Optional? But we will certainly not solve it by
> introducing yet another library.

I think that the cases of Boost.Serialize with boost::optional, and
serialization of JSON using Boost.JSON and Boost.Buffers are two
completely different cases. Boost.Serialize does not define a
serialization format but rather a framework for allowing the user to
define serialization. Boost.JSON is something else entirely; it is a
library which implements a specific protocol for describing data using
structured text.

If you look at Boost.JSON you see that it has json::parser and
json::serializer. These classes work a buffer at a time and they
support streaming; that is, that the entire input or output need not
be presented at once in order to make progress. How does Boost.Buffers
help here? Well.. it provides the means for a library such as JSON,
which has algorithms to produce or consume a buffer at a time, to
type-erase its algorithm using the buffers::source or buffers::sink
interfaces:

    /// An algorithm which consumes filled buffers
    struct  sink
    {
        virtual ~sink() = 0;

        /** Called when data is available
        */
        virtual result<void> write( span< const_buffer const > ) = 0;
    };

    /// An algorithm which produces filled buffers
    struct source
    {
        /** The results of reading from the source.
        */
        struct results
        {
            error_code ec;
            std::size_t bytes = 0;
            bool more = false;
        };

        virtual ~source() = 0;

        /** Called when more data is required

            This function is invoked when more data
            is required. The subclass should place
            zero or more bytes into the buffers
            referenced by `dest`, and return a
            `results` value with the members set
            appropriately.
            <br>
            Partial success is possible.
        */
        virtual results read( span< mutable_buffer const > dest ) = 0;
    };

This is not really any different from libraries that support
operator<<(std::ostream&), except that the vocabulary type is
buffers::sink and buffers::source instead of std::istream and
std::ostream. When Boost.JSON implements these APIs, it is not
"supporting Boost.Serialization" it is merely providing a wrapper for
the algorithms it already has (json::parser and json::serializer) to
adapt them to a common, agreed-upon interface (the buffers::sink and
the buffers::source).

Currently users complain that Boost is a huge bundle of libraries, and
at some level this is true. But it is a bundle of libraries for a
reason, because releasing the collection together as a unit allows us
to ensure that all the components work together. But more importantly
it allows a member library to leverage the advantages of having other
member libraries available to it (Config, Variant2, mp11 come to
mind). The point of implementing Boost.Buffers' sink and source
interfaces in JSON is to let us leverage the benefits of the
monolithic distribution model.

> Boost.JSON doesn't need to depend on the implementation of the circular buffer.

Right, it wouldn't. It would only be implementing the sink and source
abstract interfaces. This let's JSON participate in the growing
ecosystem of Boost.Buffers enabled algorithms. I think JSON is the
place for it, because the model can scale. Candidate libraries which
would add value by implementing source and sink do so in the library
itself. The model where a single library has to depend on an ever
increasing number of other libraries in order to implement their
wrappers is unsustainable.

--
Regards,
Vinnie
Follow me on GitHub: https://github.com/vinniefalco