Re: Boost.Buffers: network-agnostic buffer types and algorithms

From: Andrzej Krzemienski (akrzemi1_at_[hidden])
Date: 2023-02-07 07:40:01

wt., 7 lut 2023 o 00:31 Vinnie Falco via Boost <boost_at_[hidden]>
napisał(a):

> On Mon, Feb 6, 2023 at 2:43 PM Gavin Lambert via Boost
> <boost_at_[hidden]> wrote:
> > Perhaps, though as I said I find the "optional is more fundamental"
> > argument more compelling, so by that logic the compatibility code should
> > be in serialization and not optional.
>
> I just want to underscore that the question of "where does optional
> go" is completely orthogonal to this proposed Boost.Buffers library,
> and the buffers::source and buffers::sink interfaces in particular. My
> proposed library has nothing to do with serialization and everything
> to do with providing buffer-oriented interfaces.
>
> In fact, buffers::source is more fundamental than serialization.
> Boost.Serialization sits between buffers::source and the algorithm for
> serialization of an optional. This diagram shows the relationship:
>
>     { host } <---> Boost.Buffers <---> Boost.Serialization <---> {
> optional }
>
> In this diagram "host" is the host program or library which is
> implemented in terms of Boost.Buffer's abstract interfaces
> (buffers::source for the case of using Boost.Serialization). As seen
> in the diagram, the question of where the serialization spec for
> "optional" goes, is unrelated to Boost.Buffers.
>
> Another way to think of it is like this. If Boost.Serialization
> implements the buffers::source interface, then any host which uses
> Boost.Buffers will be able to make use of types archived through
> Boost.Serialization. I'm not saying whether this is particularly
> useful or not as that would depend on the specific use-case. For the
> upcoming Boost.Http.Proto library, this means that anything which
> defines a serialization for Boost.Serialization could be attached as
> the body of an HTTP message.
>
> The diagram for Boost.JSON looks different:
>
>     { host } <---> Boost.Buffers <---> Boost.JSON
>
> Here, if Boost.JSON wraps its existing streaming serializer for the
> buffers::source interface then any Boost.Buffers-enabled host can make
> use of JSON sources without depending on the Boost.JSON library.
> There's a network effect at play; every library that enables
> Boost.Buffers geometrically increases the utility of every other
> library that uses Boost.Buffers. Furthermore, this scales to any
> number of libraries and resolves the question of where the
> buffers::sink (or buffers::source) algorithm goes. It goes in the
> library which already implements a buffer-oriented data type. This
> includes but is not limited to:
>
>     * JSON
>     * XML
>     * Boost.Serialization archives
>     * Mustache templates and output
>

As indicated earlier, I support the extraction of Boost.Buffers into a
separate Boost library. Let me offer some further thoughts.

1. It is more of a question. These buffers::sink and buffers::source being
mentioned in this thread: are they already present in one of the Boost
libraries (ASIO, Beast)?

2. For any "vocabulary interface" we could separate out three parts: (1)
the interface, (2) the sinks, (3) the sources. I am slightly abusing the
terms, so let me give an example of the STL. Iterators are the interface.
The algorithms are "sinks": they do not have to know about containers. The
containers are the "sources" they do not have to know about iterators. If
the STL was a Boost library, the only thing that other Boost libraries
would depend on would be iterators -- not the algorithms, not the
containers -- because it is only the iterators -- the interface -- that
lets you plug your library into the system.

I see the same situation with the proposed Boost.Buffers. The only thing
that multiple libraries (Boost and non-Boost) would benefit from is the
buffer interface (boost::sink, boost::source). No buffer implementations,
no buffer algorithms. As you mentioned earlier, things like a circular
buffer are shared between at least two libraries, so there is a merit in
separating this, but this is just an engineering trade-off, rather than a
vocabulary interface design (like boost::sink, boost::source).

This remark applies to every library with a sound vocabulary-like
interface: the interface should go separately. But that would be a very
small library, maybe not worth a separate infrastructure.

3. There are a number of interfaces, where everyone can plug their type, in
the STD and Boost that have an overlap. We have the IOStream interface, we
have Boost.Serialization interface, std::format is coming, and we have now
Boost.Buffer being proposed. Can you make a clear distinction why
Boost.Buffers is different? Why do we need another one? Are previous ones
defective (and can be superseded), or do they play a different,
incompatible role?

I suppose that buffers have much to do with buffering, that is working with
chunks of messages. But how does this work with JSON? Can you even start
thinking about parsing JSON if you only have a part, broken at an arbitrary
position? Also, IOStreams have a layer of buffers. What is the relation
there (Between IOStream buffers and the proposed Boost.Buffers)?

4. Would Boost.Buffers satisfy my every use case for buffers? Are buffers
only about how you allocate a new chunk of memory? Or are they about
identifing a place where you can cut your message into meaningful portions?

Regards,
&rzej;