Re: [boost] [beast] Chunking example

Subject: Re: [boost] [beast] Chunking example
From: Vinnie Falco (vinnie.falco_at_[hidden])
Date: 2017-07-04 18:18:45

On Tue, Jul 4, 2017 at 10:49 AM, Bjorn Reese via Boost
<boost_at_[hidden]> wrote:
> That was scary reading, because HTTPbis seemed to be unaware that such
> deprecation would break some of IETF's own standards, e.g. RFC 3507.

I agree :) That's why I made sure that Beast is capable of sending and
receiving chunk-extensions.

> Another use-case that is used in practice is for in-band meta-data.
> ... chunk extensions are used to carry the meta-data.

That makes sense, and that's a supported use-case. You can send the
extensions and you can receive them.

> Consider a device (e.g. a sensor), that
> publishes events (e.g. measurements) that are distributed unevenly
> and unpredictably over time. Each chunk contains a single event, so
> preserving the chunk boundary is important.

You're talking about a "perpetual message" of some kind, where the end
of the message never arrives. I think that's kind of an abuse of HTTP.
Isn't Websocket more suited for that? It supports message boundaries
which are not rewritten by intermediates (unlike HTTP).

With respect to detecting chunk boundaries on input, this is possible,
but... I can't guarantee it on output. In fact I'm not even sure how
you would convince Beast to keep a message in "perpetually sending"
mode since that is a use-case not anticipated by rfc7230.

I think the sane way to do what you want is to use Beast for sending
the header, which works great, and then take responsibility for
outputting the body chunks yourself. True, you'll have to be getting
your hands dirty with the protocol details of HTTP but that's already
assumed because you 1. want to know about chunks, 2. want to control
their output, 3. want to send an infinitely long message. I'm not
losing sleep that Beast doesn't make this use-case ridiculously easy.

>> The application sees the message body only after the chunked
>> Transfer-Encoding has been removed.
> ...
> Can you elaborate with an example?

Let me rephrase that, beast::http::parser appends chunk bodies into
the body container stored in the message (std::string for example).
>From the caller's perspective they just see that the message body is
growing. There's no guarantee that each increment of growth exactly
corresponds to one chunk.

However, if you want to see chunks that is possible by subclassing
basic_parser and handling it in the on_chunk callback, which is called
after the chunk header is received and gets passed the size of the
upcoming body as well as the chunk-extension.

I suppose I could easily add an optional user-facing callback to
beast::http::parser with the signature <void(std::uint64_t,
string_view, error_code&> to allow users to easily detect chunk
boundaries during parsing. It would be a handful of lines of code
added here and there (similar to the on_header callback already
present):
<https://github.com/vinniefalco/Beast/blob/78a065ba39836d91d7e70d93de7f9140f518083b/include/beast/http/parser.hpp#L298>

I haven't done it because I don't like to design theoretical features
which are unlikely to be used. I have found that when I do such
things, especially for features that I have little experience with
(like chunk extensions) I end up discovering later that the interfaces
don't satisfy real-world needs and then have to be changed.

> Beast should not break the end-to-end principle unless explicitly
> allowed by the application.

I think if users demand control over outgoing chunks, its reasonable
to ask that they manually output the chunked message body; Beast will
take care of the header.

>  Is it possible to provide a HTTP 200 response alias for Beast?
> ...
> This is needed for SHOUTcast internet radio servers.

There are no provisions for this. Beast's parser is strict. Responses
which don't start with a valid HTTP-version generate an error.

That said, I recognize that sometimes in the real world if you want to
get in to the club you have to pay the cover. I can add a simple
customization point to basic_parser giving the derived class the
opportunity to parse the status-line if it wants, to handle this case.

However, it won't be for free - I'll kindly ask that the stakeholders
(the people who intend to use the feature) participate in the design
and code review in order that the result suits their needs. Its not my
area of knowledge so if we want to get this right, those with more
experience should weigh in.

And by the way, thanks for pointing this out! This is exactly the sort
of thing that Beast needs in order to mature into something great.

> The chunk decorator uses boost::asio::null_buffers to overload the
> field trailer.
>
> Given your desire to adapt Beast to Networking TS, do notice that
> null_buffers are not part of N4656 (they have been replaced by
> dedicated wait functions.)

Good point!! That will have to be addressed when Beast is ported.
Right now the chunk decorator is pretty raw. Again, this is a case
where a feature has been created with no stakeholders to vet the
design. I wrote the minimum amount of interface necessary to make sure
Beast supports the feature. With feedback from people actually using
it in the field, it can be improved.

Thanks