Re: [boost] New Lib "Beast", HTTP + WebSocket protocols

Subject: Re: [boost] New Lib "Beast", HTTP + WebSocket protocols
From: Vinnie Falco (vinnie.falco_at_[hidden])
Date: 2016-04-24 09:14:38

On Sun, Apr 24, 2016 at 7:09 AM, Bjorn Reese <breese_at_[hidden]> wrote:
> This looks like a nice library...

Thanks for the kind words!

> It may be a good idea if we joined the two projects together. Would you
> consider such an option?

I'm open to collaboration but also cautious. Beast.HTTP was designed
throughout to have a narrow interface. It offers only a universal
model for the HTTP message, and functions to parse, serialize,
deserialize, and send/receive on sockets. It offers both synchronous
and asynchronous functionality. And it accomplishes these goals with
an interface that resembles Boost.Asio as closely as possible, to
eliminate the learning curve for using the library.

It is this author's opinion that the more a library tries to do, the
more controversial it is and the harder it will be to get through the
boost review process. There's a strong need for simple free functions
to send and receive HTTP messages on a socket in as few lines of code
as possible. Beast.HTTP achieves this. Example code:

    using namespace beast::http;
    boost::asio::ip::tcp::socket sock(ios);
    ...
    request<empty_body> req({method_t::http_get, "/", 11});
    req.headers.replace("Host", "boost.org:80");
    req.headers.replace("User-Agent", "Beast.HTTP");
    write(sock, req);

Full, compiling example program here:
http://vinniefalco.github.io/beast/beast/intro/example.html

Remember that Beast includes both HTTP and WebSocket. The HTTP support
was driven by the need of the websocket stream to perform a HTTP
Upgrade handshake when initializing the session. I think a litmus test
for the interface quality of a HTTP library, is the ease in which it
allows an implementation to perform this websocket handshake. Here is
actual code from beast::websocket::stream which synchronously performs
the WebSocket HTTP Upgrade handshake:

    template<class NextLayer>
    void stream<NextLayer>::
    handshake(boost::string_ref const& host,
       boost::string_ref const& resource, error_code& ec)
    {
       std::string key;
       http::write(stream_,
       build_request(host, resource, key), ec);
       if(ec)
           return;
       http::response<http::string_body> resp;
       http::read(next_layer_, stream_.buffer(), resp, ec);
       if(ec)
           return;
       do_response(resp, key, ec);
    }

The function above is not possible with Boost.Http in its current
incarnation, as there is no client support nor is there support for
synchronous operations. Based on the design choices visible at the tip
of trunk (for example, requiring a stream constructed around the
socket to send and receive messages) it seems unlikely that such
simple operations will be possible in future incarnations of that
library unless major changes in its design are made.

> Last year, the Boost.Http project went through a formal Boost review.
> The summary [2] shows what the Boost community expects from a HTTP
> library.

I'll quote and address each of the points brought up in [2], the
feedback from the review of Boost.Http:

"Boost.Http currently only provides a server-side API, but the
reviewers felt that a client-side API would be usable to more users."

Beast.HTTP is completely role-agnostic, and works for building clients
and servers.

"There was also a recurring request for Boost.Http to be a header-only
 library. The HTTP parser currently used is not header-only and that
is  the main obstacle towards a header-only Boost.Http library."

Agree 100%. Beast.HTTP also uses the NodeJS parser, and that's the
only bit of code that is not header only. Thursday I started on a
header-only parser, here's what we have so far (please keep in mind,
this is a work in progress):
https://github.com/vinniefalco/rippled/blob/my-parser/src/beast/test/http/my_parser.cpp

This parser will not have any dependencies except the standard library
so if necessary the code could be pinched for other projects.

"Some of the discussion revolved around what level of abstraction
would be appropriate for Boost.Http. The views ranged from wanting
higher-  level APIs..."

Higher-level APIs are great but there is danger in offering increasing
levels of abstraction. At each increase, the target audience
diminishes and the chances of design choices made in the abstraction
becoming inappropriate for particular use-cases goes up.

No matter how strong the desire for higher level APIs, they need to be
built on a foundation. Beast.HTTP provides the correct foundation; it
is that which cannot be broken down further and it is that upon which
everything else can be built. As such, it offers library virtue with
its current feature set.

"...all the way down to simply wanting a HTTP parser/generator and
then leave all the socket and buffer management up to the user."

By design, Beast.HTTP does not perform any buffering or socket
management. Such layers can be built on top; one of the examples
creates a pipelining stream, see:
https://github.com/vinniefalco/Beast/blob/master/examples/http_stream.h
https://github.com/vinniefalco/Beast/blob/master/examples/http_async_server.h

"Boost.Http currently creates an associative array for all header
fields.  One reviewer explored the idea of using an incremental (push
or pull)  HTTP parser as part of the API to let users decide which
header fields  to copy and which to discard."

The Beast.HTTP read algorithm allows customization of the Parser
template argument (*), allowing any type that meets the requirements
to be used as the implementation for parsing messages.

(*) planned feature

"Some reviewers also felt that HTTP/2 should be part of Boost.Http,
partly because that would demonstrate the extensibility of the current
 design, and partly because the library would be in a stronger
position to attract users if it offers more than its competitors."

The IETF adopted as a goal for HTTP/2, to leave the message the same
(while changing its wire format). Therefore, Beast.HTTP's message
model is already HTTP/2-friendly.

As for the extensibility of the design, free functions to send and
receive HTTP messages are fundamentally incompatible with HTTP/2,
which requires ongoing
state to decompress headers and multiplex streams. And yet, we know
that free functions to send and receive HTTP/1 messages are useful and
sorely needed.

Furthermore HTTP/2 adds features that don't make sense for HTTP/1.
What would you do with the stream ID? It is not part of the message
model, because it describes a transport level property (and what
meaning would it have for a HTTP/1 message? or someone who is using
the message object but not using any sockets at all?) What about the
interface to inform the peer of a new receive window size? What about
setting stream priorities/weights? How do we sort all this out?

The interface for HTTP/1 should consist of a universal message model,
plus functions to send and receive those messages on sockets. The
interface for HTTP/2 should be a class template that wraps a socket
(similar in style to beast::websocket::stream), deals in the same
universal message model, and offers member functions to send and
receive those messages with associated stream IDs as well as adjust
HTTP/2-specific session parameters.

We respectfully disagree with those reviewers who feel that a single
interface should serve the needs of sending and receiving both HTTP/1
and HTTP/2 messages.

> In order to address one of these expectations
> we have a Boost Summer of Code project this year for a HTTP parser.

Hopefully this header-only parser will be done before the summer begins.

One area where Beast.HTTP could use work is in the examples. Another,
bigger project would be to develop a HTTP/2 stream class template for
Beast using its message model. These are areas where contributions
would be most welcomed and likely the most productive use of external
effort.