[beast] help with http::dynamic_body and http::buffer_body

From: Klebsch, Mario (Mario.Klebsch_at_[hidden])
Date: 2021-04-26 14:20:25

Hello,

After  using boost::asio for several years I thought, I might give beast a try. I need to write a http server processing a POST request with an upload of a big file. So I thought, I could start with libs/beast/example/http/server/async. I soon found the function handle_request() and that I have to add something like

if (req.method() == http::verb::post && req.target()=="/xxx" )

to catch the http POST transaction. But since I have to process the upload of a large file, I thought that using a http::string_body might not be the best choice. I found several other body types in the documentation.  I thought
buffer_body ought to be fine, since the documentation says: ‘This allows for serialization of body data coming from external sources, and incremental parsing of message body content using a fixed size buffer.’.

incremental processing of the input stream is what I need. But now, I have no clue, how to change the example to use a buffer_body.

I changed http::request<http::string_body> req_ to http::request<http::buffer_body> req_. But now I must process the request message in chunks.

How do I get my callback invoked for each chunk?
Without additional changes, I get a read error: need buffer.

But the documentation on buffer_body does not help me. Buffer_body does not seem to have any methods, that I can call.
It does have a member type reader, that is documented as implementation-defined. So there is nothing I can do with this reader. ☹
There is a member type value_type, that does have data, size and more mebers, but I have no idea, how I could et hold of a variable of this type containing a part of my message body.

The error message might indicate, that I need to provide a fixed size buffer to the buffer_body, but I have no idea, how to do this.

I tried to add the following code to do_read():

             req_.body().data = my_buffer;
             req_.body().size = sizeof(my_buffer);

It compiles, so req_body() seems to be the way to get hold on a http::buffer_body::value_type, but simply initializing the two variables does not seem to be sufficient. I also tried to set req_.body().more to false or to true, but it didn’t change anything.

I then thought, I might try a dynamic_body. This would store the entire upload in memory, but perhaps I get more familiar with the beat framework, when I try to use this body.

And in deed, I managed to write some code, that shows all request headers and the request body on stdout. To get the body, I had to copy the buffer contents into a std::string:

             std::string body{ boost::asio::buffers_begin(req.body().data()),
                               boost::asio::buffers_end(req.body().data()) };

Since large files have to be handled by my code, this additional copy of the data is to be avoided. So I tried to iterate the body using the iterators:

             const auto begin = boost::asio::buffers_begin(req.body().data());
             const auto end   = boost::asio::buffers_end(req.body().data());
             for (auto it = begin; it != end; it++)
                    putchar(*it);

While it runs fine on the windows box, the loop crashes on linux in the 512th iteration in it++.

Program received signal SIGSEGV, Segmentation fault.
0x0000555555592432 in boost::intrusive::list_node_traits<void*>::get_previous (n=@0x7fffffffc118: 0x1ff) at ../boost/boost/intrusive/detail/list_node.hpp:54
54         {  return n->prev_;  }
(gdb) print n
$1 = (const boost::intrusive::list_node_traits<void*>::node_ptr &) @0x7fffffffc118: 0x1ff
(gdb)

Here is the call stack:
#0  0x0000555555592432 in boost::intrusive::list_node_traits<void*>::get_previous (n=@0x7fffffffc118: 0x1ff) at ../boost/boost/intrusive/detail/list_node.hpp:54
#1  boost::intrusive::list_iterator<boost::intrusive::bhtraits<boost::beast::basic_multi_buffer<std::allocator<char> >::element, boost::intrusive::list_node_traits<void*>, (boost::intrusive::link_mode_type)0, boost::intrusive::dft_tag, 1u>, true>::operator-- (this=0x7fffffffc118) at ../boost/boost/intrusive/detail/list_iterator.hpp:107
#2  std::__advance<boost::intrusive::list_iterator<boost::intrusive::bhtraits<boost::beast::basic_multi_buffer<std::allocator<char> >::element, boost::intrusive::list_node_traits<void*>, (boost::intrusive::link_mode_type)0, boost::intrusive::dft_tag, 1u>, true>, long> (__i=..., __n=0) at /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/include/g++-v10/bits/stl_iterator_base_funcs.h:169
#3  0x000055555558daf0 in std::advance<boost::intrusive::list_iterator<boost::intrusive::bhtraits<boost::beast::basic_multi_buffer<std::allocator<char> >::element, boost::intrusive::list_node_traits<void*>, (boost::intrusive::link_mode_type)0, boost::intrusive::dft_tag, 1u>, true>, long> (__i=..., __n=-1) at /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/include/g++-v10/bits/stl_iterator_base_funcs.h:206
#4  0x0000555555589254 in std::prev<boost::intrusive::list_iterator<boost::intrusive::bhtraits<boost::beast::basic_multi_buffer<std::allocator<char> >::element, boost::intrusive::list_node_traits<void*>, (boost::intrusive::link_mode_type)0, boost::intrusive::dft_tag, 1u>, true> > (__x=..., __n=1) at /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/include/g++-v10/bits/stl_iterator_base_funcs.h:230
#5  0x0000555555589608 in boost::beast::basic_multi_buffer<std::allocator<char> >::subrange<true>::const_iterator::operator* (this=0x7fffffffc4f8) at ../boost/boost/beast/core/impl/multi_buffer.hpp:384
#6  0x0000555555589b4b in boost::asio::buffers_iterator<boost::beast::basic_multi_buffer<std::allocator<char> >::subrange<true>, char>::increment (this=0x7fffffffc4d0)
    at ../boost/boost/asio/buffers_iterator.hpp:370
#7  0x0000555555584894 in boost::asio::buffers_iterator<boost::beast::basic_multi_buffer<std::allocator<char> >::subrange<true>, char>::operator++ (this=0x7fffffffc4d0)
    at ../boost/boost/asio/buffers_iterator.hpp:229
#8  0x000055555557e7c0 in boost::asio::buffers_iterator<boost::beast::basic_multi_buffer<std::allocator<char> >::subrange<true>, char>::operator++ (this=0x7fffffffc4d0)
    at ../boost/boost/asio/buffers_iterator.hpp:237
#9  0x0000555555579ba8 in handle_request<boost::beast::http::basic_dynamic_body<boost::beast::basic_multi_buffer<std::allocator<char> > >, std::allocator<char>, session::send_lambda&> (doc_root=...,
    req=..., send=...) at t.cpp:164

I hope anybody can give my a hint on how to use this beast.

Right now it feels that it might be easier to write the http server using plain boost::asio.


Klebsch Mario
Funktion | R&D



Tel: [phone]
Raum: Braunschweig, E20



Diese E-Mail und die an sie angehängten Dateien sind ausschließlich für Personen oder Institutionen bestimmt, deren Namen oben aufgeführt sind. Sie können Informationen enthalten, die durch das Berufsgeheimnis geschützt sind und deren Weitergabe strengstens untersagt ist. Jede elektronische Nachricht kann geändert werden. ACTIA lehnt jede Verantwortung für diese Nachricht ab. Der Inhalt dieser Nachricht stellt keine Verpflichtung seitens unseres Unternehmens dar. Wenn Sie kein Empfänger sind, weisen wir Sie darauf hin, dass das Lesen, Vervielfältigen oder Verteilen strengstens untersagt ist. Wir bitten Sie daher, uns umgehend über diesen Brief zu informieren und diese Nachricht sowie eventuell beigefügte Unterlagen aus Ihrem Postfach zu löschen. Danke.

This e-mail and the files attached to it are intended exclusively for persons or institutions whose names are listed above. They may contain information that is protected by professional secrecy and the disclosure of which is strictly prohibited. Any electronic message can be modified. ACTIA declines all responsibility for this message. The content of this message does not represent a commitment on the part of our company. If you are not a recipient, we would like to point out that reading, copying or distribution is strictly prohibited. We therefore ask you to inform us immediately about this letter and to delete this message and any accompanying documents from your mailbox. Thank you.