Re: [Boost-users] Using boost::serialization in real-time without allocating memory

Subject: Re: [Boost-users] Using boost::serialization in real-time without allocating memory
From: Peter Soetens (peter.soetens_at_[hidden])
Date: 2009-09-17 09:42:01

On Thu, Sep 17, 2009 at 15:10, Peter Soetens <peter.soetens_at_[hidden]> wrote:
> Hi,
>
> I'm trying to find out if boost::serialization can be used in
> real-time applications to stream data into a fifo to another process.
> It is mandatory that no memory allocations happen during the
> serialization. I tested this with a std::vector<double> of 10 elements
> in combination with the boost::iostreams library.

My assumptions about the cause of the memory allocation were wrong. I
could trace it to this point:

#0  0x00007fdcbf4ed9a0 in operator new () from /usr/lib/libstdc++.so.6
#1  0x00007fdcbf76c984 in
std::_Rb_tree<boost::archive::detail::basic_oarchive_impl::cobject_type,
boost::archive::detail::basic_oarchive_impl::cobject_type,
std::_Identity<boost::archive::detail::basic_oarchive_impl::cobject_type>,
std:
:less<boost::archive::detail::basic_oarchive_impl::cobject_type>,
std::allocator<boost::archive::detail::basic_oarchive_impl::cobject_type>
>::_M_insert_ (this=0x94b3b8, __x=0x0, __p=0x94b3c0,
__v=@0x7fffc7bd0620)
    at /usr/include/c++/4.3/ext/new_allocator.h:92
#2  0x00007fdcbf76c734 in
boost::archive::detail::basic_oarchive::save_object
(this=0x7fffc7bd0830, x=0x7fffc7bd0860, bos=@0x611550) at
/usr/include/c++/4.3/bits/stl_tree.h:1148
#3  0x000000000040a5b5 in
boost::archive::detail::save_non_pointer_type<boost::archive::binary_oarchive,
std::vector<double, std::allocator<double> > >::save_standard::invoke
(ar=@0x7fffc7bd0830, t=@0x7fffc7bd0860)
    at /usr/include/boost/archive/detail/oserializer.hpp:231
#4  0x000000000040a5d4 in
boost::archive::detail::save_non_pointer_type<boost::archive::binary_oarchive,
std::vector<double, std::allocator<double> >
>::save_conditional::invoke (ar=@0x7fffc7bd0830, t=@0x7fffc7bd0860)
    at /usr/include/boost/archive/detail/oserializer.hpp:245
#5  0x000000000040a5f3 in
boost::archive::detail::save_non_pointer_type<boost::archive::binary_oarchive,
std::vector<double, std::allocator<double> > >::invoke
(ar=@0x7fffc7bd0830, t=@0x7fffc7bd0860)
    at /usr/include/boost/archive/detail/oserializer.hpp:294
#6  0x000000000040a612 in
boost::archive::save<boost::archive::binary_oarchive,
std::vector<double, std::allocator<double> > > (ar=@0x7fffc7bd0830,
t=@0x7fffc7bd0860) at
/usr/include/boost/archive/detail/oserializer.hpp:506
#7  0x000000000040a63c in
boost::archive::detail::common_oarchive<boost::archive::binary_oarchive>::save_override<std::vector<double,
std::allocator<double> > const> (this=0x7fffc7bd0830,
t=@0x7fffc7bd0860)
    at /usr/include/boost/archive/detail/common_oarchive.hpp:64
#8  0x000000000040a661 in
boost::archive::basic_binary_oarchive<boost::archive::binary_oarchive>::save_override<std::vector<double,
std::allocator<double> > > (this=0x7fffc7bd0830, t=@0x7fffc7bd0860,
version=0)
    at /usr/include/boost/archive/basic_binary_oarchive.hpp:63
#9  0x000000000040a689 in
boost::archive::binary_oarchive_impl<boost::archive::binary_oarchive,
char, std::char_traits<char> >::save_override<std::vector<double,
std::allocator<double> > > (this=0x7fffc7bd0830, t=@0x7fffc7bd0860)
    at /usr/include/boost/archive/binary_oarchive_impl.hpp:45
#10 0x000000000040a6b6 in
boost::archive::detail::interface_oarchive<boost::archive::binary_oarchive>::operator<<
<std::vector<double, std::allocator<double> > > (this=0x7fffc7bd0830,
t=@0x7fffc7bd0860)
    at /usr/include/boost/archive/detail/interface_oarchive.hpp:64
#11 0x0000000000404a65 in main (argc=1, argv=0x7fffc7bd0f08) at
serialize-rt-init.cpp:17

Which looks like std::map is being used to store class data (or so).
It looks like it's going to be much harder to work around this than I
assumed ?

Peter

>
> The aim is that all memory allocations happen during construction of
> the archive object, while the serialisation itself causes none.
>
> <code>
> #include <boost/archive/binary_oarchive.hpp>
> #include <boost/serialization/vector.hpp>
> #include <boost/iostreams/stream.hpp>
> #include <boost/iostreams/device/array.hpp>
>
> namespace io = boost::iostreams;
> int main(int argc, char *argv[])
> {
>  vector<double> d(10, 1.1);
>  char sink[1000];
>  memset( sink, 0, 1000);
>  io::stream<io::array_sink>  out(sink,1000);
>
>  boost::archive::binary_oarchive oa(out);
>  //oa << d; // should not allocate
>  return 0;
> }
> </code>
>
> The setup code does 10 memory allocations according to valgrind:
> ==12995== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 8 from 1)
> ==12995== malloc/free: in use at exit: 0 bytes in 0 blocks.
> ==12995== malloc/free: 10 allocs, 10 frees, 913 bytes allocated.
> ==12995== For counts of detected errors, rerun with: -v
> ==12995== All heap blocks were freed -- no leaks are possible.
>
> If we uncomment 'oa << d' we get 2 more (unwanted) allocations:
> ==13010== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 8 from 1)
> ==13010== malloc/free: in use at exit: 0 bytes in 0 blocks.
> ==13010== malloc/free: 12 allocs, 12 frees, 1,001 bytes allocated.
> ==13010== For counts of detected errors, rerun with: -v
> ==13010== All heap blocks were freed -- no leaks are possible
>
> I'm guessing that the 2 allocations in the serialisation path come
> from a temporay std::string object, when writing the
> 'serialization::archive' string into the archive. Wouldn't it be
> possible to rewrite this library code as such that there are no
> allocations / strings created ? A similar pattern is observed for
> deserializing: 2 allocs in the serialisation path.
>
> Peter
>
> PS: I tried to find out who defines the macro
> BOOST_ARCHIVE_SIGNATURE() but I couldn't find the definition/#define
> !?
>