Re: [boost] [explore] Library Proposal: Container Streaming

Subject: Re: [boost] [explore] Library Proposal: Container Streaming
From: David Bergman (David.Bergman_at_[hidden])
Date: 2009-12-01 16:29:20

On Dec 1, 2009, at 1:57 PM, Vladimir Prus wrote:

> David Bergman wrote:
> 
>> On Dec 1, 2009, at 12:47 PM, Vladimir Prus wrote:
>> 
>>> Robert Ramey wrote:
>> 
>>>> I didn't have to write even one line to template code to
>>>> implement the example.
>>> 
>>> I though boost.serialization was written by you?
>> 
>> The example is quite distinct from the library being used...
>> 
>> In other words, there is a special quality of Boost.Serialization that is of importance: it
>> exists. So, we are allowed to use it, and by using it, we have to write very little code
>> ourselves, beside a number of include statements. Great!
> 
> If this new proposed library is accepted, you get to write just as little code, if not less.

There are not that many lexemes left, especially if one does a "typedef boost::archive::text_oarchive arch;", i.e., it is hard to beat

        arch(std::cout) << v;

or (by using my suggested wrapper function)

        output(v);

Granted, you would have to define such typedef (one line) or wrapper (five lines) somewhere, but the main concern is the various locations of invocation, no?

>>> And that code still has
>>> to be parsed by the compiler.
>> 
>> This is a different issue. Is that your problem, that you think compilers have a problem with some
>> complex constructs inside the Boost.Serialization library? Does the compiler fail? Consume too
>> much resources? Too much time?
> 
> Yes, too much time and resources.

How much time? I am not asking to be cute, but just curious as to how much extra building time is needed for Boost.Serialization. I have used it heavily in a lot of projects, and have not been disturbed by it.

When you say resources, do you mean that compiler (or linker) use a lot of memory to handle Boost.Serialization?

> When building debug variant, an program that does nothing
> but creates std::vector<int> is a 53K object file. When I add serialization, I get:
> 
> - a useless warning shown below, and no, I am not willing to apply const_cast or declare things
>  const just because Boost.Serialization has strange position about what const really means

I did not follow you here; could you explain what that strange position is?

> - a 387K object file

Let me check this. I am using -O2 on a Snow Leopard box with Apple's GCC 4.2.1, with the demo file. With *no* inclusion of Boost.Serialization headers nor reference to that library, I get a 10kB executable (basically a simple Hello World app with some vector manipulations). I append the source code at the end of this post for self-containment purposes :-)

Let us introduce Boost.Serialization, and see what happens with a varying number of such invocation locations (log << vi) in our single demo file, compiling to an executable:

no Boost.Serialization: 10kB
N=0 (i.e., just including Boost.Serialization headers and linking with library): 27kB
N=1: 51kB
N=2: 55kB
N=10: 55kB
N=100: 55kB
N=500: 65kB
N=1000: 73kB
N=4000: 118k

Performing a mental linear regression, and using N=1 as a fixed constant, we get a linear function in extra (disk) space of

        41kB + N * 20B

Note that this extra space is relative code containing no Boost references at all.

So, basically 20 bytes for each such invocation location. If you think that is strenuous, you could always wrap the invocation in a function (and force it to not be inlined), in which case you get a constant space addition of 41kB.

NOTA BENE: more than a few hundred invocation locations in a project seems simply weird.

> When doing a release build, the source with serialization takes 1.3 second to compile, while
> the empty one takes 0.3 second.

Here you have a better case, methinks. Let us do the similar tests as above, but now for compilation+linking time (linking time is much lower than compilation time...), where I measure the user + system time spent, which should be fairer for extrapolations to bigger builds (and my concurrent threads running on my box...):

no Boost.Serialization: 0.45s
N=0: 1.59s
N=1: 1.59s
N=2: 1.62s
N=10: 1.62s
N=100: 1.85s
N=500: 2.68s
N=1000: 4.02s
N=4000: 13.52s

Again fixating the constant at N=1, and performing some mental linear regression again, we get a linear function in extra build time of

        1.14s + N * 3.4ms

The biggie here is of course the constant, of an extra 1.14s, as you also reported. But note that it is relative *no* Boost use at all.

When I wrap the output in the template function 'output' shown in the bottom of this post, and force that function not to be inlined (actually using a function pointer), I get

N=1: 1.64s
N=1000: 2.11s
N=4000: 4.15s

Using mental linear regression again, we get:

        1.19s + N * 1.2ms

NOTA BENE: I used no precompiled headers or such "cheating" which is the norm for bigger projects :-)

> It does not seem like this overhead is justifiable for a trivial task of just getting my
> std::vector printed. 

I agree, after measuring and being somewhat surprised by that large constant. But, again, it is relative no Boost use at all, and no precompiled headers or other compiler/linker tricks. In my regular code, the addition of Boost.Serialization does not add much compilation time. After all, it is just a few ms per invocation location.

> - Volodya
> 
> ../../../boost/mpl/print.hpp: In instantiation of 
> ‘boost::mpl::print<boost::serialization::STATIC_WARNING_LINE<98> >’:
> ../../../boost/serialization/static_warning.hpp:92:   instantiated from 
> ‘boost::serialization::static_warning_test<false, 98>’
> ../../../boost/archive/detail/check.hpp:98:   instantiated from ‘void 
> boost::archive::detail::check_object_tracking() [with T = std::vector<int, std::allocator<int> >]’
> ../../../boost/archive/detail/oserializer.hpp:313:   instantiated from ‘static void 
> boost::archive::detail::save_non_pointer_type<Archive>::invoke(Archive&, T&) [with T = 
> std::vector<int, std::allocator<int> >, Archive = boost::archive::text_oarchive]’
> ../../../boost/archive/detail/oserializer.hpp:525:   instantiated from ‘void 
> boost::archive::save(Archive&, T&) [with Archive = boost::archive::text_oarchive, T = 
> std::vector<int, std::allocator<int> >]’
> ../../../boost/archive/detail/common_oarchive.hpp:69:   instantiated from ‘void 
> boost::archive::detail::common_oarchive<Archive>::save_override(T&, int) [with T = std::vector<int, 
> std::allocator<int> >, Archive = boost::archive::text_oarchive]’
> ../../../boost/archive/basic_text_oarchive.hpp:80:   instantiated from ‘void 
> boost::archive::basic_text_oarchive<Archive>::save_override(T&, int) [with T = std::vector<int, 
> std::allocator<int> >, Archive = boost::archive::text_oarchive]’
> ../../../boost/archive/detail/interface_oarchive.hpp:64:   instantiated from ‘Archive& 
> boost::archive::detail::interface_oarchive<Archive>::operator<<(T&) [with T = std::vector<int, 
> std::allocator<int> >, Archive = boost::archive::text_oarchive]’
> d.cpp:13:   instantiated from here
> ../../../boost/mpl/print.hpp:55: warning: comparison between signed and unsigned integer expressions
> <d.cpp><d_empty.cpp>_______________________________________________
> Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

I have not seen this before.

/David

--------------
My test code:

#include <iostream>
#include <vector>
#ifndef SKIP_SER
#include <boost/serialization/vector.hpp>
#include <boost/serialization/map.hpp>
#include <boost/serialization/list.hpp>
#include <boost/serialization/string.hpp>
//#include <boost/date_time.hpp>
#include <boost/archive/text_oarchive.hpp>

typedef boost::archive::text_oarchive arch;

template<typename Cont>
std::ostream& output(Cont const& cont, std::ostream & stream = std::cout) {
  boost::archive::text_oarchive(stream) << cont;
  return stream;
}

#endif

int main(int argc, char * argv[]) {
#ifndef SKIP_SER
  boost::archive::text_oarchive output_log(std::cout);
#endif
  // simple vector example
  std::vector<int> vi;
  vi.push_back(1);
  vi.push_back(2);
  vi.push_back(3);
#ifdef SKIP_SER
  std::cout << "Hello!";
#endif
  
  // A Ruby script of mine generates a number of invocations as this (I did not want to use Boost.PP for this)
  output_log << vi;

}