Re: [Boost-users] booser::serialization compilation times

From: Andreas Sæbjørnsen (andreas.saebjoernsen_at_[hidden])
Date: 2008-07-10 18:42:38

I wanted to give you an update on the effect of the changes you
suggested. I turned off debugging as you suggested
so that no debug symbols are generated and that
      - reduced the binary to 17MB when compiling with "-O3".
      - the compilation time was also reduced to 17 minutes.
      - the number of symbols in the object file was reduces to 46707.
      - binary becomes 29 MB
I also used
      BOOST_CLASS_IMPLEMENTATION(pixel,
boost::serialization::object_serializable);
that does not seem to reduce compilation time. Most of the symbols are
weak symbols on the form
    0000000000000000 V
_ZTVN5boost7archive6detail19pointer_iserializerINS0_15binary_iarchiveE12SgUpcThreadsEE
or weak symbols with default values
   0000000000000000 W
_ZNK5boost7archive6detail11oserializerINS0_15binary_oarchiveE19SgAsmTypeDoubleWordE14is_polymorphicEv

But 17 minutes and 1.2 GB of compiler memory usage is still not great.
Is there any other common techniques used to reduce compilation
time when using template meta-programming that you are aware of?

thanks,
Andreas

On Wed, Jul 9, 2008 at 10:48 PM, Robert Ramey <ramey_at_[hidden]> wrote:
> a) Try buld for release.  This should drop the binary down to a reasonable
> size.
> If it doesn't that would be interesting to know.
>
> b) Don't use inline code for serialization definitions.
>
> c) Separate the serialization code into separate modules so they
> will only need to be compiled when the class definition changes.
>
> d) Consider creating as many as one module per class.
>
> e) Consider making a library of all the modules containing your
> serialization code.  In this way, even if you make a new applicaiton,
> the serialization code won't have to be recompiled.
>
> f) Consider using a polymorphic archive (in a library as above).
> In this way, one compiliation will server for ALL archive
> classes.
>
> g) Review the demo_pimpl.cpp for an example on how to do this.
>
> The result of the above will be that only those modules whose
> header is changed, will need to be compiled.
>
> That is, the compilations will take just as long, but should be
> necessary far less frequently.
>
> This approach is useful for all large applications.
>
> You might not like this idea - but you might want to consider
> using MS VC as compiler.  Its much faster than GCC.  Making
> your code conformant with both is very little effort.
>
> Robert Ramey
>
> Andreas Sæbjørnsen wrote:
>> We are a team that develops a C/C++/Fortran/Binary  compiler project
>> called ROSE and we want to serialize the Abstract Syntax Tree (AST).
>> This AST has 436 different custom class types and uses maps, sets,
>> vectors, lists and hash maps. These classes have about 1229 different
>> variables. For convenience reasons we wanted to serialize the AST
>> using boost::serialize, but we have some compiler performance
>> problems. Currently the compilation takes  (GCC 4.1.2)
>>     -48 minutes to compile and link a program that takes seconds to
>> compile without serialization
>>     -The resulting binary goes from <1MB to 122 MB in size.
>>     -The compilation uses 2 GB of RAM
>>     -The object file contains 98075 symbols with serialization and 41
>> without
>>
>> If the templates for loading the AST is not instantiated the
>> instantiated templates for saving the AST takes about
>>     - 19 minutes to compile and link
>>     - The resulting binary goes from <1MB to 77MB
>>     - The compilation uses 1.1GB of RAM
>>
>> We have our own custom serialization mechanism that is not that easy
>> to support, but it does not visibly increa
>>
>> My machine is a 2.66 Ghz Quad Core Xeon 5355, 16 GB RAM and 2 TB of
>> striped storage.
>>
>> Is there any trick I can use to reduce the compilation time? When the
>> compilation time is 48 minutes boost::serialization is unfortunately
>> not an option for us although it is a great tool at runtime.
>>
>> thanks,
>> Andreas
>
>
>
>
> _______________________________________________
> Boost-users mailing list
> Boost-users_at_[hidden]
> http://lists.boost.org/mailman/listinfo.cgi/boost-users
>