Re: [boost] [proto] Looong compile times and other issues

Subject: Re: [boost] [proto] Looong compile times and other issues
From: John Maddock (boost.regex_at_[hidden])
Date: 2011-09-12 04:42:49

----- Original Message ----- 
From: "Eric Niebler" <eric_at_[hidden]>
To: <boost_at_[hidden]>
Sent: Sunday, September 11, 2011 8:11 PM
Subject: Re: [boost] [proto] Looong compile times and other issues

On 9/11/2011 1:44 PM, Joel Falcou wrote:
> Le 11/09/2011 19:18, John Maddock a écrit :
>> So.... I thought I had all the expression template stuff pretty much
>> done, and then I tried some real world use cases (compiling all the
>> students_t distribution functions with my extended-precision FP type)
>> and everything fell apart:
>>
>> * VC10 wouldn't compile the code at all - more or less runs the system
>> out of swap space (takes about 10 minutes or more!), then exits with an
>> internal compiler error (if I break the code down into it's parts and
>> instantiate each part separately it does compile though - that's not a
>> solution though!).
>> * GCC-4.4.x fails to compile the code due to clashes between
>> boost::math::complement (a function) and boost::proto::complement (a
>> class). I suspect this is an old gcc bug (finding structures via ADL) -
>> I guess the solution is to not derive my number type from a proto-type
>> so ADL can't find proto:: classes? Or will I hit this from some other
>> unforeseen lookup?

>This is not a gcc bug. At least, the gcc developers would say so. There
>is some ambiguity in the standard about whether ADL should find classes
>in addition to functions -- despite of the fact that finding classes is
>never what the user wants.

Oh :-(

>John, where does complement show up unqualified in your code?

It's in my concept-checking code, the actual (abreviated) error message is:

t.cpp:13:   instantiated from here
../../../../trunk/boost/proto/tags.hpp:30: error: 'struct 
boost::proto::tag::complement' is not a function,
../../../../trunk/boost/math/distributions/complement.hpp:156: error: 
conflict with 'template<class Dist, class RealType> 
boost::math::complemented2_type<Dist, RealType> 
boost::math::complement(const Dist&, const RealType&)'
../../../../trunk/boost/math/concepts/distributions.hpp:132: error:   in 
call to 'complement'
../../../../trunk/boost/proto/proto_fwd.hpp:488: error: 'template<class T> 
struct boost::proto::complement' is not a function,
../../../../trunk/boost/math/distributions/complement.hpp:156: error: 
conflict with 'template<class Dist, class RealType> 
boost::math::complemented2_type<Dist, RealType> 
boost::math::complement(const Dist&, const RealType&)'
../../../../trunk/boost/math/concepts/distributions.hpp:132: error:   in 
call to 'complement'

The point is that boost::math::complement is a documented public interface 
so:

using namespace boost::math;

real_type a = cdf(complement(distribution_type, real_type));

Is going to be common place code - so I really don't want these kinds of 
ambiguities coming up - not least because the error messages are 
sufficiently cryptic that it's not immediately obvious what the issue is.

>> * GCC-4.5.0 Fails with an internal compiler error :-(
>> * GCC-4.6.0 Builds the code OK, but takes a long time - though possibly
>> just barely acceptable.
>>
>> I suspect this is a "triple template" problem:
>>
>> * proto is complex template library.
>> * my number class is a fairly large/complex template in it's own right.
>> * the code above gets instantiated from deep within Boost.Math's
>> internals.
>>
>> Other than using proto::switch in the grammar (which I'm doing already),
>> is there anything I can do to reduce the template load on the proto side
>> of things?
>>
>> Otherwise since I'm only using a tiny fraction of proto's capabilities,
>> the only other option I can see is to rip it out and replace with a
>> mini-proto designed to minimalize template instantiations within this
>> particular use case.
>
> What kind of transform do you use ? I foudn out that colalpsing complex
> trasnform into primitive one eased the problem.

>Right. The nice transform syntax that allows composition via function
>types is rather expensive at compile time. It's great for knocking
>together a DSEL quickly, but once you have it all working, you can
>replace them one by one with the equivalent primitive transforms, like
>Joel suggests.

Ummm, well I confess I'm not using any transforms - just the expression 
template creation - then I unpick the expression myself.  Partly that 
allowed me to get things going quickly, and partly it's allowed me to do all 
kinds of interesting things with the expression.  The actual code that 
unpicks the expression is somewhat verbose, but not all that complex (not 
too much template depth... I hope).

> Next to that :
>  - improve CT through forward declaration. proto is a bit woobey on this
> side and I think it may help if we split proto files in more grain
> fashion

>Joel, can you give an example of where this might help>

> - fusion is also a big hitter : lots of PP and lots of forced
> instanciation instead of lazy specialization.

>I avoid fusion, sadly. Proto is not built on top of fusion for just this
>reason. Proto is expensive enough all by itself. :-P

> As for the ADL, can your terminals be made PODs ? We can see if adding
> ADL barriers in proto helps

>Proto already has ADL barriers. The expression types are in
>boost::proto::exprns_, for instance. I would have to see the code to
>know what gcc is complaining about.

> For the ICE, is there any hints in the core dump ?

>There usually isn't, AFAIK.

Not for GCC, for MSVC I do get some kind of error location - but it's 
different for every problem case :-(

>John, if you can post your code publicly, perhaps one of the proto
>cognoscenti can help.

OK to reproduce you will need:

* Current trunk.
* The contents of sandbox/big_number.
* A copy of MPFR.

Then:

#include <boost/math/big_number/mpfr.hpp>
#include <boost/math/concepts/distributions.hpp>
#include <boost/math/distributions.hpp>

void foo()
{
   using namespace boost;
   using namespace boost::math;
   using namespace boost::math::concepts;
   function_requires<DistributionConcept<students_t_distribution<boost::math::mpfr_real_50> 
 > >();
}

If the MPFR dependency proves an issue, then I can try and produce a test 
case without it.

For those prefering to read code, the proto-dependent stuff is mostly all 
in:

boost/math/big_number/big_number_base.hpp  (grammar and meta programming).
boost/math/big_number.hpp (The actual terminal, and expression evaluation).

Thanks, John.