Thanks, I will try this. However, being the inquisitive type, can you explain to me what the problem is? Cheers, Tom -----Original Message----- From: boost-users-bounces@lists.boost.org [mailto:boost-users-bounces@lists.boost.org] On Behalf Of Mathieu Malaterre Sent: 05 October 2009 15:28 To: boost-users@lists.boost.org Subject: Re: [Boost-users] Use of ccache with Boost On Mon, Oct 5, 2009 at 10:45 AM, Dalton, Tom <tom.dalton@eds.com> wrote:

Further to my questions from last week:

I've tried both methods - first I tried changing the toolset to make the gcc toolset use "ccache g++" as the compiler executable. I had some problems with that, so I tried the symlink approach suggested by Roland. This is also failing, with some seemingly very subtle bug that I can't

suss out.

Use cmake. apt-get install ccache cd /path/to/boost mkdir bincache cd bincache export CC="ccache gcc" export CXX="ccache g++" cmake .. make -j8 cheers -- Mathieu _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

I am working on a toolchain that lives outside of the system path. All the tools (gcc, ccache, cmake, valgrind etc) live in /some/random/path, with their own gcc/gcc-3.4.6 or ccache/ccache-2.4 directories under that. Within those directories, the packages binary install is under a directory like ccache-2.4-bin So the full path to g++ in this case is /some/random/path/gcc/gcc-4.4.1/gcc-4.4.1-bin/bin/g++ The build system (ant) then builds the tools with a controlled environment (e.g. by setting up PATH, LD_LIBRARY_PATH etc.). This approach works fine fro the tools built with autoconf/automake and the tools that are binary distributions (e.g. java jdk), and works fine with boost as long as I'm not trying to be clever. In /some/random/path/ccache/ccache-2.4/ccache-2.4-bin/bin I created symlinks to ccache names gcc and g++. I then set the PATH to run the boost build to be /some/random/path/ccache/ccache-2.4/ccache-2.4-bin/bin:/some/random/path /gcc/gcc-4.4.1/gcc-4.4.1-bin/bin:{rest of the system path} As stated, this approach works for all the other tools, but causes the boost build to diverge as described below and I don't understand why. I also tried adding symlinks named cpp and c++ but the behaviour described below didn't change... Thanks for your help, Tom -----Original Message----- From: boost-users-bounces@lists.boost.org [mailto:boost-users-bounces@lists.boost.org] On Behalf Of Roland Bock Sent: 05 October 2009 18:13 To: boost-users@lists.boost.org Subject: Re: [Boost-users] Use of ccache with Boost Dalton, Tom wrote:

Further to my questions from last week:

I've tried both methods - first I tried changing the toolset to make the gcc toolset use "ccache g++" as the compiler executable. I had some problems with that, so I tried the symlink approach suggested by Roland. This is also failing, with some seemingly very subtle bug that I can't

suss out.

The build progresses a little way, until it gets to libs/program_options/src/cmdline.cpp

When g++ isn't a symlink to ccache, I get the following output as shown by the attachment build.log.1.fragment. When I have gcc & g++ symlinks in the ccache bin dir pointing to ccache (the PATH is ccache/bin:real_gcc/bin), the output is like the second attachment and the file fails.

Hmm? What symlinks do you have in the ccache folder? I only required symlinks in /usr/local/bin, as described here http://ccache.samba.org/ccache-man.html I am using 4.2.4 and boost compiles with no problems at all. In daily life, I am using CMake, like Mathieu, but direct calls to g++ or compilation with configure&make never failed on me. Regards, Roland

The build logs diverge from that point onwards as more and more stuff either fails, or is skipped due to previous failures.

There are 2 issues that I can see from those logs. Firstly the line numbering on the .hpp includes is one-out on the ccache-enabled build. Does this mean the preprocessor is behaving strangely? Secondly, on line 15 of the failing fragment file, it looks like part of the output is cut off (at the end of the line, it gets "boost::_bi::list2<boost::_bi::v0.000008 sec system; 0.000164 sec

user".

Can anyone help me? I don't know what would cause either of these issues.

Tom

PS Using gcc 4.4.1, ccache 2.4 and tried boost 1.39.0 and 1.40.0, with

the same results.

-----Original Message----- From: boost-users-bounces@lists.boost.org [mailto:boost-users-bounces@lists.boost.org] On Behalf Of Dalton, Tom Sent: 29 September 2009 16:20 To: boost-users@lists.boost.org Subject: Re: [Boost-users] Use of ccache with Boost

Daniel and Roland,

Thanks guys, that gives me some good avenues to explore.

Tom

-----Original Message----- From: boost-users-bounces@lists.boost.org [mailto:boost-users-bounces@lists.boost.org] On Behalf Of Daniel James Sent: 29 September 2009 16:07 To: boost-users@lists.boost.org Subject: Re: [Boost-users] Use of ccache with Boost

2009/9/29 Dalton, Tom <tom.dalton@eds.com>:

...

I'm using Boost on Linux however for various reasons I need to rebuild

...

parts of it occasionally. For other C/C++-based tools that are built with autoconf/automake, I've started to use ccache (http://ccache.samba.org/) to speed up recompilation. This is done simply by setting the CC / CXX environment variables to "ccache gcc" /

...

"ccache g++" respectively. I have had a look through the Jam documentation but I can't see an easy way to achieve the same thing when building the Boost distribution. The closest I came was a bit on

...

the trac that simply said I can add a new toolset, but had no real instructions as to how.

Create a file called 'user-config.jam' in your home directory, containing something like:

using gcc : : ccache g++ ;

Or possibly with the version number:

using gcc : 4.4 : ccache g++-4.4 ;

For more information see:

http://www.boost.org/doc/libs/1_40_0/doc/html/bbv2/advanced.html#bbv2. ad vanced.configuration

Daniel _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

---------------------------------------------------------------------- --

_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

Thanks a lot for your help Roland, these are all good points. A) ccache is meant to handle compiler changes as the cached objects hash is computed using the compiler binary size and modification time. This alone should be enough to avoid hash collisions between the same object built with a different compiler, but it also used the pre-processed source as part of the hash function input. Furthermore, the main part of my toolchain is built with gcc-4.4.1, it's just some legacy, separated components that use 3.4.6. Also, I've tried cleaning the cache out using ccache and manually removing the dir before starting the build. So good thought but I don't think this is the problem. B) Seems unlikely, but possible. If this was the case, I would expect it not to be a conflict between my versions of gcc, but between the gcc I am using (gcc-4.4.1 in this case) and the compiler in the underlying OS (Red Hat EL 4.6 in this case, which uses a ?custom? 3.4.6 compiler that identifies itself as "Red Hat 3.4.6-9"). However, if this were the case then I'd expect the problem to occur even with non-ccache enabled builds. I think this is worth looking at as a last resort, but I'm not convinced it's the issue. C) ?? I think it's this one ;-) I think the big pointer is still looking at the line number discrepancy in the .hpp files in that error. In the non-ccache build the .hpp line numbers are different to the ccache-enabled one. The headers are part of the original distribution and are not modified by the build (as far as I can tell), but maybe the pre-processor or some preliminary stage of compilation is doing something strange to them? I'm not sure how I can go about seeing what's actually being used at that stage though. Ideally, we need some way to see all the inputs to the compiler at both these stages (ccache and non-ccache) to confirm that they are the same. I don't know if the jam -dX argument can be used to display enough information, but then that leads to the issue seen in the build fragment 2 file, on line 16, that the full stdout/stderr output from the compiler is not actually visible, but is truncated by the "v0.000008 sec system; 0.000164 sec user" string. Is it possible boost's build system isn't allocating enough space for this output? Or maybe there is a problem with the way I am recording that build log - I do "ant 2>&1 | tee build.log". Maybe tee is getting confused by the 2 streams into 1 and truncating... I will do some more investigation in the next day or two. In the meantime I have worked around the problem by building boost without ccache in the path and everything else with. But my brain needs to know why this happens, especially as a bug in ccache would be quite serious! Cheers, Tom -----Original Message----- From: boost-users-bounces@lists.boost.org [mailto:boost-users-bounces@lists.boost.org] On Behalf Of Roland Bock Sent: 06 October 2009 12:29 To: boost-users@lists.boost.org Subject: Re: [Boost-users] Use of ccache with Boost Tom, ok, understood. I have two ideas left, based on the fact that you seem to have several versions of gcc (3.4.6 and 4.4.1 at least). A) Switching: ============= I could imagine that switching compilers is not handled very well by ccache. So you might need to clear the cache before switching from 3.4.6 to 4.4.1 or vice versa (or change the CCACHE_DIR environment variable). B) Mixup: ========= Is it possible that some parts of your project get compiled by one version, others by another? Rather unlikely, but if A does not yield anything... You could use ps or similar to monitor which instance of gcc is actually getting called. For instance, when I call "ps ex" while compiling, I get information like /usr/lib/gcc/x86_64-linux-gnu/4.2.4/cc1plus -fpreprocessed /home/rbock/.ccache/CreateArti.tmp.rbock.metafeed.de.30536.ii -quiet -dumpbase CreateArti[...] You could call ps repeatedly and analyze the output for inconsistencies. C) ??: ====== If the ps-log looks clean and clearing the cache does not help, I'll be out of ideas, I'm afraid. Maybe the ccache authors can help in that case? I'd be interested to hear if you figure it out, btw :-) Regards, Roland Dalton, Tom wrote:

I am working on a toolchain that lives outside of the system path. All the tools (gcc, ccache, cmake, valgrind etc) live in /some/random/path, with their own gcc/gcc-3.4.6 or ccache/ccache-2.4 directories under that. Within those directories, the packages binary install is under a directory like ccache-2.4-bin

So the full path to g++ in this case is /some/random/path/gcc/gcc-4.4.1/gcc-4.4.1-bin/bin/g++

The build system (ant) then builds the tools with a controlled environment (e.g. by setting up PATH, LD_LIBRARY_PATH etc.). This approach works fine fro the tools built with autoconf/automake and the tools that are binary distributions (e.g. java jdk), and works fine with boost as long as I'm not trying to be clever.

In /some/random/path/ccache/ccache-2.4/ccache-2.4-bin/bin I created symlinks to ccache names gcc and g++. I then set the PATH to run the boost build to be

/some/random/path/ccache/ccache-2.4/ccache-2.4-bin/bin:/some/random/path

/gcc/gcc-4.4.1/gcc-4.4.1-bin/bin:{rest of the system path}

As stated, this approach works for all the other tools, but causes the boost build to diverge as described below and I don't understand why. I also tried adding symlinks named cpp and c++ but the behaviour described below didn't change...

Thanks for your help,

Tom

-----Original Message----- From: boost-users-bounces@lists.boost.org [mailto:boost-users-bounces@lists.boost.org] On Behalf Of Roland Bock Sent: 05 October 2009 18:13 To: boost-users@lists.boost.org Subject: Re: [Boost-users] Use of ccache with Boost

Dalton, Tom wrote:

...

Further to my questions from last week:

I've tried both methods - first I tried changing the toolset to make the gcc toolset use "ccache g++" as the compiler executable. I had some problems with that, so I tried the symlink approach suggested by Roland. This is also failing, with some seemingly very subtle bug that I can't

...

suss out.

The build progresses a little way, until it gets to libs/program_options/src/cmdline.cpp

When g++ isn't a symlink to ccache, I get the following output as shown by the attachment build.log.1.fragment. When I have gcc & g++ symlinks in the ccache bin dir pointing to ccache (the PATH is ccache/bin:real_gcc/bin), the output is like the second attachment and the file fails.

Hmm? What symlinks do you have in the ccache folder? I only required symlinks in /usr/local/bin, as described here http://ccache.samba.org/ccache-man.html

I am using 4.2.4 and boost compiles with no problems at all. In daily life, I am using CMake, like Mathieu, but direct calls to g++ or compilation with configure&make never failed on me.

Regards,

Roland

...

The build logs diverge from that point onwards as more and more stuff

...

either fails, or is skipped due to previous failures.

There are 2 issues that I can see from those logs. Firstly the line numbering on the .hpp includes is one-out on the ccache-enabled build. Does this mean the preprocessor is behaving strangely? Secondly, on line 15 of the failing fragment file, it looks like part of the output is cut off (at the end of the line, it gets "boost::_bi::list2<boost::_bi::v0.000008 sec system; 0.000164 sec user". Can anyone help me? I don't know what would cause either of these issues.

Tom

PS Using gcc 4.4.1, ccache 2.4 and tried boost 1.39.0 and 1.40.0, with

...

the same results.

-----Original Message----- From: boost-users-bounces@lists.boost.org [mailto:boost-users-bounces@lists.boost.org] On Behalf Of Dalton, Tom Sent: 29 September 2009 16:20 To: boost-users@lists.boost.org Subject: Re: [Boost-users] Use of ccache with Boost

Daniel and Roland,

Thanks guys, that gives me some good avenues to explore.

Tom

-----Original Message----- From: boost-users-bounces@lists.boost.org [mailto:boost-users-bounces@lists.boost.org] On Behalf Of Daniel James Sent: 29 September 2009 16:07 To: boost-users@lists.boost.org Subject: Re: [Boost-users] Use of ccache with Boost

2009/9/29 Dalton, Tom <tom.dalton@eds.com>:

...

I'm using Boost on Linux however for various reasons I need to rebuild parts of it occasionally. For other C/C++-based tools that are built

...

...

with autoconf/automake, I've started to use ccache (http://ccache.samba.org/) to speed up recompilation. This is done simply by setting the CC / CXX environment variables to "ccache gcc"

...

...

/ "ccache g++" respectively. I have had a look through the Jam documentation but I can't see an easy way to achieve the same thing when building the Boost distribution. The closest I came was a bit on

...

...

the trac that simply said I can add a new toolset, but had no real instructions as to how.

Create a file called 'user-config.jam' in your home directory, containing something like:

using gcc : : ccache g++ ;

Or possibly with the version number:

using gcc : 4.4 : ccache g++-4.4 ;

For more information see:

http://www.boost.org/doc/libs/1_40_0/doc/html/bbv2/advanced.html#bbv2.

...

ad vanced.configuration

Daniel _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

...

--

_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

---

Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

Tom, out of curiosity: have you figured it out? Regards, Roland Roland Bock wrote:

Dalton, Tom wrote:

...

Thanks a lot for your help Roland, these are all good points.

You're welcome, sorry to hear they are all not helping :-(

Best of luck and endurance with this problem! Reading this thread, I am convinced you'll find the root to it :-)

Regards,

Roland

PS: I am afraid I can't help with jam, ant or tee _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users