|
|
Boost : |
From: troy d. straszheim (troy_at_[hidden])
Date: 2005-10-12 12:11:04
On Wed, Oct 12, 2005 at 10:14:34AM -0500, Rene Rivera wrote:
> troy d. straszheim wrote:
> > same code. The "before" picture:
> >
> > 326.17s user 153.82s system 92% cpu 8:40.03 total
> >
> > the first ~30 seconds of which is bjam calculating the dependencies
> > for each test module.
>
> To add to what Dave said... We've been personally bitten by assuming
> that bjam is slow in certain places. Only to humiliate ourselves after
> looking at the profile data :-)
Sorry, I had no idea the phrase "bjam calculating dependencies" was a
touchy subject. Please replace with "build system doing whatever it
does before the first compilation starts, as timed by looking at my
watch". I only understand the before/after picture to mean that bjam,
the compiler and linker are all being told to repeat work
unnecessarily; I see low-hanging fruit, and I want to pick and eat it.
:) I'm *not* taking (and won't take) veiled cheap shots.
I've pasted the full profiling info at the bottom of this mail. Be
interested to hear what y'all make of it.
> I'm assuming you deleted the results from the previous run, right?
Heh, yeah. :) I watched the compilations go by, and I double checked
that ccache was turned off.
> Yea... You just put all the sources into the bjam target. For example:
>
> run test1.cpp test2.cpp test3.cpp : : : std::locale-support : my_test :
> debug ;
I just tried this, and test1, test2, test3 each get compiled
separately. Here's the thing: the serialization unit tests each
precompile to roughly 50k lines, of which at most 200 actually differ
from test-to-test. When you then consider how much duplicated work
there is for the compiler and linker, after all the templates in those
49,800 lines are instanatiated... there's your speedup. But one can
just use the test_many.cpp which #includes all the other .cpps.
Whatever you guys are comfortable with.
> > 3) There are "demo" tests in the Jamfile, which are pulled in and
> > compiled as tests by #defining main to test_main and #including the
> > .cpp from the examples directory. This means these demos need
> >
> > <lib>../../test/build/boost_prog_exec_monitor
> >
> > not boost_unit_test_framework. I'm stumped on this one. How to
> > split up the tests into two groups, each of which takes different
> > libs? Should one toss these from the standard testing run
> > completely and just put a separate Jamfile over in examples/ ?
> > You could get the same code into both the tests and the examples
> > by taking the contents of main() out into some other #inlcuded
> > file, but then the examples would get obscured. Dunno.
>
> I don't understand your confusion. You just do exactly what you said you
> want to do and refer to <lib>../../test/build/boost_prog_exec_monitor in
> the sources for those example/tests.
<lib>../../test/build/boost_prog_exec_monitor is fed directly to rule
"run" inside rule run-template, which is invoked from run-(w)invoke,
which is called sometimes from test-bsl-run_files, sometimes from
test-bsl-run. But after this change, some files currently fed to
test-bsl-run (namely those containing the string "demo") will need the
exec_monitor, and some (all the others) will need the
unit_test_framework. I could copy/past the whole group of rules and
make some of them run-template-exec-monitor and some
run-template-unit-test-framework, but that seemed ugly. Hopefully
I've been clear, I have a lot of balls in the air.
Thanks for your help,
-t
Here's that profiling info:
****** before ******
gross net # entries name
41 41 27247 MATCH
453 453 1329 PWD
0 0 1 find-to-root
6 6 96 GLOB
3454 2 1 boost-build
0 0 1 _poke
0 0 98 set-as-singleton
2 2 3113 DEPENDS
0 0 292 NOTFILE
0 0 91 ALWAYS
3 3 429 feature
3 0 312 free-feature
0 0 4 path-feature
0 0 1 dependency-feature
1 0 5 variant
980 398 5277 select-properties
30 29 7913 relevant-features
72 19 97 include-tools
47 27 5141 flags
104 92 36819 split-path
12 10 908 split
452 186 131436 get-values
382 382 152093 get-properties
8 1 1940 is-subset
93 93 39106 intersection
102 102 52329 unique
45 45 5277 normalize-properties
151 39 6055 sort
112 112 30312 bubble
112 112 28217 difference
0 0 1 declare-target-type
3451 6 615 load-jamfiles
671 3 1233 subproject
665 29 1234 SubDir
10 10 2463 FSubDir
262 44 5855 root-paths
283 20 9615 tokens-to-simple-path
309 169 15183 simplify-path-tokens
51 51 15183 reverse
89 89 15183 strip-initial
67 20 12173 FDirName
71 71 17277 join-path
0 0 1 project-root
3 3 1858 FGrist
16 6 1234 adjust-path-globals
0 0 1 project
0 0 1 path-global
2 0 5 import
0 0 3 declare-build-fail-test
0 0 3 declare-build-succeed-test
3448 0 18 test-bsl-run_files
3448 0 90 test-bsl-run_archive
1851 1 54 run-invoke
0 0 276 in-invocation-subdir
1 1 360 ECHO
3446 0 180 run-template
3446 2 180 run
3444 5 180 boost-test
1597 0 36 run-winvoke
0 0 2 test-suite
1 1 652 type-DEPENDS
0 0 90 get-library-name
3403 4 104 declare-local-target
0 0 104 expand-target-names
1 1 922 FGristFiles
13 3 194 declare-basic-target
6 3 284 expand-source-names
10 9 2784 ungrist
25 25 5990 select-gristed
1 1 198 declare-fake-targets
3387 1 90 main-target
2078 43 1318 expand-target-subvariants
6 6 1318 get-BUILD
18 18 7204 select-ungristed
1667 78 2636 expand-build-request
229 94 7640 segregate-free-properties
20 14 2636 report-free-property-conflicts
35 10 2636 remove-incompatible-builds
48 21 2636 segregate-overrides
29 29 5272 feature-default
6 6 5272 replace-properties
84 20 3864 multiply-property-sets
33 16 8416 distribute-feature
168 44 2646 fixup-path-properties
39 22 2636 make-path-property-sets
11 6 2636 remove-default-properties
2 2 2636 ungrist-properties
42 29 2636 split-path-at-grist
276 13 2186 toolset::requirements
263 21 2186 impose-requirements
1 1 598 std::locale-support
0 0 598 force-NT-static-link
0 0 598 toolset::require-boost-spirit-support
480 4 182 dependent-include
63 11 1228 target-path-of
4 4 2476 directory-of
24 3 1238 top-relative-tokens
4 4 1516 join
1 0 1228 protect-subproject
1 1 1228 protect-subdir
751 7 1228 enter-subproject
62 7 2546 relative-path
0 0 2 template
1 0 6 lib
0 0 10 template-modifier
26 12 624 target-id-of
0 0 6 dll
0 0 2 unless
0 0 3 install
0 0 3 stage
30 7 1318 split-target-subvariant
3025 29 704 subvariant-target
2 2 704 get-tag-features
52 26 1408 rename-target
2 2 704 FAppendSuffix
20 19 463 set-target-variables
2664 3 182 generate-dependencies
2638 8 364 link-libraries
2084 32 614 find-compatible-subvariant
3 2 738 is-link-compatible
6 0 4 library-file
22 1 96 Objects
0 0 187 object-name
21 2 187 Object
7 0 553 MakeLocate
6 6 553 MkDir
0 0 304 NOUPDATE
2 0 187 C++
2 2 187 Cc-platform-specifics
0 0 187 C++-action
0 0 4 LibraryFromObjects
0 0 4 Archive
0 0 4 Archive-action
0 0 4 Ranlib
0 0 4 Ranlib-action
37 14 868 common-variant-tag
12 0 181 depend-on-static
13 10 364 depend-on-libs
3 0 183 depend-on-shared
1361 0 90 build-test
0 0 180 RMOLD
1354 5 90 run-test
20 1 90 executable-file
6 1 92 main-from-objects
3 2 90 Link-EXE
1 0 92 .do-link
1 0 92 Link-action
0 0 92 Chmod
0 0 90 test-executable(EXE)
1 1 90 capture-run-output
4 4 1822 INCLUDES
0 0 90 succeeded-test-file
36 4 90 dump-test
1 1 270 get-var-value
0 0 299 toolset::require-shared-libraries-support
4 0 2 dll-files
0 0 2 Link-DLL
0 0 180 toolset::require-wide-char-io-support
51 40 1732 HdrRule
3 3 1732 NOCARE
4 4 2029 remember-binding
****** after ******
gross net # entries name
15 15 7289 MATCH
54 54 161 PWD
0 0 1 find-to-root
0 0 16 GLOB
369 2 1 boost-build
0 0 1 _poke
0 0 18 set-as-singleton
0 0 793 DEPENDS
0 0 52 NOTFILE
0 0 11 ALWAYS
1 1 109 feature
1 0 72 free-feature
0 0 4 path-feature
0 0 1 dependency-feature
1 0 5 variant
95 36 605 select-properties
8 7 905 relevant-features
11 3 17 include-tools
7 3 901 flags
20 14 4451 split-path
5 2 196 split
42 20 15140 get-values
30 30 17589 get-properties
2 1 340 is-subset
10 10 4586 intersection
14 14 6081 unique
3 3 605 normalize-properties
18 4 695 sort
14 14 3496 bubble
10 10 3241 difference
0 0 1 declare-target-type
366 0 71 load-jamfiles
71 1 145 subproject
70 1 146 SubDir
1 1 287 FSubDir
25 1 703 root-paths
34 1 1119 tokens-to-simple-path
37 25 1791 simplify-path-tokens
3 3 1791 reverse
9 9 1791 strip-initial
6 1 1421 FDirName
7 7 2013 join-path
1 0 1 project-root
0 0 226 FGrist
1 0 146 adjust-path-globals
0 0 1 project
0 0 1 path-global
2 0 5 import
0 0 3 declare-build-fail-test
0 0 3 declare-build-succeed-test
363 0 2 test-bsl-run_files
363 0 10 test-bsl-run_archive
190 0 6 run-invoke
0 0 36 in-invocation-subdir
0 0 40 ECHO
363 0 20 run-template
363 0 20 run
363 0 20 boost-test
173 0 4 run-winvoke
0 0 2 test-suite
0 0 252 type-DEPENDS
0 0 10 get-library-name
360 1 24 declare-local-target
0 0 24 expand-target-names
0 0 138 FGristFiles
4 1 34 declare-basic-target
2 1 44 expand-source-names
2 2 480 ungrist
1 1 694 select-gristed
0 0 38 declare-fake-targets
356 0 10 main-target
208 2 150 expand-target-subvariants
0 0 150 get-BUILD
0 0 820 select-ungristed
167 10 300 expand-build-request
25 10 872 segregate-free-properties
3 0 300 report-free-property-conflicts
4 1 300 remove-incompatible-builds
2 1 300 segregate-overrides
4 4 600 feature-default
1 1 600 replace-properties
9 2 440 multiply-property-sets
4 0 960 distribute-feature
17 6 310 fixup-path-properties
6 2 300 make-path-property-sets
3 2 300 remove-default-properties
0 0 300 ungrist-properties
9 6 300 split-path-at-grist
27 0 250 toolset::requirements
27 3 250 impose-requirements
0 0 70 std::locale-support
0 0 70 force-NT-static-link
0 0 70 toolset::require-boost-spirit-support
52 0 22 dependent-include
3 0 140 target-path-of
0 0 300 directory-of
1 0 150 top-relative-tokens
2 2 188 join
0 0 140 protect-subproject
0 0 140 protect-subdir
83 2 140 enter-subproject
11 0 290 relative-path
0 0 2 template
0 0 6 lib
0 0 10 template-modifier
2 0 80 target-id-of
1 0 6 dll
0 0 2 unless
0 0 3 install
0 0 3 stage
4 0 150 split-target-subvariant
315 1 80 subvariant-target
1 0 80 get-tag-features
6 2 160 rename-target
0 0 80 FAppendSuffix
7 7 143 set-target-variables
280 0 22 generate-dependencies
277 1 44 link-libraries
206 1 70 find-compatible-subvariant
1 0 82 is-link-compatible
4 0 4 library-file
8 1 16 Objects
0 0 107 object-name
7 2 107 Object
1 0 153 MakeLocate
1 1 153 MkDir
0 0 64 NOUPDATE
0 0 107 C++
0 0 107 Cc-platform-specifics
0 0 107 C++-action
0 0 4 LibraryFromObjects
0 0 4 Archive
0 0 4 Archive-action
0 0 4 Ranlib
0 0 4 Ranlib-action
6 2 100 common-variant-tag
1 0 21 depend-on-static
2 1 44 depend-on-libs
2 0 23 depend-on-shared
133 0 10 build-test
0 0 20 RMOLD
132 0 10 run-test
2 0 10 executable-file
2 0 12 main-from-objects
0 0 10 Link-EXE
1 1 12 .do-link
0 0 12 Link-action
0 0 12 Chmod
0 0 10 test-executable(EXE)
0 0 10 capture-run-output
1 1 1743 INCLUDES
0 0 10 succeeded-test-file
3 1 10 dump-test
0 0 30 get-var-value
0 0 35 toolset::require-shared-libraries-support
4 0 2 dll-files
1 0 2 Link-DLL
1 1 20 toolset::require-wide-char-io-support
47 40 1733 HdrRule
0 0 1733 NOCARE
1 1 2038 remember-binding
Boost list run by bdawes at acm.org, gregod at cs.rpi.edu, cpdaniel at pacbell.net, john at johnmaddock.co.uk