Re: [jamboost] Boost.Build stress test

From: Ali Azarbayejani (ali_at_[hidden])
Date: 2003-03-20 16:10:23

Vladimir Prus wrote:
>
> 
> > way...indeed, Jam files have better syntax, but the BB system is
> > so complex that it remains incomprehensible to a regular user as a
> > practical matter.
> 
> Do you think it's so complex that regular user won't ever understand it? Or
> there's something wrong with docs which can be improved?

My only point was that the BB system shares the same problem as my
master makefile system in that the guts of it is unapproachable by a
regular user (as opposed to a power user), so if it is not working as
documented for some reason, the user gets frustrated and has to ask or
look for help instead of being able to diagnose it easily and
immediately. Users don't like that.

I think the documentation is already okay for using the system...when
it works. Bjam already has some good diagnostic features (e.g. -d+2)
and others could be addressed later. The matter is secondary for me
at the moment because I've already put in the effort to understand
and, for now, I just want to get it to work on existing
projects...I'll worry about regular users later.

> > (However, note that it took me 2-3 weeks to figure out how to
> > write those 236 lines!...Jam/BB is *very* difficult to learn.)
> 
> If you tell what can be improved, we'll try to.

The best design document for learning a complex system is a layering
diagram. I created one for BB for my own use using the "import"
declarations in the new/*.jam files (I think it's not perfect because
you don't have to import to use rules from another module, right?).
Anyway, even if not perfect, that was finally very helpful in
directing my understanding of the system. I've attached my layering
diagram if you want to see it.

> I'm very interested in do-nothing timing. Can you run "bjam -d+10" on an
> unmodified tree and send me the output. This will tell what consumes most
> time.

Attached.

> I see that all problems with those are listed below, so I'll comment later.
> May I suggest one experiment? Go to builtin.jam, find 'compile-action' rule
> and inside it 'adjust-properties'. Change the return line to read:
> 
> return $(properties) ;
> 
> and try again. This might not reduce command line length sufficiently, but
> it's interesting to check.

I skipped this because your fix earlier in the week solved my archive
problem and Juergen's fix to jam.h:MAXLINE solved the rest of the
problems. Looks like it was a compiled-in Jam limitation rather than
a shell limitation.

> > <F> (DESIGN) "Remote" builds are performed in root rather than
> > remote directory. Consider building each target in the
> > directory of its own Jamfile (rather than in the directory of
> > the root Jamfile). This will alleviate command line overflow,
> > guarantee consistent run-time behavior for a project when used
> > on its own or as a sub-project, and help avoid internal
> > implementation errors having to do with translating paths in
> > many different contexts.
> 
> Do you mean that current dir for all compiler invocations is equal to the
> directory where you've started bjam? I understand you, but the change is
> controversial. For one thing, it will immediately broke *compilation* mode in
> Emacs --- the error message will use file names relative to some other
> directory than the one where you've run the compilation.

I sympathize because I am an Emacs user, but I have to say that this
is an Emacs problem and ought ideally to be solved by providing elisp
code to do Emacs-based compiles. A user of my system encountered a
similar problem because my system does change directory before
building...I will check to see what his solution was (if any).

However, it looks like Juergen's solution of changing jam.h:MAXLINE to
a higher number solves my current problem, so maybe we can skip this
for now along with all the associated controversy and elisp. (It's
still not a bad idea to reduce the command line verbosity where
possible, cuz it's going to bite sooner or later...like as soon as I
build Lapack.)

> > Implementing suggestions above for <E> and <F> will alleviate
> > this problem and use of the function builtin.variant() can
> > address (iii), but there may still be a library that overflows
> > the command line (Lapack, e.g., has something like 1600
> > files). The overflow case should be detected and handled in
> > the following way. For an Archive, can add object members
> > incrementally (as many as will fit on the command line, a la
> > the "xargs" utility). For a DLL, make an Archive
> > incrementally as above and link to the Archive.
> 
> Oh... is this common approach for creating DLL with too many sources?
> It looks like very general and usefull. One question, though: do you know
> if it works on Windows. I mean, can you use "_declspec(export)" on symbol
> in static lib and then have the symbol exported when the static link becomes
> dynamic?

I don't know if it is common, but it is a workaround that I have used.
It will work on Windows/GCC, but I assume you mean will it work on
MSVC. On MSVC, you will have to provide a ".def" file with "EXPORT"
statements and I'm not sure if there is a standard automatic way to
create this file. But we can cross this bridge when we come to it. --------------3F9938CB513BE94B4C9BBE5E Content-Type: text/plain; charset=us-ascii;
name="layers.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="layers.txt"

==============================================================================
DOCUMENTS
m1.txt
m2-transformations.txt
readme.txt

==============================================================================
CONFIG FILES

boost-build.jam	Identifies build system location as here
site-config.jam	Config file
user-config.jam	Config file

==============================================================================

EXTENSIONS

bison.jam	Bison Parser Generator Tool
borland.jam	Borland C/C++ Toolset
gcc.jam	GCC C/C++ Toolset
kylix.jam	Borland Kylix C/C++ compiler
lex.jam	Lex Lexical Analyzer Tool
msvc.jam	MSVC C/C++ Compiler Toolset
qt.jam	Qt UI Tools
stlport.jam	Adjustments related to STL PORT Tool

==============================================================================
BOOST.BUILD SYSTEM
------------------------------------------------------------------------------
BOOTSTRAP

bootstrap.jam	-> build-system, doc, modules

------------------------------------------------------------------------------
BUILD SYSTEM

build-system.jam	-> build-request, builtin, errors, feature, make,
modules, os, project, property-set, sequence,
test-config, toolset, version

------------------------------------------------------------------------------
BUILTIN

builtin.jam	-> alias, class, errors, feature, generators, os,
prebuilt, regex, scanner, stage, symlink, toolset,
type, virtual-target

modifiers.jam	-> class, errors, feature, generators, modules,
numbers, property, property-set, sequence, symlink,
type, virtual-target

------------------------------------------------------------------------------
DERIVED TARGETS, PREBUILT, PROJECT, TESTING

alias.jam	-> class, errors, regex, property, targets, type
symlink.jam	-> class, feature, modules, numbers, os, property,
targets, virtual-target
stage.jam	-> class, errors, property, regex, targets, type
make.jam	-> class, errors, property, property-set, regex,
targets, type

prebuilt.jam	-> class, errors : error, feature, generators,
property, regex, targets, type

project.jam	-> assert, class, errors, modules, numbers, path,
print, project, project-root, sequence, targets
project-root.jam	-> class, modules, path, print, project, regex,
sequence, set

testing.jam	-> class, feature, property, targets, toolset

------------------------------------------------------------------------------
TARGETS

targets.jam	-> build-request, class, common, errors, numbers,
property, property-set, regex, sequence,
virtual-target

------------------------------------------------------------------------------
VIRTUAL TARGET

virtual-target.jam	-> class, property-set, type, utility

test.jam	-> build-request, class, container, feature, numbers,
os, path, property, sequence, string

------------------------------------------------------------------------------
TYPES/GENERATORS, BUILD-REQUEST, TOOLSET, SCANNER

generators.jam	-> assert class, container, errors, feature, numbers,
set, type, utility
type.jam	-> class, errors, feature, generators, property, regex

build-request.jam	-> assert, class, container, errors, feature, numbers,
property, regex, set, sequence

toolset.jam	-> assert, errors, feature, numbers, property

scanner.jam	-> class, property

------------------------------------------------------------------------------
PROPERTIES

property.jam	-> assert, class, errors, feature, sequence, utility
property-set.jam	-> class, feature, sequence

------------------------------------------------------------------------------
FEATURES

feature.jam	-> assert, class, errors, regex, sequence, set, utility

------------------------------------------------------------------------------
MID-LEVEL UTILITIES

common.jam	-> modules, os
container.jam	-> assert, class, numbers, sequence, utility
common.jam	-> modules, os
doc.jam	-> class, container, modules, print, sequence, set
os.jam	-> modules, regex
path.jam	-> assert, errors, modules, regex, sequence
print.jam	-> assert, modules, numbers, regex, string
utility.jam	-> assert, class, errors

------------------------------------------------------------------------------
MODULES, CLASSES, ERRORS, and LOW-LEVEL UTILITIES

class.jam	-> assert, errors, numbers, modules, set
modules.jam	-> assert
assert.jam	-> errors, modules
errors.jam	-> sequence
sequence.jam	-> assert, numbers
numbers.jam	-> assert, errors, sequence, string
regex.jam	-> assert
set.jam	-> assert
string.jam	-> assert, regex

------------------------------------------------------------------------------
VERSION

version.jam	-> 

 --------------3F9938CB513BE94B4C9BBE5E Content-Type: text/plain; charset=us-ascii;
name="do-nothing.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="do-nothing.txt"

 gross net # entries name
1310000 1310000 463547 MATCH
290000 290000 2319 GLOB
0 0 81 PWD
0 0 1 find-to-root
175840000 0 1 boost-build
610000 610000 23684 IMPORT
175840000 470000 6019 modules.import
130000 130000 35912 CALLER_MODULE
260000 260000 26280 modules.peek
175840000 860000 6046 modules.load
0 0 209 modules.poke
0 0 65 modules.record-binding
40000 40000 6198 modules.binding
60000 60000 18401 RULENAMES
0 0 38 class.class
0 0 110 EXPORT
0 0 1 doc.help
5510000 200000 4028 class.new
5120000 370000 4029 class.instance
3420000 1100000 9310 class.__init__
80000 80000 9376 BACKTRACE
280000 150000 9450 assert.nonempty-variable
460000 80000 1031 container.vector
0 0 1031 container.node
190000 170000 4078 numbers.increment
0 0 3 object(vector)@1.front
0 0 3 object(vector)@1.pop-front
0 0 1 feature.feature-space
0 0 11 os.name
153540000 100000 942 property-set.create
152730000 146000000 953 sequence.insertion-sort
760000 760000 3940 sequence.unique
520000 160000 238 property-set.property-set
10000 10000 38 feature.feature
280000 280000 28559 feature.grist
0 0 101 feature.extend
0 0 101 feature.extend-feature
0 0 130 feature.validate-feature
0 0 1 property.property-map
0 0 1 prebuilt.prebuilt-file-generator
30000 10000 14 generators.generator
1600000 130000 3084 sequence.transform
10000 0 14 feature.expand
0 0 31 feature.expand-subfeatures
0 0 31 feature.expand-subfeatures-aux
0 0 51 feature.validate-value-string
2700000 2230000 17252 regex.split
10000 10000 15 feature.expand-composites
0 0 19 feature.expand-composite
0 0 14 generators.register
0 0 412 object(prebuilt-file-generator)@1.target-types
0 0 207 object(prebuilt-file-generator)@1.id
0 0 2 builtin.variant
0 0 15 property.validate
0 0 19 property.validate1
30000 30000 627 utility.ungrist
570000 440000 14051 feature.valid
990000 280000 14032 feature.attributes
180000 20000 116 property.refine
0 0 42 feature.compose
10000 10000 20 type.register
10000 0 12 builtin.declare-type
0 0 12 type.registered
0 0 19 object(property-map)@1.insert
0 0 13 type.register-suffixes
0 0 6 type.set-generated-target-suffix
0 0 1 stage.stage-exe-generator
8380000 4130000 268318 modules.call-in
0 0 25 type.validate
0 0 2 scanner.register
0 0 2 type.set-scanner
20000 20000 3 numbers.extend
10000 0 1 builtin.lib-generator
0 0 1 builtin.searched-lib-generator
20000 0 1 toolset.using
10000 0 5 generators.register-standard
0 0 1 boostbook.init
2100000 0 182 project.load
2100000 0 182 project.load-jamfile
1180000 70000 2121 project.find-jamfile
1110000 70000 2251 path.glob
200000 110000 11604 path.native
90000 90000 11604 path.native-UNIX
7980000 30000 2582 path.make
7950000 40000 2582 path.make-UNIX
8660000 1930000 40647 path.join
1790000 1400000 274731 path.is-rooted
8050000 2490000 40531 path.join-imp
1460000 800000 22336 path.parent
80000 80000 22336 path.has-parent
1460000 1140000 61442 regex.match
0 0 248 project.module-name
280000 10000 22 project.initialize
0 0 22 project.project-attributes
10000 10000 47 modules.clone-rules
180000 0 22 project-root.load
180000 0 44 path.glob-in-parents
120000 20000 44 path.all-parents
60000 0 62 path.pwd
1850000 90000 4058 path.root
0 0 3 project-root.project-root-object
30000 0 3 object(project-root-object)@1.initialize
10000 0 44 project-root-context.project-root
0 0 22 object(project-root-object)@1.get-location
0 0 148 object(project-attributes)@1.set
0 0 29 property-set.empty
160000 90000 7893 project.attribute
70000 70000 8007 object(project-attributes)@1.get
10000 0 22 object(project-root-object)@1.register-project
0 0 22 object(project-root-object)@1.intern-constants
20000 20000 1000 set.difference
0 0 22 project-rules.project
0 0 90 project.attributes
0 0 22 project.register-id
0 0 1 Jamfile<.>.show
0 0 18 ECHO
10000 0 35 path.reverse
260000 260000 23728 sequence.join
0 0 18 object(project-root-object)@1.get-module
1830000 0 60 project-rules.use-project
1790000 0 60 project.use
0 0 1 feature.subfeature
0 0 2 feature.get-subfeature-name
0 0 1 feature.extend-subfeature
0 0 2 builtin.register-linker
0 0 2 builtin.linking-generator
10000 0 1 generators.register-composing
10000 0 2 builtin.register-c-compiler
10000 0 2 builtin.C-compiling-generator
10000 0 28 toolset.flags
0 0 13 property.validate-property-sets
450000 230000 6282 feature.split
10000 10000 13 toolset.normalize-condition
0 0 28 toolset.add-flag
0 0 9 Jamfile<.>.replace
1110000 0 66 type.main-target-rule
630000 20000 66 targets.main-target-requirements
90000 30000 132 property.translate-paths
20000 20000 523 property.split-conditional
2800000 2800000 266323 sequence.less
2130000 30000 476 object(property-set)@1.refine
100000 100000 3070 object(property-set)@1.raw
10000 0 66 targets.main-target-default-build
240000 0 66 targets.main-target-usage-requirements
2570000 0 136 object(property-set)@1.add
20000 10000 66 targets.typed-target
10000 0 66 targets.basic-target
10000 10000 149 targets.abstract-target
140000 10000 66 targets.main-target-alternative
20000 20000 1890 object(typed-target)@1.project
70000 40000 2052 project.target
0 0 18 targets.project-target
20000 20000 1890 object(typed-target)@1.name
100000 0 222 object(project-target)@1.main-target
30000 0 65 targets.main-target
0 0 66 object(main-target)@1.add-alternative
0 0 66 object(typed-target)@1.default-build
0 0 1 build-request.from-command-line
20000 20000 168 object(vector)@1.get-at
0 0 168 object(vector)@1.get
164650000 0 1 object(project-target)@1.generate
164650000 10000 156 object(main-target)@1.generate
0 0 156 object(main-target)@1.apply-default-build
30000 0 156 object(property-set)@1.add-defaults
0 0 3 feature.add-defaults
0 0 23 feature.defaults
164640000 0 156 object(main-target)@1.generate-really
1900000 0 156 object(main-target)@1.select-alternatives
120000 20000 363 class.is-a
100000 40000 363 class.is-derived
60000 60000 661 class.bases
0 0 156 assert.equal
1820000 0 156 object(typed-target)@1.match-rank
0 0 1040 object(property-set)@1.base
40000 40000 349 set.intersection
200000 200000 2772 sequence.length
410000 70000 1351 sequence.select-highest-ranked
340000 20000 1233 sequence.max-element
164620000 10000 156 object(typed-target)@1.generate
1310000 0 70 object(typed-target)@1.final-properties
740000 0 168 object(property-set)@1.evaluate-conditionals
90000 50000 92 property.evaluate-conditionals-in-context
630000 0 70 object(property-set)@1.run-actions
20000 20000 44 feature.run-actions
17830000 10000 168 targets.generate-dependencies
30000 30000 1094 object(property-set)@1.dependency
0 0 168 object(property-set)@1.incidental
30000 30000 4532 object(property-set)@1.free
36320000 60000 70 object(typed-target)@1.generate-sources
36320000 130000 1039 targets.generate
3360000 80000 1039 project.find-target
3240000 160000 2078 project.lookup-with-load
1130000 270000 2191 project.lookup
0 0 161 object(project-target)@1.has-main-target
920000 40000 884 virtual-target.from-file
0 0 526 type.type
1040000 90000 1495 virtual-target.file-target
560000 170000 1537 virtual-target.abstract-file-target
120000 120000 1537 virtual-target.virtual-target
0 0 526 object(file-target)@1.suffix
20000 20000 993 project.remove-trailing-slash
200000 0 155 object(property-set)@1.propagated
50000 30000 140 object(typed-target)@1.get-usage-requirements
20000 20000 1039 object(file-target)@1.usage-requirements
10040000 20000 70 object(typed-target)@1.construct
6630000 110000 1083 generators.construct
20000 10000 1083 generators.ensure-type
60000 60000 15842 object(file-target)@1.type
0 0 1083 generators.increase-indent
40000 40000 9694 generators.indent
30000 30000 9776 generators.generators.dout
880000 620000 16518 object(file-target)@1.str
130000 130000 29030 object(file-target)@1.action
3680000 90000 1083 generators.construct-with-caching
6520000 20000 199 generators.construct-without-caching
280000 40000 199 generators.find-viable-generators
20000 20000 2418 type.all-bases
10000 0 2023 object(property-set)@1.as-path
10000 0 44 property.as-path
10000 10000 2 feature.minimize
0 0 2 feature.move-subfeatures-to-the-end
0 0 2 sequence.filter
0 0 10 feature.in-features
0 0 4 property.path-order
0 0 4 feature.implied-feature
10000 10000 398 object(prebuilt-file-generator)@1.source-types
90000 60000 398 object(prebuilt-file-generator)@1.match-rank
0 0 398 object(prebuilt-file-generator)@1.requirements
10000 10000 193 object(prebuilt-file-generator)@1.optional-properties
6250000 30000 193 generators.try-one-generator
5440000 30000 69 object(lib-generator)@1.run
4670000 0 82 object(prebuilt-file-generator)@1.run
4120000 100000 28 object(prebuilt-file-generator)@1.convert-multiple-sources-to-consumable-types
4020000 60000 949 object(prebuilt-file-generator)@1.convert-to-consumable-types
50000 40000 949 object(prebuilt-file-generator)@1.consume-directly
20000 0 2219 type.is-derived
3840000 40000 911 generators.construct-types
320000 30000 921 numbers.less
20000 20000 2763 numbers.check
20000 0 1842 numbers.log10
0 0 54 string.chars
260000 250000 1842 numbers.range
400000 10000 55 object(prebuilt-file-generator)@1.construct-result
150000 10000 55 object(prebuilt-file-generator)@1.generated-targets
10000 10000 4502 object(file-target)@1.name
0 0 27 object(C-compiling-generator)@1.action-class
260000 20000 911 builtin.compile-action
130000 110000 939 virtual-target.action
20000 10000 2441 class.is-instance
150000 100000 939 virtual-target.register
30000 30000 1823 object(file-target)@1.project
760000 10000 662 object(vector)@1.str
660000 10000 1502 utility.str
50000 50000 12196 object(compile-action)@1.sources
30000 30000 8548 object(compile-action)@1.action-name
10000 10000 193 generators.base-to-derived-type-conversion
0 0 166 object(vector)@1.push-back
30000 0 199 generators.select-dependency-graph
10000 0 365 object(vector)@1.size
0 0 1083 generators.decrease-indent
2820000 260000 1768 virtual-target.clone-template
1090000 40000 884 virtual-target.clone-action-template
20000 20000 5248 object(compile-action)@1.properties-ps
10000 10000 4504 object(compile-action)@1.targets
30000 30000 884 object(compile-action)@1.set-targets
0 0 10 object(prebuilt-file-generator)@1.action-class
250000 140000 70 object(typed-target)@1.check-for-unused-sources
220000 180000 3676 virtual-target.traverse
40000 40000 5407 object(file-target)@1.root
3030000 10000 70 object(typed-target)@1.set-usage-requirements
0 0 70 object(file-target)@1.set-usage-requirements
220000 0 70 object(typed-target)@1.create-subvariant-dg
90000 30000 70 virtual-target.subvariant-dg
0 0 2093 object(file-target)@1.dg
60000 0 42 object(searched-lib-generator)@1.run
20000 10000 102 feature.get-values
20000 0 42 builtin.searched-lib-target
3310000 0 18 object(linking-generator)@1.run
0 0 18 object(linking-generator)@1.action-class
20000 10000 18 builtin.link-action
320000 0 9 object(file-target)@1.depends
270000 110000 921 sequence.merge
9200000 130000 2959 object(file-target)@1.actualize
9200000 160000 2959 object(file-target)@1.actualize-no-scanner
2900000 210000 2959 object(file-target)@1.actual-name
580000 180000 2392 object(file-target)@1.specific-main-target
0 0 1824 object(subvariant-dg)@1.main-target
0 0 1824 object(subvariant-dg)@1.properties-ps
260000 0 1824 object(project-target)@1.reference-properties
1380000 80000 954 type.generated-target-suffix
1300000 40000 996 object(property-map)@1.find
1260000 720000 996 object(property-map)@1.find-replace
70000 70000 1480 virtual-target.register-actual-name
190000 100000 2042 object(file-target)@1.actualize-location
590000 70000 1019 object(file-target)@1.path
0 0 912 object(compile-action)@1.path
20000 20000 2903 DEPENDS
40000 30000 1050 common.MkDir
10000 10000 1050 NOUPDATE
0 0 44 NOTFILE
8840000 30000 1480 object(file-target)@1.actualize-action
8840000 170000 912 object(compile-action)@1.actualize
100000 10000 18 object(link-action)@1.adjust-properties
5980000 30000 18 object(link-action)@1.actualize-sources
8580000 150000 912 object(compile-action)@1.actualize-sources
350000 40000 1768 type.get-scanner
1260000 100000 894 object(compile-action)@1.adjust-properties
870000 40000 981 object(subvariant-dg)@1.all-target-directories
840000 20000 28 object(subvariant-dg)@1.compute-target-directories
310000 130000 604 scanner.get
850000 850000 3364 property.select
0 0 24 builtin.c-scanner
0 0 24 scanner.scanner
790000 650000 8197 scanner.install
0 0 8197 object(c-scanner)@1.pattern
3510000 1270000 1824 toolset.set-target-variables
910000 460000 6242 toolset.find-property-subset
0 0 17 gcc.link-dll
0 0 42 object(searched-lib-target)@1.actualize-location
0 0 136 object(searched-lib-target)@1.real-name
0 0 136 object(searched-lib-target)@1.search
0 0 136 object(searched-lib-target)@1.shared
0 0 1 gcc.link
0 0 1 UPDATE
200000 200000 9635 virtual-target.remember-binding
11360000 60000 2113 scanner.hdrrule
11300000 2320000 2113 object(c-scanner)@1.process
190000 170000 4226 regex.transform
170000 170000 9706 virtual-target.binding
630000 510000 7593 SEARCH_FOR_TARGET
70000 70000 7593 INCLUDES
40000 40000 7593 NOCARE
 --------------3F9938CB513BE94B4C9BBE5E--