Re: boost-build.jam documentation issues

From: David Abrahams (dave_at_[hidden])
Date: 2005-08-11 15:03:26

Andrey Melnikov <melnikov_at_[hidden]> writes:

> David Abrahams wrote:
>> Andrey Melnikov <melnikov_at_[hidden]> writes:
>
>> In the next week I have to prepare a
>> presentation about building and testing Boost, and I'm going to need
>> to review and understand all the documentation. Maybe we can work
>> together to whip it into shape.
>
> cool

Great.

>>>What is /usr/share/boost-build? Did this path came from the
>>>environment variables? It looks just like the default BB search path
>>>on Unix. If it's the default location, the message should be
>>>changed.
>> 
>> This is due to the unfortunate exposure of an impelementation detail
>> to users. http://groups.yahoo.com/group/jamboost/message/10103
>> explains.
>
> I didn't understand that message.

The point is that bjam acquires global variables corresponding to all
the shell variable settings on startup. BBv2 then appends a directory
to its value of $(BOOST_BUILD_PATH) [which I was saying in the message
was a poor design choice] before doing the search. That's the
implementation detail. Then it reports that the file can't be found
on its internal version of $(BOOST_BUILD_PATH), which is misleading.

>>>Also we could direct users directly to 
>>>http://www.boost.org/doc/html/bbv2.html
>> 
>> That's a good idea. Would you like CVS write privileges?
>
> Yes. Do you need my SF username? My SF username is nimnul.

Okay, I have set you up. Please read
https://sourceforge.net/docman/display_doc.php?docid=14033&group_id=1
to get started.

> What is the right way to submit the changes? 

Check things into CVS.

> Do we have multiple branches in CVS? 

We have a 1.33 release branch, which you should probably not touch at
this point. You can make your own branch if you want to.

> I actually never used CVS, only Clearcase and Perforce.

The SourceForge site docs have good introductory material.

>>>The documentation is in three places again:
>>>http://www.boost.org/tools/build/jam_src/#jambase_replacement
>> 
>> This describes the low-level behavior of the bjam tool; we don't
>> really want that exposed in the regular documentation.
>
> Are we going to have 3 documentation levels:
> - Brief Tutorial/Getting Started/FAQ documentation
> - User reference
> - Developer/extender reference.

Yes, that was the idea.

>> Also, I think we may be doing a few too many "smart" things at
>> startup. 
>
> I don't understand you here.

I mean that the startup procedure may be more complex than necessary.
The following explains in detail:

>> At least we can throw out the BOOST_ROOT business.
>
> Do you mean to remove all references to Boost from BB documentation?

No, I mean removing the part of the behavior described in the link
above that has to do with BOOST_ROOT, namely:

As a backward-compatibility measure for older versions of
Boost.Build, when the BOOST_ROOT variable is set, we first search
for boost-build.jam in $(BOOST_ROOT)/tools/build and
$(BOOST_BUILD_PATH). If found, it is loaded and initialization is
complete. 

> How we are going to separate BB in the future? Do we want it to
> remain in main Boost distribution? 

Yes.

> I think it would be better to ship BB as a separate package 

It is being shipped also as a separate package.

> or at least not inside BOOST_ROOT.

I don't see why that would be better. At any rate, it seems like a
very unimportant change to make.

>>>Should http://www.boost.org/tools/build/jam_src/ be merged into bbv2 
>>>documentation tree some day?
>> 
>> 
>> Probably not. 
>
> I meant into bbv2 developer/extender reference. Not into user
> reference. Now it's outside of bb documentation tree.

I know what you meant.

>> We want to move off of the jam substrate onto a rewrite
>> based on Python as soon as BBv2 can replace BBv1. 
>> If any of the
>> documentation at http://www.boost.org/tools/build/jam_src/index.html
>> is still relevant after the Python translation is done, then by all
>> means, let's move it.
>
>
> I don't know anything about these translation plans. Are they
> documented somewhere? 

Yes, in the mailing list archive. You have a lot of catching up to
do. Please try to use the discussion history here as a way to get up
to speed.

> Do we want to cancel Jam language and rewrite BB in Python? 

Probably, eventually. First, we are going to rewrite all of the BBv2
guts in Python. You can build bjam --with-python so that it embeds a
Python interpreter. That allows the jam language to remain for
Jamfiles while we rewrite the internal logic.

> Or we want Jam interpreter to be written in Python?

Probably not.

>>>Some comments regarding doc/html/bbv2/reference.html#bbv2.reference.init
>>>
>>> > bjam's first job upon startup is to load the Jam code that implements 
>>> > the build system. To do this, it searches for a file called
>>> > boost-build.jam, first in the invocation directory, then in its 
>>>parent > and so forth up to the filesystem root, and finally in the 
>>>directories > specified by the environment variable BOOST_BUILD_PATH.
>>>
>>>What is "invocation directory"? Is this the current directory, 
>> 
>> 
>> Yes, the current directory at the time bjam was invoked. I thought
>> that would be clearer than "current" since some people might be
>> confused by that term.
>
> "The current directory at the time bjam was invoked" sounds better to 
> me. I think we can expect that all BB users know what "current 
> directory" is. I think any C++ developer knows it. It isn't a shell 
> term. I can program for Windows without knowledge about cmd.exe, but I 
> cannot if I don't know what is current directory. It's important during 
> interpertation of relative paths, and its very hard not to understand 
> how relative paths work and be a developer.

Yes, but the "current directory" can change during the course of
execution of the program. If you add "at the time bjam was invoked"
then it is unambiguous.

> >> > When found, the file is interpreted, and should specify the build
>>> > system location by calling the boost-build rule:
>>> >
>>> > rule boost-build ( location ? )
>>>
>>>This is boost-build rule declaration syntax. Users don't need to declare 
>>>a new boost-build rule. They even don't need to know the declaration 
>>>syntax. They only need to know what to put into boost-build.jam. So we 
>>>should have just
>>>
>>> boost-build /usr/share/boost-build/kernel
>> 
>> 
>> Indeed.
>> 
>> boost-build <replaceable>path/to/boost-build</replaceable> ;
>> 
>> would be more appropriate, though. Many people don't end up with
>> Boost.Build in that directory.
>
> This isn't perfect either because <replaceable> looks like a
> feature. 

No, no; that's BoostBook XML, which is what the docs are written in!
> I 
> think we should highlight path/to/boost-build using italics 

That's exactly how <replaceable>...</> is formatted.

> or something like this and provide an example besides general
> syntax.
>
> I have no idea what is the best convention to specify syntax. I
> think we should add such a convention to the documentation and use
> it everywhere for consistency.

It's already there; at least I started it. Some of it needs to be
made more consistent. If you look at the XML sources you can see that
I inserted many editorial comments, about half of which Volodya hadn't
had time to respond to, last time I looked.

>>>there. The last /kernel part isn't obvious. 
>> 
>> 
>> 
>> It isn't right either.
>> 
>> boost-build /usr/share/boost-build ;
>> 
>> would be correct for an installation of BBv2 in
>> /usr/share/boost-build.
>> 
>> 
>>>We should mention in the documentaion that the path isn't to bb
>>>installation root, but to kernel directory where bootstrap.jam can
>>>be found. Should BOOST_BUILD_PATH point to /kernel too?
>> 
>> 
>> Nope and nope. I understand why you were misled by the
>> boost-build.jam in the top BBv2 directory.
>
> Do you mean that despite /usr/share/boost-build/kernel works, it's 
> conceptually incorrect and shouldn't be used by users?

Right.

>>>Also, what is the reason to have so many boost-build.jam files? I think 
>>>we can live just with the file located at root.
>> 
>> Which ones are you talking about? The one in the root of BBv2 is used
>> to forward its work on to the one in kernel/
>
> Take a look at all these boost-build.jam found in Boost CVS:
>
> boost\boost-build.jam
> boost\libs\python\example\boost-build.jam
> boost\tools\build\v1\example\boost-build.jam
> boost\tools\build\v2\boost-build.jam
> boost\tools\build\v2\example\boost-build.jam
> boost\tools\build\v2\kernel\boost-build.jam
> boost\tools\build\v2\test\boost-build.jam
> boost\tools\build\v2\test\module-actions\boost-build.jam
> boost\tools\build\v2\test\startup\boost-root\boost-build.jam
> boost\tools\build\v2\test\startup\boost-root\build\boost-build.jam
> boost\tools\build\v2\test\startup\bootstrap-env\boost-build.jam
> boost\tools\build\v2\test\startup\bootstrap-explicit\boost-build.jam
> boost\tools\build\v2\test\startup\no-bootstrap1\boost-build.jam
> boost\tools\build\v2\test\startup\no-bootstrap2\boost-build.jam
> boost\tools\build\v2\test\startup\no-bootstrap3\boost-build.jam
> boost\tools\build\v2\test\testing-primitives\boost-build.jam
> boost\tools\build\v2\test\v1-testing\boost-build.jam
>
> 1) Are all these files really required?

Yes. The ones in the test subdirectory are being used... for
tests of the build system!

> 2) Does all these files point to the right place?

Yes.

> Not only boost-build in the top directory is misleading. 

What's misleading about it?

> For example, why boost\tools\build\v2\example\boost-build.jam points
> to ../kernel, and not to ..?

No good reason, I'm sure. Maybe that should be fixed.

> Also, why do we have build-system.jam bootstrap.jam boost-build.jam in 
> the top directory? Can we put them into kernel/ folder, so the top 
> folder isn't cluttered with files useless for an end user?

I don't know. Maybe. But why does an end user care about the
internal directory structure of BBv2?

> Should we move other such useless files to a new subfolder?

What are you assuming is useless?

Most of the things you're poking at here don't rate highly on my scale
of "confusing things about BBv2."

> Should we separate
> - bb core (kernel, build, toolsets, options, util)
> - documentation/examples
> - internal bb developer stuff (debian new notes python test)
> - user-modified settings and user extensions?

Maybe. Again, I think these are of little importance at this stage.

>>> > In this case, running bjam anywhere in the project tree will
>>> > automatically find the build system.
>>>
>>>This sentense is misleading. It sounds like boost-build.jam was
>>>designed for per-project configuration, and user will have to have
>>>boost-build.jam in all her projects.
>> 
>> 
>> That's one acceptable way to do it.
>
> Yes, it's acceptable. But it's not the recommended way. It's useful
> only in complex scenarios. In simple cases so many files introduce
> complexity, and complexity scare potential users. The documentation
> should show that BB is simpler than other tools. That it scales up
> and down very well according to real user needs and users won't need
> to create 100 files just to compile "hello world".

I agree in principle.

>>> > Locations where those files a search are summarized below:
>>> >
>>> > Table 26.1. Search paths for configuration files
>>>
>>>The shell is rarely used on Windows. %% syntax is uncommon to most of 
>>>the users. Recommended way to set per-user or per-site environment 
>>>variables is through GUI. Most users know only basic shell syntax and 
>>>won't be able to guess that %HOMEDRIVE%%HOMEPATH% is a string 
>>>concatenation and really their profile directory is meant.
>> 
>> That may be true, but 26.1 isn't the tutorial anymore. Most people
>> shouldn't be reading this section of the documentation unless they
>> need to understand the low-level details of the build system. We want
>> the information to be complete and precise enough for people who want
>> to understand exactly which directory is being referred to. Can you
>> think of a better rephrasing for that audience?
>
> I'll think about this.
>
> One more problem I see with this table is that it's not clear that 
> %BOOST_BUILD_PATH% isn't getenv(BOOST_BUILD_PATH) because the path can 
> also be autodetected using boost-build.jam.

Yes, that's the confusion of the implementation detail I mentioned
earlier, leaking into the documentation.

>>>%SystemRoot% is <drive>:\WINDOWS or <drive>:\WINNT. Do you really
>>>suggest to put something into windows system folders? Or actually 
>>>%SystemDrive%\ is meant?
>> 
>> 
>> According to the code, it's SystemRoot
>> 
>> path = [ modules.peek : SystemRoot ] $(user-path) ; 
>> 
>> Is there a better default place for a site-wide configuration on
>> Windows?
>
> I think that even %SystemDrive%\ is still better that %SystemRoot%.
>
> The recommended place is SHGetFolderPath(CSIDL_COMMON_APPDATA). We can 
> approximate that call with 
> HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\Shell 
> Folders\Common Appdata or with %ALLUSERSPROFILE%\Application Data

I'm not a Windows expert, so I'll take your word for it.

>>> > Boost.Build comes with default versions of those files, which can
>>> > serve as templates for customized versions.
>>>
>>>The files aren't "default" because they cannot be used as is. 
>> 
>> 
>> Sure they can. You can create just a user-config and allow the
>> system's site-config to take effect. What problem are you referring
>> to?
>
> I must edit user-config before I can use it.

"serve as templates" indicates that fact perfectly well IMO.

> Also, what are the reasons to have toolsets configured in
> user-config, and not in site-config? I don't see any scenarios when
> users would like to have different toolset configurations. 

We did see those scenarios.

> The only thing which should be configured by each user separately is
> maybe the default toolset. 

Please, a little less presumption.

Maybe my site administrator has Python-2.3 installed and I want to
test against a Python-2.4 installed in my personal local directory (or
think GCC versions if you prefer).

> Also there are performance reasons. 

Performance reasons for what?

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com