Re: [jamboost] Target id syntax

From: Ali Azarbayejani (ali_at_[hidden])
Date: 2003-06-02 08:57:15

Volodya,

Vladimir Prus wrote:
> 
> The question really was: why build system should handle patching --- why not 
> extract/patch once, and live with it. You've answered my question.

That was actually my first design (to extract and patch once), but 
occasionally a new or revised patch gets checked in and you want users 
to be able simply to update from the repository and re-build and end up 
with the right answer. This requires proper dependencies.

>>The user needs to convey the information that 'foo/b' requires patching,
>>so such a declaration is necessary. The only way I know to represent
>>the declared information in the BBv2 system is by creating a main target.
> 
> 
> Since we're speaking about the right way, not what's possible currently, I 
> might imagine something like
> 
> patch-tree patch-foo : foo patches ;
> 
> So, when the user gets new patches, he would run 
> 
> bjam patch-foo ;
> 
> and the sources will be updated.

Yes,..in fact this is the high-level rule that I already use, and Dave 
proposed the same solution. But it's desirable to have a lower-level 
single-file patch rule as well that doesn't depend upon a tree 
structure...there may be occasion to patch a file in some other context 
and you wouldn't want to have to build a single-file tree just for that 
purpose.

So "patch-tree" can be implemented in terms of "patch-file". For 
modularity and maintenance reasons, this is desirable. Thus with the 
"patch-tree" high-level rule, the individual files are not expressed 
explicitly, but they would still end up as main-targets if "patch-tree" 
is implemented with "patch-file". Of course, you could implement 
"patch-tree" as its own updating rule, but that would defy modular 
design and present maintenance and extensibility issues.

> The situation where "foo/b" is both a source and a target resembles me what I 
> had with gettext support (that I've did locally for V1). The translations 
> file is converted into machine form. But the translation files is itself 
> updated from sources. The way I did it, translation files was updated from 
> sources every time, and that was troublesome: I don't want them updated 
> unless I'm willing to correct translations.
> 
> So, I've decided that in V2, translations will only be updated after explicit 
> request. In the case above, it probably makes some sense too. Imagine:
> user has locally modified "foo/b", and new version of patch for the file 
> arrives. He rebuilds and "foo/b" is patched. Original version is stored.
> Another version of patch arrives, he does the same again, and now the original
> version of file is gone. I'm not sure this is real problem 

Actually, this is a real problem because patches are always stored as a 
patch from the original file in the tar. The solution is that if a 
patch changes, the file must be re-extracted and then re-patched.

This actually presented some issues with BBv2 as well, which I had to 
work around. The extract action has to depend upon the patches (so that 
if a patch changes, the original files get re-extracted).

At one time, I asked why feature <dependency> can't refer to a plain 
file (rather than just main targets) and you asked me for a use case. 
This is the use case. I need to say something like

AllPatches = foo/b foo/d

extract foo : foo.tar.gz : <dependency>patches/$(AllPatches) ;

But, as I now know, <dependency> isn't intended for establishing such 
relationships, so I had to make all patches into main targets. And 
that's another (bad) reason why the patches need to be main targets.

This use case illustrates why I propose the following requirements on 
the system:

(1) A source file should be usable wherever (almost?) a main-target or 
intermediate file is usable. In particular, you should be able to 
express a dependency on either a main-target or a file.

(2) Source files can be in subdirectories (or sibling directories, or 
any directory for that matter); they should be expressable naturally 
using pathnames with slashes in them. N.B. This patching business is 
not a degenerate example either; of the C++ libraries I have from my 
users, a majority of them use subdirectories in a single project to 
organize their source files for a single library.

(3) Consequently, Target IDs should support references to both 
main-targets and files, and should support pathnames and ids with slashes.

>>Also, the extracted or patched file must be a main target because other
>>external projects will want to use these extracted and patched files and
>>there's no way to properly refer to them if they are not main-targets.
> 
> 
> That's an argument! I still wonder, however:
> 1. Why other project need to access patches files. Can you explain in more 
> detail?

Yes. In my modular structure, I have one kind of module ("Package 
Module") that is responsible for unpacking and patching elements of a 
tarfile and another kind of module ("Software Module") that is 
responsible for building libraries and/or executables. A Software 
Module doesn't care whether its source files come directly from the 
repository or whether they come from a Package Module. Thus, to build a 
library from source that comes from a packaged tarfile, a Software 
Module must reach into the Package Module to obtain the final sources 
(extracted and patched).

In BB terminology, Package Module and Software Module are different 
kinds of "project". And they are not necessarily related (although 
currently, the related Software Modules belong to the Package Module).

> 2. The same effect can be achieved by subprojects: i.e. placing Jamfile in foo 
> and declared main target "b" there. It might not be ok for your use case.

Correct..it is not OK. I don't have control over "foo" or its 
contents...that comes straight from an unmodified third-party tarfile.

> The 
> problem is that "foo/b" looks like you declare a target in different 
> directory or different project.

Aha! I agree with you completely, but I believe this is a problem with 
target id syntax.

> Further, there's another question: if you 
> possible have (using @ to separate target name)
> 
> location_at_foo/b
> location/foo_at_b
> 
> is there potential for confusion?

I don't think there's confusion.

The first one is target "foo/b" in project "location/Jamfile" or file 
"foo/b" relative to project "location/Jamfile" (i.e. "location/foo/b").

The second one is target "b" in project "location/foo/Jamfile" or file 
"b" relative to project "location/foo/Jamfile" (i.e. "location/foo/b").

This seems crystal clear to me. I don't think that it is a problem that 
file "location/foo/b" can potentially be referred to in several ways 
(also "._at_location/foo/b"), implying membership of a source file in 
multiple projects at once. It would be unusual to have 
"location/Jamfile" AND "location/foo/Jamfile" AND be referring to file 
"location/foo/b" as a source of both projects, but, on the other hand, 
it is conceivable that you would WANT to have a source serve two 
separate projects and the notation is crystal clear.

> 3. Do users define all those main target manually? If not, probably the best 
> solution is to define one main-target to apply all patches, and make external
> targets depend on specific file, not on main target. That would still require 
> separating target and project with something else than "@".

Currently the targets representing the source files (patched or 
unpatched) are listed manually. There really is no way around 
this...someone has to indicate what the source files of interest are. 
The patches, on the other hand, are stored in a tree, so they are 
detected and applied automatically.

Your solution is along the lines of what Dave proposed. Unfortunately, 
they probably all need to be main-targets because all of the "source 
files" are technically built targets (technically the "sources" of the 
Package Module are the tar.gz file and the patches). The interesting 
part, however, is that if the external project can simply refer to them 
as "package_at_foo/b", the external project does not need to know whether 
"foo/b" is (a) a plain file, (b) a main-target representing an extracted 
(unpatched) file, or (c) a main-target representing an extracted and 
patched file.

--Ali