File locations (was: Directory Structure)

From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2000-12-14 09:40:30

Here's my $0.03:

For the most part, I agree with Lois.  I posted something sorta similar
back in October (see http://www.egroups.com/message/boost/5598).  Having
standardized directory names in each library's tree is Very Important (and
that's now part of the guidelines, but more on that below).

However, there is one issue that I disagree with: having library .hpp
files in the top-level boost directory.  Having single .hpp files for an
entire library is not a bad idea (I won't take sides here; this is a
separate issue) -- I think that having them all in the top-level directory
is not Good.  Jens pointed out that this will grow to be unmanageable -- I
agree; this does not seem to be suitable for the top-level boost-x.y.z
directory, especially since Beman has pointed out that he expects Boost to
grow much larger than it currently is.

One idea that might solve many of the issues being discussed is having a
"make install" -- the idea of separating the source tree from the tree
that contains the final .a/.so and .hpp files.  I'll call this latter tree
the "install" tree.

The install tree does not have to have the same directory structure as the
source tree.  Indeed, this is a very common thing to do.  As such, each
library author can have *everything* -- including their .hpp files --
under src/libname (or libs/libname, or whatever).

For example, Lois' model was mostly good (although I'd name the top-level
directory "src", not "boost"):

boost_x.y.z/			// Level 0
  src/				// Level 1
    regex/			// Level 2
      src/			// Level 3
      doc/
      example/
      test/
    threads/
      src/
      doc/
      example/
    // ...etc.

It hasn't been decided yet what a top-level "make" would do, but I think a
top-level "make install" could copy all the relevant files to the install
tree (when I say "make install", feel free to replace that with "sc_make
install" or whatever build tool is finally used, or even "./install.py"!).
Indeed, this should even be fairly easy to do portably, since it's mainly
just copying files.  The user can specify the install tree with some kind
of command line argument, such as "make install prefix=/usr/local/boost",
for example.

Note, however, that neither the source nor install trees are necessarily
where the .a/.so files are built.  Hence, there can be three trees
involved:

- source tree (i.e., expansion of the tarball)
- build tree (where you build the .a/.so files)
- install tree (where the .hpp and .a/.so files are placed for general
  consumption)

(skip the VPATH rationale below if you're already familiar with the
concept)

This model is particularly useful when you download a tarball and need to
build/install it for multiple architectures/compilers/options.  Instead of
having multiple copies of the source tree and compiling each of them
separately, you can have a single tree that can be [simultaneously!] used
to build and install multiple targets.  This is *extremely* handy,
particularly for system administrators who have to deploy packages on a
wide variety of platforms/compilers/etc.

Of course, not all make's support such things.  But the design of boost
should not preclude users who have this capability from using it.

Using VPATH (or VPATH-like) builds, the build tree is more-or-less the
same directory structure as the source tree, and is mostly handled by the
build tool, so we don't need to discuss that here (at least, not right now
:-).

The install tree can have a very simple, standardized format:

$prefix/
  lib/		- Argument to -L, *.a/*.so files go here
  include/
    boost/	- Argument to -I
      regex/
      threads/
      // ...etc.

Library file tree ($prefix/lib/):
---------------------------------

I would vote that the libraries be named "libboost_libname.[a|so]", such
as "libboost_regex.so", so that users do "-lboost_regex".  This pretty
much guarantees no name collisions with anything else in $prefix/lib/.

You could instead put .a/.so files in $prefix/lib/boost/ and just name
them libregex.a and just do "-lregex".  But this has the disadvantage that
you'd have to supply a second -L argument (-L$prefix/lib
-L$prefix/lib/boost) if there are any other libraries in $prefix/lib/ that
you wanted to use.

Specifically -- we should not preclude the possibility that boost will be
installed with the same $prefix as other packages (see my point below
about users not updating their boost installation often).

Header file tree ($prefix/include/):
------------------------------------

Note that $prefix/include/boost/ could contain the "single header file"
for each library (if the "single header" concept is decided to be Good),
such as regex.h, and threads.h.

$prefix/include/boost/regex/ can contain regex's "other" .hpp files.  As
pointed out previously, this gives the nice granularity model, so that
beginners can do:

        #include <boost/regex.hpp>

while advanced users can do:

        #include <boost/regex/header1.hpp>
        #include <boost/regex/header2.hpp>
        // etc.

Other possible trees:
---------------------

In the GNU model, documentation frequently is installed under
$prefix/share/boost/.  So the .html files could be installed under

$prefix/
  share/
    boost/
      regex/
      threads/
      // ...etc.

A user can point their browser to a
localhost:/$prefix/share/boost/index.html and get to all the
documentation.  As long as .html files are constructed properly with
relative links, moving them to a different tree isn't a problem.

Overall:
--------

This scheme has the following benefits:

- it is well known and well understood by the open source community.
- it provides user friendliness for new and advanced users.
- a single -I and -L argument can be used to get any/all boost components.
- library authors "reclaim" their .hpp files back to their own source
  tree, putting all of a library's files in one place, rather than split
  across two directory trees.  i.e., this is the "consistency" argument.
- a single source tree can be used to install to multiple installation
  trees (e.g., for different compilers, runtime DLLs, etc.).
- it allows the user to have multiple simultaneous installations
  specifying a different prefix for each.
- it guarantees no name collisions, particularly with the .a/.so files.
  This has pretty much already been addressed with the .hpp files by
  putting them in the boost/ subdirectory, but this hasn't really been
  discussed with .a/.so files.
- it is user-friendly.  Many users won't care about [a potentially]
  complicated build process.  They just want to *use* boost.  Separating
  the build/install and the use of boost is important to this concept.
- it is system administrator-friendly.  While boost is still growing and
  evolving, don't forget that many users just want to *use* boost, and not
  participate in all the development.  Hence, you may find that some
  people get new boost tarballs infrequently -- they only install new
  versions when there's something in their version that is broken, or
  there's a new feature that they need.  These users will prod their
  sysadmins to install boost in a central place so that everyone can use
  it.  And the sysadmin wants to spend an absolute minimum of time
  installing a package (trust me).  So the more standard, the better.

Sorry, that last point got to be much longer than I had anticipated.  :-)

(donning my oxygen tank and flame-retardant suit)
Comments?

-----

A few clarifications about the "Directory Structure" section in the boost
guidelines (more/lib_guide.htm):

- The subdirectory "doc" appears to be missing its <code></code> tags.
- The "Required" column uses the word "several" in multiple places, such
  as "If several doc files".  What is the definition of "several"?  Would
  it be better to specify a number, such as "If more than 1 doc file"?
- Or, even better, should it be mandated that *all* doc files (regardless
  of whether there are 1 or 100) go in doc/?  That way, the user will
  always know where to find the docs; be in the top-level or in doc/ --
  they'll always be in doc/.

More to the point -- are these guidelines, or required standards?
There's a blanket statement: "Not every guideline applies to every
library, but a reasonable effort to comply is expected", but that still
leaves things open to interpretation.

I'm not saying that everything must be absolute, but clearing up ambiguous
language would certainly benefit everyone.

-----

Finally -- a note on CVS.  Yes, CVS can be quite evil, particularly when
you want to rename/move directories.  As noted, you can really only do
this properly (i.e., without trashing the history) by directly mucking
around in the CVSROOT.  Since boost is hosted on SourceForge, this can't
be done, since no one can directly access the CVSROOT.

However, the SourceForge folks have shown to be quite accommodating.
I'd be willing to bet that they'll help out -- has anyone mailed them
and asked what can be done about this?  I can't imagine that boost is
the first project to encounter this problem.  And if they don't allow
such thing, perhaps the boost CVSROOT can be moved elsewhere...?

{+} Jeff Squyres
{+} squyres_at_[hidden]
{+} Perpetual Obsessive Notre Dame Student Craving Utter Madness
{+} "I came to ND for 4 years and ended up staying for a decade"