Re: [boost] date_time -> serialization (Was: spirtit -> serialization)

Subject: Re: [boost] date_time -> serialization (Was: spirtit -> serialization)
From: Andrey Semashev (andrey.semashev_at_[hidden])
Date: 2014-06-16 13:25:08

On Monday 16 June 2014 11:42:58 Julian Gonggrijp wrote:
> Andrey Semashev wrote:
> > If the tool is able to extract the information from the headers then why
> > do we need the config files? We should minimize the amount of information
> > to be managed by developers - to just the "optional" annotations.
> 
> I agree with this sentiment, but we need a way to cache the full
> dependencies that doesn't require us to change the entire structure of
> Boost.

Ok, I just don't think that library repos are a good place for this cache.

> > But this
> > doesn't mean this list has to be stored in git and managed by developers.
> > You can employ the approach taken by most package managers (e.g. in Linux
> > and OS X ports, I think). There are downloadable packages (which would
> > correspond to sub-modules or links to their git repos) and auto-generated
> > metadata to help resolving dependencies _prior_ to downloading anything.
> 
> Why emphasise "prior"? The user requests module X so I download X
> anyway. Then I can look up what X depends on, whether those data are
> summarized within X or somewhere outside it.

This is a common courtesy of package managers and installers. The tool should 
state what it's going to download and install (and often this alerts the user 
enough so that he cancels). This is a useful protection against undesirable 
situations like "let me install that one library... oh, now it installs half 
of Boost for no apparent reason."

> Where do you store the metadata if not within the module? I like the
> idea of taking automatically generated data out of the versioning
> system, but it should be minimally invasive.

I'm specifically not restricting this part, other than that metadata has to be 
available without downloading the submodule. As a simple example, an ftp 
server should be enough to set up a Boost distribution repository. The 
metadata is stored in a (compressed) text file or several files (better in one 
file though to speedup its downloading). The tool is able to download this 
file and build the dependency graph upon user's request before installing 
anything. If you feel git or another VCS suits better for this metadata, you 
can use it instead, but I don't see much value in version controlling for this 
data, and VCSs seem to add quite some overhead.

Another reason I want to separate the metadata from git repos - and I'm 
fantasizing now - is that I can see this tool being used without git at all - 
to download source packages and install Boost on the user's machine. For 
example, if I want to install a subset of Boost 1.57 on my machine, I'd like 
to be able to do that easily, without dealing with git submodules and without 
downloading the whole git history. The tool will just resolve dependencies, 
download and extract a set of archives for me.

> > The metadata should be
> > automatically updated when the packages are uploaded (i.e. official
> > snapshots are uploaded or a referred git tag is added).
> 
> Do you envision this in the current situation where "packages" are
> loaded as sub-modules of the boost super-project, or in a new
> situation where the boost super-project is taken away and "packages"
> are standalone (but with dependencies)?

What I meant is that there should be some kind of "official" Boost repository 
which will be used by the packaging tool (let's call it boost-pkg for 
brevity). That repository will serve the metadata for boost-pkg. The metadata 
will be updated when a certain new release is published into it, whether that 
is a new library release through a git tag or the whole Boost release. I'm not 
sure if updating the metadata upon a tag creation can be automated, but at 
least for major Boost releases this should be doable.

Continuing my fantasy, the repository may contain standalone packages for 
library and Boost releases, which may be useful for Boost users (in the above 
example, I would be downloading archives of Boost 1.57 from this repository). 
It may also contain references to git repositories - tags or branches. boost-
pkg would offer a unified interface so that it is possible to use either of 
them - e.g. download official Boost 1.57 and a newer release of that one 
library X, which fixes a critical bug for me.

Potentially, boost-pkg could replace the superproject and be used to checkout 
the whole Boost from git on develop or master branch, which would be useful 
for developers and testers. For testers this would help to checkout the tested 
library from develop and everything else from master. Although it is also 
possible to do with plain git, when you have checked out everything.

> In a nutshell, both for users and maintainers:
> 1. Clone the superproject non-recursively.
> 2. Request specific modules to be installed by Boost.Build;
> dependencies are tracked by the handler tool which uses git to clone
> more modules.

That would require at least Boost.Build to be checked out as well.

> In addition, for maintainers:
> 3. Create/update the configuration file by running the handler tool
> before pushing to the public repo (this could be a git hook).

This would be a blocker, for me at least. I'm sure I will forget doing that 
and will be very much annoyed. A git hook that scans the headers on every push 
doesn't sound very good.