[Boost-testing] Possible switch to Boost.Build V2

From: Vladimir Prus (ghost_at_[hidden])
Date: 2006-02-21 10:11:38

Hello!

As many of you know, the Boost.Build system version 2 is in development for
quite some time, and it becomes desirable to switch C++ Boost to V2. I
would like to explain where we stand with this project.

I've spend last few weeks making sure that there are no technical issues in
V2 preventing tests to work, and that tests for all libraries are
up-to-date. Specifically, I wrote a script that runs regressions with V1
and V2, and compares the results. At the moment, on gcc/Linux the results
are identical.

Two volunteers on boost-build mailing list were running tests on
msvc/Windows. We're also getting close. Except for Python tests, that Dave
will hopefully look into, we've just two unresearched differences.

However, I think it becomes impractical to continue this process inside 
boost-build mailing list. While it would be good to verify that V2 produces 
the same results as V1 on *all* compilers currently in use, folks on 
boost-build don't have that many versions at hand. And obviously, only the
results produced by regular regression runners matter when it comes to
working on release.

I think that the best thing now is to gradually move regular regression
tests to V2. In my opinion, the ideal process will looks like this:

  - One regression runner switches to V2
  - Next day I look at all test failures and classify them as "V2 bug",
    "library bug", or "unclear". I'll work with library authors to resolve
    bugs from the third category and will fix the V2 bugs.
  - Another regression runner switches to V2

With such a process, any given regression runner will only have to set up
Boost.Build V2 once, which will not take much time, and after that I'll be
mostly working with library authors. The plan that when one person switches
to V2 he won't be running V1 tests anymore, so the amount of required disk
space won't double. 

I've talked with Thomas Witt, release manager for 1.34, and he does not mind
the switch for 1.34.

So, what do regression runners think of this idea? 

- Volodya

The comparison work we've done before guarantees that there won't be big 
issues on gcc and msvc. We might get problems on other toolsets, but I'd 
expect them to be fixed quickly, because:

  - The most time-consuming task was syncing V1 and V2 Jamfiles. It's
already
    done.
  - Global configuration problems can be fixed soon. As example, the HP-CXX
    (formely Tru64) toolset for V2 went from non-existance to fully passing
V2
    internal regression tests in less than a week. 
  - Failures on specific tests can be solved as part of regular bug triage
    that's done before release. As example, the first run of msvc/Windows
    tests was done on Feb 13, and one week again we're mostly in shape.
    I would expect fixing failures for any other toolset to take
considerably
    less time.

Because of greadual switch, at any given time only one regression runner
will 
be in "V2-switch" mode. All other regression runners can be used to resolve 
bugs in libraries.

It means that at the works case, any given regression runner will be 
unavaiable for regular use for one week. And we have just one toolset that 
never seen any user -- qcc. For all others the switch to V2 probably won't 
add noticable time to the release process. Typically, release process takes 
longer than expected, because library developers don't have the time to fix 
bugs in their libraries. This gives us extra time to fix V2-related issues.

There are some good reasons to switch to V2 for this release. The biggest
one 
is that separation between V1 and V2 fractures developement effort and user 
base. 

Another reason is that getting V1 and V2 Jamfiles in sync is very time 
consuming task. If we decide to make the switch after release, or for the 
next release, it will mean this task will have to be repeated. 

So, what is your opinion about switch to V2?