Boost Testing :
From: Vladimir Prus (ghost_at_[hidden])
Date: 2006-02-21 10:11:38
As many of you know, the Boost.Build system version 2 is in development for
quite some time, and it becomes desirable to switch C++ Boost to V2. I
would like to explain where we stand with this project.
I've spend last few weeks making sure that there are no technical issues in
V2 preventing tests to work, and that tests for all libraries are
up-to-date. Specifically, I wrote a script that runs regressions with V1
and V2, and compares the results. At the moment, on gcc/Linux the results
Two volunteers on boost-build mailing list were running tests on
msvc/Windows. We're also getting close. Except for Python tests, that Dave
will hopefully look into, we've just two unresearched differences.
However, I think it becomes impractical to continue this process inside
boost-build mailing list. While it would be good to verify that V2 produces
the same results as V1 on *all* compilers currently in use, folks on
boost-build don't have that many versions at hand. And obviously, only the
results produced by regular regression runners matter when it comes to
working on release.
I think that the best thing now is to gradually move regular regression
tests to V2. In my opinion, the ideal process will looks like this:
- One regression runner switches to V2
- Next day I look at all test failures and classify them as "V2 bug",
"library bug", or "unclear". I'll work with library authors to resolve
bugs from the third category and will fix the V2 bugs.
- Another regression runner switches to V2
With such a process, any given regression runner will only have to set up
Boost.Build V2 once, which will not take much time, and after that I'll be
mostly working with library authors. The plan that when one person switches
to V2 he won't be running V1 tests anymore, so the amount of required disk
space won't double.
I've talked with Thomas Witt, release manager for 1.34, and he does not mind
the switch for 1.34.
So, what do regression runners think of this idea?
The comparison work we've done before guarantees that there won't be big
issues on gcc and msvc. We might get problems on other toolsets, but I'd
expect them to be fixed quickly, because:
- The most time-consuming task was syncing V1 and V2 Jamfiles. It's
- Global configuration problems can be fixed soon. As example, the HP-CXX
(formely Tru64) toolset for V2 went from non-existance to fully passing
internal regression tests in less than a week.
- Failures on specific tests can be solved as part of regular bug triage
that's done before release. As example, the first run of msvc/Windows
tests was done on Feb 13, and one week again we're mostly in shape.
I would expect fixing failures for any other toolset to take
Because of greadual switch, at any given time only one regression runner
be in "V2-switch" mode. All other regression runners can be used to resolve
bugs in libraries.
It means that at the works case, any given regression runner will be
unavaiable for regular use for one week. And we have just one toolset that
never seen any user -- qcc. For all others the switch to V2 probably won't
add noticable time to the release process. Typically, release process takes
longer than expected, because library developers don't have the time to fix
bugs in their libraries. This gives us extra time to fix V2-related issues.
There are some good reasons to switch to V2 for this release. The biggest
is that separation between V1 and V2 fractures developement effort and user
Another reason is that getting V1 and V2 Jamfiles in sync is very time
consuming task. If we decide to make the switch after release, or for the
next release, it will mean this task will have to be repeated.
So, what is your opinion about switch to V2?