|
|
Boost Testing : |
From: Aleksey Gurtovoy (agurtovoy_at_[hidden])
Date: 2007-01-17 21:50:00
Hi Thomas,
> I was trying to get a picture from reading the list, but I've to
> admit I am lost. It would really help if you could give me a short
> rundown of events, i.e. what was broken, what got fixed and the
> current status, i.e. are we back to normal or should I wait to get
> panicked in face of the gazillion failures.
Sure:
1. Regressions were switched to Boost.Build V2.
2. The switch brought in numerous changes to the bjam output format,
breaking the "bjam.log -> proccess_jam_log -> <runner>.xml ->
report pages" chain at the very first step.
Vladimir worked on fixing proccess_jam_log to account for these
changes before the switch, but since we don't have any test cases
for this, verification of his fixes was basically reduced to
ironing out differences between V1 and V2 results _as displayed by
report pages_. In their turn, the report page generation scripts
did not perform even basic checks of the XMLs produced by
proccess_jam_log, so together these two lead us to believe that we
successfully switched the regression subsystem to V2 when in fact
we didn't.
Specifically, up until two days ago proccess_jam_log was producing
incomplete XMLs for many of the failed tests, which were then
rendered as passing in the report pages
(http://article.gmane.org/gmane.comp.lib.boost.devel/153279).
3. Roland Schwarz brought to our (Meta's) attention an inconsistency
in our Borland results, researching which forced me to take a deep
look at proccess_jam_log <-> reporting tools interaction for the
first time since V2 switch, realize that we have an unknown number
of regressions slipping by, and insert the corresponding
diagnostics into report generation scripts -- which resulted in the
current flood of reported regressions.
Now, as Jeff correctly noted in
http://article.gmane.org/gmane.comp.lib.boost.devel/153271, some of
the test cases that are currently reported as regressions, while
not having output from all the required steps, can be with a
reasonable degree of certainty asserted as passing
(http://article.gmane.org/gmane.comp.lib.boost.devel/153279). The
fix that clears these up is in the CVS now, meaning that in a
couple of hours we should see some reduction in the number of
regressions.
The rest of them are either actual regressions, passing tests with
(old) undecidable XMLs, or a side-effects of remaining bugs in
process_jam_log -- we will be able to see which are which as soon
as we recieve a fresh set of results (processed by the updated
process_jam_log) from every runner.
Hope this clarify things!
-- Aleksey Gurtovoy MetaCommunications Engineering