Boost logo

Boost Testing :

From: Aleksey Gurtovoy (agurtovoy_at_[hidden])
Date: 2007-01-17 21:50:00

Hi Thomas,

> I was trying to get a picture from reading the list, but I've to
> admit I am lost. It would really help if you could give me a short
> rundown of events, i.e. what was broken, what got fixed and the
> current status, i.e. are we back to normal or should I wait to get
> panicked in face of the gazillion failures.


1. Regressions were switched to Boost.Build V2.

2. The switch brought in numerous changes to the bjam output format,
   breaking the "bjam.log -> proccess_jam_log -> <runner>.xml ->
   report pages" chain at the very first step.

   Vladimir worked on fixing proccess_jam_log to account for these
   changes before the switch, but since we don't have any test cases
   for this, verification of his fixes was basically reduced to
   ironing out differences between V1 and V2 results _as displayed by
   report pages_. In their turn, the report page generation scripts
   did not perform even basic checks of the XMLs produced by
   proccess_jam_log, so together these two lead us to believe that we
   successfully switched the regression subsystem to V2 when in fact
   we didn't.

   Specifically, up until two days ago proccess_jam_log was producing
   incomplete XMLs for many of the failed tests, which were then
   rendered as passing in the report pages

3. Roland Schwarz brought to our (Meta's) attention an inconsistency
   in our Borland results, researching which forced me to take a deep
   look at proccess_jam_log <-> reporting tools interaction for the
   first time since V2 switch, realize that we have an unknown number
   of regressions slipping by, and insert the corresponding
   diagnostics into report generation scripts -- which resulted in the
   current flood of reported regressions.

   Now, as Jeff correctly noted in, some of
   the test cases that are currently reported as regressions, while
   not having output from all the required steps, can be with a
   reasonable degree of certainty asserted as passing
   ( The
   fix that clears these up is in the CVS now, meaning that in a
   couple of hours we should see some reduction in the number of

   The rest of them are either actual regressions, passing tests with
   (old) undecidable XMLs, or a side-effects of remaining bugs in
   process_jam_log -- we will be able to see which are which as soon
   as we recieve a fresh set of results (processed by the updated
   process_jam_log) from every runner.

Hope this clarify things!

Aleksey Gurtovoy
MetaCommunications Engineering

Boost-testing list run by mbergal at