Re: [boost] [Report] 1472 regressions on RC_1_34_0

From: Aleksey Gurtovoy (agurtovoy_at_[hidden])
Date: 2007-01-15 18:00:12

Jeff Garland writes:
> Roland Schwarz wrote:
> > Douglas Gregor wrote:
> >> Boost Regression test failures
> >> Report time: 2007-01-15T04:07:53Z
> >>
> >> This report lists all regression test failures on release platforms.
> >>
> >> Detailed report:
> >>   http://engineering.meta-comm.com/boost-regression/CVS-RC_1_34_0/developer/issues.html
> >>
> >> The following platforms have a large number of failures:
> >>   intel-vc8-win-9.1
> >>
> >> 1472 failures in 62 libraries (74 are from non-broken platforms)
> > 
> > :-(
> 
> Seems like most of the new ones are bogus 

*Some* of the new ones are "bogus", in a sense that while the XML
produced by process_jam_log is incomplete, we can assert with a
reasonable degree of certainty that the test actually does pass (as
it's the case with the test case you quoted). 

We cannot say that for test cases like this, for instance
(http://tinyurl.com/u4o5l):

    Test output: BoostConsulting - algorithm/string - replace / msvc-8.0
    --------------------------------------------------------------------
    Report Time: Mon, 15 Jan 2007 11:23:51 +0000

    Notes 
    [Reporting Tools Internal Error] This test case's XML is
    missing one or more log entries of the regression run's steps
    associated with the test case's type ("run"). Please contact
    reporting tools maintainers about this problem.

    Compiler output [2007-01-14 17:08:58 UTC]:

        call "C:\Program Files (x86)\Microsoft Visual Studio 8\VC\vcvarsall.bat" amd64 >nul
    cl /Zm800 -nologo @"Z:\results\boost\bin.v2\libs\algorithm\string\test\replace.test\msvc-8.0\rls\adrs-mdl-64\instr-set-optrn\thrd-mlt\replace_test.obj.rsp" 

    replace_test.cpp

In fact, it's very likely that, despite the absence of any errors in
the output, this one actually fails. The reason why it is likely to be
the case is articulated in the note: the test case is declared as a
"run" test, yet the XML does not contain any output for steps that
follow the test compilation.

If you browse through the newly reported "Internal Error" regressions
in a few different libraries, you will see that most of the actually
into "most likely broken" category. 

> -- here's one from date-time that looks like it ran just fine, but
> something is wrong with the XML that's 'causing the failure'

The XML is missing the output from the "compile" step.

> -- as you can see, the test actually passed.

True. I'll see if we can sort these out easily before process_jam_log
is fixed to produce correct XML.

-- 
Aleksey Gurtovoy
MetaCommunications Engineering