|
|
Boost Testing : |
From: Aleksey Gurtovoy (agurtovoy_at_[hidden])
Date: 2005-04-09 23:18:31
John Maddock writes:
>> The fact that the change would likely have gone unnoticed if I hadn't been
>> working on some updates for a client and _known_ that the program
>> had compiled 26.5 hours ago, and failed 16 hours ago and could chase
>> down who'd changed what is rather disturbing.
>>
>> We _must_ make some changes in how the regression tests are run (or
>> perhaps how the results are analyzed) if we ever hope to have
>> reliable stuff going out the door.
>>
>> I believe this throws into serious question whether it is reasonable
>> to expect to do a release of a new version of boost in the immediate
>> future (starting a freeze in 7 days).
>
> I agree, I've just looked into some apparent regex regressions, for
> example this one: http://tinyurl.com/4g79r was actually fixed in cvs
> TWO MONTHS AGO! Looking at the old style regression logs that Martin
> is still posting here:
> http://boost.sourceforge.net/regression-logs/cs-Linux.html shows that
> test is actually passing on his system, but the MetaCom postprocessing
> is showing hopelessly out of date results.
And why exactly do you think it has something to do with our
"postprocessing"? The page clearly states that report has been
generated on "Sat, 09 Apr 2005 00:27:16 +0000", while the results'
time for compiler output is "2005-02-04 15:01:16 UTC". It takes
exactly 3 minutes to download and unpack the latest Martin's XML
from "ftp://fx.meta-comm.com/boost-regression/CVS-HEAD/" and see for
yourself that the output shown on the results page is precisely the
output recorded in the XML file Martin uploads to us.
>
> Sorry guys, but something is broken somewhere, and has been for a
> while by the looks of it, it's currently pretty pointless trying to
> fix any regressions until we know that any changes will actually be
> reflected in the posted results...
Post the results that are up-to-date, and the reports will reflect it.
-- Aleksey Gurtovoy MetaCommunications Engineering