|
|
Boost Testing : |
From: Martin Wille (mw8329_at_[hidden])
Date: 2005-04-10 02:34:59
Aleksey Gurtovoy wrote:
> John Maddock writes:
>
>>>The fact that the change would likely have gone unnoticed if I hadn't been
>>>working on some updates for a client and _known_ that the program
>>>had compiled 26.5 hours ago, and failed 16 hours ago and could chase
>>>down who'd changed what is rather disturbing.
>>>
>>>We _must_ make some changes in how the regression tests are run (or
>>>perhaps how the results are analyzed) if we ever hope to have
>>>reliable stuff going out the door.
>>>
>>>I believe this throws into serious question whether it is reasonable
>>>to expect to do a release of a new version of boost in the immediate
>>>future (starting a freeze in 7 days).
>>
>>I agree, I've just looked into some apparent regex regressions, for
>>example this one: http://tinyurl.com/4g79r was actually fixed in cvs
>>TWO MONTHS AGO! Looking at the old style regression logs that Martin
>>is still posting here:
>>http://boost.sourceforge.net/regression-logs/cs-Linux.html shows that
>>test is actually passing on his system, but the MetaCom postprocessing
>>is showing hopelessly out of date results.
(FWIW, John didn't claim the error is in the postprocessing at your
site, Aleksey. He only said the symptoms become visible there)
> And why exactly do you think it has something to do with our
> "postprocessing"? The page clearly states that report has been
> generated on "Sat, 09 Apr 2005 00:27:16 +0000", while the results'
> time for compiler output is "2005-02-04 15:01:16 UTC". It takes
> exactly 3 minutes to download and unpack the latest Martin's XML
> from "ftp://fx.meta-comm.com/boost-regression/CVS-HEAD/" and see for
> yourself that the output shown on the results page is precisely the
> output recorded in the XML file Martin uploads to us.
>
>
>>Sorry guys, but something is broken somewhere, and has been for a
>>while by the looks of it, it's currently pretty pointless trying to
>>fix any regressions until we know that any changes will actually be
>>reflected in the posted results...
>
>
> Post the results that are up-to-date, and the reports will reflect it.
>
I think, for John as for other developers, the term "postprocessing"
includes everything that is run after the actual tests were run.
Especially, this includes creating XML and collecting the XML results.
Technically, creating the XML is part of the build system and collecting
the results is part of local postprocessing at the testers' sites.
Obviously, one of those two steps went wrong, since the old-style
results look correct.
I have deleted a couple of test results which looked outdated for
various reasons. So, the symptoms should go away for now. However, the
cause stays unknown (and I currently don't have enough time to
investigate this thoroughly).
In the course of this cleanup, I also found out that several tests were
moved, again without notifying the testers. I will stop complaining
about this since the complaints apparently didn't help at all. Please,
understand that I'm unable to sort these things out every day.
Regards,
m