Re: [boost] Re: [1.33.0] Let's start preparations...

From: Douglas Gregor (doug.gregor_at_[hidden])
Date: 2005-03-09 01:39:47

On Mar 8, 2005, at 10:39 PM, Victor A. Wagner Jr. wrote:

> At Tuesday 2005-03-08 12:16, you wrote:
>> On Mar 8, 2005, at 4:52 PM, Martin Wille wrote:
>>> Aleksey Gurtovoy wrote:
>>>> Martin Wille writes:
>>>>> - the testing procedure is complex
>>>> Internally, yes. The main complexity and _the_ source of fragility
>>>> lies in "bjam results to XML" stage of processing. I'd say it's one 
>>>> of
>>>> the top 10 issues by solving which we can substantially simplify
>>>> everybody's life.
>>>
>>> I agree. This processing step has to deal with the build system 
>>> (which in complex itself) and with different compiler output. Other 
>>> complexity probably stems from having to collect and to display test 
>>> results that reflect different cvs checkout times.
>>
>> To improve the reproducibility of results and make testing more 
>> predictable, we might want to have the regression scripts always 
>> check out using a given date/time tag, e.g., 12:00am EST each night. 
>> That way, all of the tests for the day will be on the same codfe. If 
>> it helps fix other problems with regression testing, great!
>
> wow, you want me to only run the tests once a day instead of 4 times?  
> Surely, you're not suggesting that I'd get differing results if I 
> checked out using the same time more than once.

Obviously, this is not the case. Looking at the summary page, however, 
we get a view across many CVS states, so it's hard to tell which 
version of the source code we're looking at. The answer isn't to submit 
only once per day (more testing is better, always!), but to have at 
least one build from each tester that references the source code at 
12:00am EST. When a tester is capable of submitting more builds in a 
day, we have the most up-to-date build AND the 12:00am EST build in the 
results, which will make it very easy to see what's we've broken in a 
given day.

        Doug