Re: [boost] [Boost.Test] New testing procedure?

From: Toon Knapen (toon.knapen_at_[hidden])
Date: 2004-05-24 08:54:46

Aleksey Gurtovoy wrote:

>>If Gennadiy can't test on more compilers before checking in, and
>>respond faster to problems introduced in its source, and if library
>>authors genuinely find Boost.Test useful, maybe we need to move to a
>>different model wherein the test library's own tests are run on a CVS
>>branch of the code (?) so that Gennadiy can see and deal with his
>>problems before they are merged into the main trunk and break
>>everything else?
> 
> 
> The other regression runners have to speak for themselves, but we can
> definitely setup that here at Meta. As long as it's only the test and 
> config libraries which will be tested on the branch, we have enough 
> compilation resources to do that while preserving the twice-daily cycle 
> for the main trunk (or the release branch). 
> 
> IMO we should go for it.

OK for me too. I suppose this is not too difficult to automate too.

Before going furhter let me tell you that I'm the guy running the 
regression tests on IBM, SGI and HP. There was some discussion on why 
thet were not update daily and some guessed that they were not 
automated. First of all: they are automated and normally run daily by 
means of a cron job.

So why have'nt they run for quite some time now: well I fail to compile 
the regression test tools (exactly the reason why this thread started) 
on all platforms all of a sudden (well not all of a sudden as I don't 
verify the results every day). I'm currently in contact with IBM and HP 
to help me out and find workarounds or compiler updates to compile these 
tools succesfully. But this takes time.

If you have questions about the regression results (and since I don't 
verify the results daily), you can always contact me (that's why I've 
put my name on the IBM regression test page for instance) and I will try 
to help you as good as I can.

To answer another question in this thread: no, running the regression 
tests does not eat to much resources. They run every night as you also 
can see on the IBM regression test page.

A big problem IMHO though is that few authors verify the regression 
status on all the different platforms. How could one otherwise explain 
that no one ever contacted me (even during this discussion) directly 
regarding some questions he might have regarding the regression tests 
(although my name is clearly visible on the status page for that sole 
purpose). This suggests that not many have ever taken a look at the 
status page for IBM/VisualAge. Actually only one developer ever 
contacted me asking for help to port his library succesfully to 
IBM/VisualAge. (don't get me wrong, I don't want to criticise boost 
developers who are all doing a great job but I'm just trying to indicate 
that it's not easy either to port the libraries).

And again, I don't have the time every day to verify the regression 
tests and to read the ml (due to workload and/or traveling). So I 
usually find out if something breaks a week or two after the 
modification is made. And at that point, it's very difficult the exact 
reason why the test breaks almost having to reverse all changes until 
the situation where it worked and from then on trying to find a workaround.

toon