[boost] Re: [Boost.Test] New testing procedure?

From: David Abrahams (dave_at_[hidden])
Date: 2004-05-21 20:00:49

"Rozental, Gennadiy" <gennadiy.rozental_at_[hidden]> writes:

>> According to http://tinyurl.com/3f32c tests have been failing 
>> (IIUC for several days if not longer) because of 
>> protected/private access violations in the test library.  See 
>> the end of http://tinyurl.com/35ole for details.  IMO this 
>> situation is unacceptable, and I'm looking for practical 
>> solutions.  Problems with the Boost.Test affect our ability 
>> to get meaningful feedback on the state of other libraries.
>> 
>> If Gennadiy can't test on more compilers before checking in, 
>> and respond faster to problems introduced in its source, and 
>> if library authors genuinely find Boost.Test useful, maybe we 
>> need to move to a different model wherein the test library's 
>> own tests are run on a CVS branch of the code (?)  so that 
>> Gennadiy can see and deal with his problems before they are 
>> merged into the main trunk and break everything else?
>
> I do test on as much compilers I have accessible.  

I'm sure you do.  That hasn't stopped problems in Boost.Test from
becoming a problem for the rest of Boost (several times at a critical
moment for a release), so as I said I'm looking for practical
solutions.

> In this particular case
> It was working on all compilers but Borland. 

??! The links I posted were for problems uncovered by Comeau.  Did you
look at http://tinyurl.com/35ole?  Several other compilers also detect
the issue.

> I posted request for workaround while ago. Still no response.

I could be wrong, but AFAICT there's a bug in the library.  It
shouldn't be up to others to find workarounds for that.

> I had to test other configuration and I committed it. Actually I was
> surprised that so many compilers had an issue with completely
> innocent using declaration. Anyway I think it should be fixed as of
> last night. 

Not according to the links I posted *this morning*.

> I found some hack that should work on complaining compilers. I will
> see the results of regression test today.  As for creating separate
> brunch for Boost.Test development, I do not really mind. But I
> believe it will create an extra headache for regression testers (and
> me). 

Not if it's automated.

> Essentially we will need to have two copies of development
> tree. 

I doubt it.  How many libraries does Boost.Test depend on?  Are you
sure we can't just do this with branched copies of boost/test and
libs/test?

> And run Boost.Test unit test in a separate tree. I will need
> to keep moving files back and forth 2 branches. 

Better to take the effort to be responsible for not breaking things
than to develop quickly and dangerously when your library is a
correctness bottleneck for so many others.

> Also I wonder how it will interact with release procedure.  You may
> noticed that I am trying to introduce modification in Boost.Test in
> "packages", meaning I do not d code modifications all the time. I am
> not sure that several days of "no regression test on some
> compilers", worth all that trouble. 

I think you underestimate the problems it causes, and probably also
how long the particular problem in question has existed in the
codebase (I think it's been over a week).  Many people have been
talking about trying to shorten the time it takes to get test results
for any particular library.  To go from 12-24 hours to "several days"
is probably the wrong direction.

> Especially since all the developers could always rollback Boost.Test
> modification locally for development testing.

It makes a lot more sense to me that the author of one library should
do a small amount of extra work to stay out of the way of everyone
else than to make all the other library authors adapt to Boost.Test's
problems.  More importantly, that's not even possible for the test
results library authors want to get from compilers they don't own.

-- 
Dave Abrahams
Boost Consulting
http://www.boost-consulting.com