Re: [boost] [testing] Refactoring serialization tests, and possibly others.

From: Robert Ramey (ramey_at_[hidden])
Date: 2005-09-25 22:36:07

I should note that the serialization library changes only very 
infrequently - maybe once every few weeks.

I would question re-runing the serialization library tests just because 
something that the serialization library depends upon changes.  Certainly 
this doesn't do me any good.  If someone breaks/changes and api that the 
serialization library depends upon, does that person even find out about it? 
Much less track it down?  Not that this is a big deal as it almost never 
happens.  The last time it happened that I remember was when all the 
serialization tests for comeau started to fail when something in Boost Test 
changed.  That was 6 months ago and nothing has changed.

A main cause of this problem bjam dependency analysis re-runs all tests on 
Library X even if library X hasn't changed.

But boost has other scalability problems.  The testing/release regimen 
permits libraries to be more tightly coupled than they probably should be.

Ideally I would like to see something like:

a) Each maintainer has his own branch.
b) He makes changes on his own machine
c) He tests his changes on his own machine.
b) He makes changes on his branch.
c) Whenever he makes a change, tests are run on that branch for that 
library.
d) When he is statisfied with c) above, he merges changes from his branch 
into the main trunk.
e) Whenever d) occurs, tests are run on the main branch for any libraries 
newly merged in.
f) The health of the main trunk "should" be very good.  Periodically, and 
much more frequently than now, (maybe every 60 days or so), Exhaustive tests 
are run on the trunk - just as they are now.  Any problems should be small. 
These are fixed and a new release is made - zip an tarbal.

This would

a) diminish requirements for testing immensly.
b) testing resources would not be wasted on repeating the same tasks.
c) library authors could not rely on other libraries to test their code.  I 
know no one does this intentionally, but we're all human, and w're under 
pressure and writing tests IS a pain and, etc....
d) tests would by necessity have to be more complete and focused. 
Presumable these would build incrementally.
e) result in much more frequent new releases.  If a bug is found, it would 
be practical to just wait for the next release to fix it rather then trying 
to "stop the presses"
f)Errors would be easier to find.  Since we're all testing our changes 
agains the trunk rather than everyone elses changes any error during 
development has to be library writers.  An error that shows up when changes 
are merged to the trunk has to be due to an API change in another library.
g) Authors wouldn't be under pressure to get feature Y crammed in to make 
the next release date.  Each would work at his how pace knowing that shortly 
after its ready, it will appear in the official release.
h) Author's wouldn't have to hold back changes that they think are OK 
because they just might throw a monkey wrench into the whole works.

It seems to me that this should be the main benefit of the source control 
system and that we are not receiving that benefit.

Robert Ramey

Rene Rivera wrote:
> David Abrahams wrote:
>> "troy d. straszheim" <troy_at_[hidden]> writes:
>>
>>> Rene Rivera wrote:
>>>
>>>> One thing I should make clear about my suggestion to reduce the
>>>> combinations... Is that it doesn't mean reducing the number of
>>>> types that get serialized, rather increasing those and having more
>>>> complex and comprehensive test.
>>>
>>> I need to think about it more... for now, I carpet bomb.  But it
>>> clearly would get out of control as types, archives, platforms and
>>> versions proliferate.   No question.
>>
>> It sounds like, from what Rene has said, it already is out of
>> control.
>
> I did another incremental test run today which spent most of the time
> retesting serialization, probably some minor change. But it ran from
> 10:30am to 8:00pm, 11.5 hours. So yes, I consider that out of control.
>
>> Anything that makes it very difficult and/or expensive for a tester
>> to complete a testing run needs to be fixed rather urgently.  I hope
>> you realize that, and I think you probably do, but from your posting
>> it wasn't clear.
>
> I hope everyone realizes this.