Re: [boost] [testing] Refactoring serialization tests, and possibly others.

From: David Abrahams (dave_at_[hidden])
Date: 2005-10-12 12:10:45

"Robert Ramey" <ramey_at_[hidden]> writes:

> troy d. straszheim wrote:
>> On Tue, Sep 20, 2005 at 05:35:57PM +0200, troy d. straszheim wrote:
> ...
> I have one concern about this.
>
> Originally, I had larger test modules.  I breaking it down into smaller 
> tests worked out better for me.  But then I wasn't using the unit test 
> framework.  Since then, I see that the unit test framework is a better 
> choice for this.  Each test program attempts to focus on one feature of the 
> library.  But within that test, various ways of using the feature are 
> excercised.  When a bug is discovered, a small section is added to the test 
> so that the things will keep moving forward.  This turns out to be a good 
> fit with the unit test framework and I'm very pleased to see things moving 
> in this direction.

Be aware that if you consolidate the tests into a single executable
you will only see a single square in the regression test tables.

> Running the serialization tests for one compiler typically takes
> about an hour.  So it is a lot of time - but its not totally out of
> control - but it is heading there.
>
> There are a number of ideas on how to deal with this which I won't rehash 
> here except for one.
>
> I would like to see the "Getting Started" or installation or
> whatever it is included a "configuration/system validation phase"
> whereby running the tests is a normal part of the boost installation
> procedure.

Yikes!  Installing already takes way too long, in my experience.  Why?

> This would result in the following:
>
> a) all tests would be run on all platforms actually being used.
> b) Amount of testing would actually be increased by many fold as it would be 
> done by all who installed the package
> c) Variety of testing would be larger
> d) Installation problems would be trapped on installation rather than when 
> users are having problems with their code.

How many such problems have we seen?  Is there any evidence that this
would do something other than make installation longer for users?

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com