Re: [boost] [testing] Refactoring serialization tests, and possibly others.

From: Robert Ramey (ramey_at_[hidden])
Date: 2005-10-12 14:51:36

David Abrahams wrote:
> "Robert Ramey" <ramey_at_[hidden]> writes:
>
>> troy d. straszheim wrote:
>>> On Tue, Sep 20, 2005 at 05:35:57PM +0200, troy d. straszheim wrote:
>> ...
>> I have one concern about this.
>>
>> Originally, I had larger test modules.  I breaking it down into
>> smaller tests worked out better for me.  But then I wasn't using the
>> unit test framework.  Since then, I see that the unit test framework
>> is a better choice for this.  Each test program attempts to focus on
>> one feature of the library.  But within that test, various ways of
>> using the feature are excercised.  When a bug is discovered, a small
>> section is added to the test so that the things will keep moving
>> forward.  This turns out to be a good fit with the unit test
>> framework and I'm very pleased to see things moving in this
>> direction.
>
> Be aware that if you consolidate the tests into a single executable
> you will only see a single square in the regression test tables.

I'm aware of that.  I expect all the tests to pass.  When one fails, the 
output from the Boost Test shows me which aspect or usage of the feature 
fails so I'm fine with this.  As long as I have one feature per test (or try 
to) the system works great for me.  I will note one think. For my tests 
here, I use the Beman's compiler_status.cpp program which I "upgraded" (and 
loaded into the vault) to show all the results for different build types 
(release/debug/static library/...).  The prepares a gigantic table which is 
very useful to see that some failures are related to a particular build.
>

>> I would like to see the "Getting Started" or installation or
>> whatever it is included a "configuration/system validation phase"
>> whereby running the tests is a normal part of the boost installation
>> procedure.
>
> Yikes!  Installing already takes way too long, in my experience.  Why?

>
> How many such problems have we seen?  Is there any evidence that this
> would do something other than make installation longer for users?

a) OK - make it optional.  Well its already optional so it would be just a 
question of enhancing the documentation to make it easier for new users to 
"validate" the instalation.
b) On a regular basis we have new users who have problems.  This may be due 
the fact they are using a new library (e.g. stlport 4.?) or compiler 
variation (gcc releases more frequently than we do).  So when they ask a 
question about the serialization library, I really need to know if its a 
boost or system configuration issue or its an issue with the library itself.
c) It would spread the testing effort.
d) we would be informed of new anomolies right away.

Robert Ramey