Re: [boost] [testing] Refactoring serialization tests, and possibly others.

From: Robert Ramey (ramey_at_[hidden])
Date: 2005-09-26 16:43:59

David Abrahams wrote:
> "Robert Ramey" <ramey_at_[hidden]> writes:
>> So if its not the same discussion but it is related.  Of course we
>> can test less.  But the root of the problem is that probably only a
>> small percentage of the effort invested in testing is actually
>> testing anything.  I know I've brought this up before but made no
>> headway so I won't harp on it anymore.
>
> It's hard to understand your objection to retesting library A in
> combination with a changed library B on which it depends, since that
> is essentially what you're doing with the various parts of the
> serialization with your N x M x K testing.

LOK, I could also say its hard to understand why you don't object to 
re-testing A in combinarion with B but do object to doing the same thing 
with NxMxK serialization library.

But there are some differences.

The serialization library is still one library with one person taking 
responsability for reviewing the results and reconciling any issues.  any 
failure in any combination will be addressed.

The serialization libary is change only once every few weeks whereas 
something in the whole of boost changed at least once / day.  So the impact 
on testing time is much greater from retesting every AxB combination.

Libraries A and B expect to change their public API only very occasionally. 
If the API is being tested independently, then re-testing by other libraries 
should be redundant.   This doesn't apply to the internal behavior of any 
particular library.  Regarding the serialization library in particular,  I 
have strived to make the different aspects indepent and interact with each 
other only through the most narrow of API.  This is the basic motivation for 
splitting the view of things in to archive and serialization.   Similarly I 
strived to make the DLL versions identical in usage to the static library 
usage and all the archives implement the same API.  The reason that we have 
the MxNxKxL(don't forget deug and release !) is that I was successful in 
doing this.  Never-the-less, it has been a struggle and now the test 
results show pretty good othogonality except for compiler quirks which show 
some combinations trip compilers ICE's.   I don't know that I could have 
arrived at this point without testing all the combinations from time to 
time.  Libraries A & B don't have this situation.

Robert Ramey