Re: [boost] [testing] Refactoring serialization tests, and possibly others.

From: David Abrahams (dave_at_[hidden])
Date: 2005-09-26 21:47:38

"Robert Ramey" <ramey_at_[hidden]> writes:

> David Abrahams wrote:
>> "Robert Ramey" <ramey_at_[hidden]> writes:
>>> So if its not the same discussion but it is related.  Of course we
>>> can test less.  But the root of the problem is that probably only a
>>> small percentage of the effort invested in testing is actually
>>> testing anything.  I know I've brought this up before but made no
>>> headway so I won't harp on it anymore.
>>
>> It's hard to understand your objection to retesting library A in
>> combination with a changed library B on which it depends, since that
>> is essentially what you're doing with the various parts of the
>> serialization with your N x M x K testing.
>
> LOK, I could also say its hard to understand why you don't object to 
> re-testing A in combinarion with B but do object to doing the same thing 
> with NxMxK serialization library.

The former is hard to change, and it's not clear that the change is
desirable.

> But there are some differences.
>
> The serialization library is still one library with one person taking 
> responsability for reviewing the results and reconciling any issues.  any 
> failure in any combination will be addressed.
>
> The serialization libary is change only once every few weeks whereas 
> something in the whole of boost changed at least once / day.  So the impact 
> on testing time is much greater from retesting every AxB combination.

Not for those who only do clean test runs.

> Libraries A and B expect to change their public API only very occasionally. 
> If the API is being tested independently, then re-testing by other libraries 
> should be redundant.   This doesn't apply to the internal behavior of any 
> particular library.  Regarding the serialization library in particular,  I 
> have strived to make the different aspects indepent and interact with each 
> other only through the most narrow of API.  This is the basic motivation for 
> splitting the view of things in to archive and serialization.   Similarly I 
> strived to make the DLL versions identical in usage to the static library 
> usage and all the archives implement the same API.  The reason that we have 
> the MxNxKxL(don't forget deug and release !) is that I was successful in 
> doing this.  Never-the-less, it has been a struggle and now the test 
> results show pretty good othogonality except for compiler quirks which show 
> some combinations trip compilers ICE's.   I don't know that I could have 
> arrived at this point without testing all the combinations from time to 
> time.  Libraries A & B don't have this situation.

It's possible to set things up so you run all the combinations but the
regular Boost test suite runs fewer.

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com