Re: [boost] Breaking existing libraries

Subject: Re: [boost] Breaking existing libraries
From: Johan Nilsson (r.johan.nilsson_at_[hidden])
Date: 2008-11-24 03:10:19

Vladimir Prus wrote:
> Johan Nilsson wrote:
>
>> Sure, there are many potential problems with such an approach:
>>
>> - Some library tests are likely to also excercise code at the
>> implementation/detail level.
>> - Already limited testing resources (on the other hand, interface
>> changes are likely to be detected by running such tests for a
>> single, reasonably compliant, compiler).
>> - ...
>>
>> Anyway, this is mostly me thinking out loud; perhaps someone smarter
>> than me could expand on the feasibility of such an idea.
>
> I hope somebody not claiming to be smarter than you is also allowed
> to comment ;-)

Ok, I expressed myself badly. Disregard the "smarter" stuff (even though I 
don't think it should've stopped anyone here from commenting).

>
> An approach similar to this is used by the GDB project -- there are
> several versions
> of the interface from GDB to GUI frontends, and there are presently
> basically two sets
> of tests -- one is fixed and tests specific version of the interface.

For Boost, this could be equivalent to the interface as accepted during the 
last public review. How "fixed" are such tests really?

> The second set
> tests all the current behaviour, when it's considered good to go,
> tests for the
> current behaviour will be copied into a new set, and won't be touched
> any more.

Is extending the interface considered equal to breaking the interface?

>
> The primary problem with that approach, in case of GDB, is that it's
> just not practical
> to conditionalize every bit of output on used interface version, so
> the testcases are
> written to tolerate some "good" additional fields of output. And the
> testcases have to
> be modified when new "good" additional fields appear, so the
> maintenance burden increases.

If this would happen without review, it's almost semantically equivalent to 
silently breaking the interface. At least if you are talking about the 
"fixed" tests.

>
> Probably, for C++ Boost this is not an issue, and it's possible to
> freeze the tests of earlier published interface, and don't touch
> them. But for C++ Boost, the performance
> is a real problem. E.g, I suspect that two complete Spirit
> testsuites, for different versions, will be very hard on test
> resources.

Yeah, that might be a problem. However, if limiting 
"interface-regression-testing" to one or two well-conforming compilers 
running on modern hardware, it might not be such a big problem.

/ Johan