|
|
Boost Testing : |
From: Robert Ramey (ramey_at_[hidden])
Date: 2007-05-07 11:43:32
Gennadiy Rozental wrote:
> "Robert Ramey" <ramey_at_[hidden]> wrote in message
> news:f1mdto$pif$1_at_sea.gmane.org...
>>>>>> This doesn't address neither testing against new version of your
>>>>>> library by developers of other libraries
>>
>> Developers can test which every they want. Regression testers
>> can test whatever they want. I presume they would
>> want to continue testing the HEAD. This doesn't help me
>> much but if some finds it useful to some purpose, it's fine
>> by me.
>
> Developers quite possibly want to run tests against your new version
> not locally, but using power of regression testing facility. Who is
> going to install patches on regression tester sites?
Since the changes are checked into the HEAD as they always have
been, boost regression testing would not change in any way. It
work exactly as before. That fact that I'm testing locally against
the last released version wouldn't be detectable by boost testing.
(except perhaps by noticing that the time test boost serialization
will be much reduced - which needs to happen in any case)
Remember, I just want to test on my own system against the last
released version and make those changes available to users of
the last release system. I'm not proposing anything other than that.
>>>> I am not sure how your reply relates to my comment. What I meant
>>>> is: How can YOU test that your library works with the next version
>>>> of let's say Boost.Test I am planning to include into next release.
>>
>> Ahhh - now that is the question. This can only fail if the next
>> version of Boost.Test has a different interface than the previous
>> version. If
>
> Not necessarily. Boost.Test may not be the best example. But let's
> say you want to employ new feature that implemented in next version
> of Boost.Test, but is not yet part of Boost release?
At this point, I would be reluctant to make my library dependent on features
of other libraries which are still in development. In the unlikely event
I were to do something like this, I would just put those changes in the
HEAD and leave them there. This not a problem that I face. My current
problem is:
a) Testing all the things in the serialization library which can possibly
fail (an XP maxim) takes too long. So the test suite for the serialization
library has to be cut down.
b) Which makes it even less useful for me.
c) And doesn't test combinations which I feel I need - example - release
mode with profiling.
d) Release time for boost as a whole is taking so long I've got to keep
3 versions in my head - boost 1.33 - released, boost 1.34, and boost/HEAD.
Perhaps I could dedicate increasingly scarce resource of my aging
brain to keeping all this straight - but why stress my self. With my
method, I can tell users - problem solved - just update to the "next"
serialization release.
>> if the interface changes then a test failure would be an error in
>> boost test which is beyond my scope, expertise and authority to
>> address. So there is no point in my testing for it. If the
>
> Another possibility is that next version of library finally remove
> some deprecated interfaces and you aren't aware of this until you
> start testing against new version.
Again - this is the problem of interface change. Deprecation is taken
far too lightly by boost developers. I realize it makes the developer's
job a lot easier - but it creates a big problem for library users. It
basically requires that users constantly go back and fiddle with thier
applications which has a large economic cost. Interface enhancement
is OK - Interface enhancement is a big problem. Imagin the chaos
if someone decides - oh std::list could have been better designed -
we're going to fix it. The correct fix is to make a new interface
(say std::xlist or whatever). If you want your software to last
a generation, you can't be deprecating stuff every year.
>> author of Boost Test wants to use the serialization library
>> to test his new version - of course he's free to do so.
>>
>> If a library that I depend upon changes its interface and/or support
>> too frequently, I'll have to decide on case by case basis
>> whether using the library is more trouble than it's worth.
>
> More complicated your library is, more dependencies it has. And more
> chances that once in a while at least one of them is going to cause
> you some trouble with new/updated/deprecated interfaces. Backward
> comparitibilty has it's limits. At some point developer may be faced
> with a choice of breaking it or missing some good opportunities.
Of course this may occur from time to time. But this has to be the
developer's choice - not boosts. Right now we have wide variation
of views - spirit 1.6x is still available (thank goodness) so xml
serialization
is still backwards compatible to older/broken compilers. Boost Test doesn't
do this so it forces library developer's to make a choice. Different
developer's have made different choices - which is how its
always going to be.
>> In any case, It seems current regression testing isn't going
>> to change much in the near future so I would expect these
>> kinds of errors will continue to be caught by the current system.
>
> What kind of errors? Change in a dependency is not necessarily an
> error. Though I may be. And then you are stuck until it's fixed.
In the case where I depend on a feature which has been deprecated
or interface which has been changed - I'm still stuck as I am
now. This is a problem beyond my power to fix - I either have
to adjust to it (by waiting) or work around it (by reducing
dependence). Just as I have to now.
My "solution" doesn't address every probem, just the one's
mentioned above. I'm not striving for a "definitive fix"
(which I think is a hopeless quest) but rather continuous
incremental improvement - which is what I expect to achieve.
>> Hmm - my "formalization" is to discourage libraries from
>> changing their interface and/or support very often.
>
> You can't enforce freezing of development of all the libraries you
> depend on.
I agree, I have to decide which dependencies are worth maintaining.
> Eventually it's always choice of either totally controlled
> "in-house implementation" or dealing with occasional change. Also
> "often" is very subjective term. For some people once in a few years
> is too often. Others are OK with monthly releases.
agreed.
> Advantage of
> independent versioning system is that it should allow you to develop
> library in your own pace and don't have to keep up with every release
> of component you depend on (with some limitations).
I don't thing a more elaborate versioning system is going to achieve this.
Robert Ramey