|
|
Boost Testing : |
From: Gennadiy Rozental (gennadiy.rozental_at_[hidden])
Date: 2007-05-10 11:55:12
"Robert Ramey" <ramey_at_[hidden]> wrote in message
news:f1nhf4$k4o$1_at_sea.gmane.org...
> Gennadiy Rozental wrote:
>> "Robert Ramey" <ramey_at_[hidden]> wrote in message
>> news:f1mdto$pif$1_at_sea.gmane.org...
>>>>>>> This doesn't address neither testing against new version of your
>>>>>>> library by developers of other libraries
>>>
>>> Developers can test which every they want. Regression testers
>>> can test whatever they want. I presume they would
>>> want to continue testing the HEAD. This doesn't help me
>>> much but if some finds it useful to some purpose, it's fine
>>> by me.
>>
>> Developers quite possibly want to run tests against your new version
>> not locally, but using power of regression testing facility. Who is
>> going to install patches on regression tester sites?
>
> Since the changes are checked into the HEAD as they always have
> been, boost regression testing would not change in any way. It
This is not good from two standpoints:
1. You are *forcing* other developers to test against your latest release,
while they may prefer to continue to rely on baseline (last boost release)
version. If any problems arise will you rollback your changes? How is going
to fix the libraries that depends on you?
2. What If I want to test against some version in between your HEAD and
baseline version?
> work exactly as before. That fact that I'm testing locally against
> the last released version wouldn't be detectable by boost testing.
> (except perhaps by noticing that the time test boost serialization
> will be much reduced - which needs to happen in any case)
> Remember, I just want to test on my own system against the last
> released version and make those changes available to users of
> the last release system. I'm not proposing anything other than that.
>
>>>>> I am not sure how your reply relates to my comment. What I meant
>>>>> is: How can YOU test that your library works with the next version
>>>>> of let's say Boost.Test I am planning to include into next release.
>>>
>>> Ahhh - now that is the question. This can only fail if the next
>>> version of Boost.Test has a different interface than the previous
>>> version. If
>>
>> Not necessarily. Boost.Test may not be the best example. But let's
>> say you want to employ new feature that implemented in next version
>> of Boost.Test, but is not yet part of Boost release?
>
> At this point, I would be reluctant to make my library dependent on
> features
> of other libraries which are still in development. In the unlikely event
> I were to do something like this, I would just put those changes in the
> HEAD and leave them there. This not a problem that I face. My current
How could you "just put it there" if library development is on a branch for
example?
> problem is:
>
> a) Testing all the things in the serialization library which can possibly
> fail (an XP maxim) takes too long. So the test suite for the
> serialization
> library has to be cut down.
Why? I do not see any reason. The system should be build in a way that the
only person who cares about time independent library's unit test is run is
library developer. Once it's done you may proceed with your own pace.
Testing changes locally. committing them and checking results of regression
tests.
> b) Which makes it even less useful for me.
> c) And doesn't test combinations which I feel I need - example - release
> mode with profiling.
> d) Release time for boost as a whole is taking so long I've got to keep
> 3 versions in my head - boost 1.33 - released, boost 1.34, and boost/HEAD.
In some sense you will always be required to remember what your older
version did. At least for support. But again. With independent library
versioning you may do your own releases as frequent a you wish and it should
help you to keep track of your incremental development.
> Perhaps I could dedicate increasingly scarce resource of my aging
> brain to keeping all this straight - but why stress my self. With my
> method, I can tell users - problem solved - just update to the "next"
> serialization release.
I must say I understand you very well. My proposition is to adopt the same
procedure for all the libraries. And as a result speedup the whole boost
release procedures. Many users in big corporations can't rely on "beta"
releases. They require an "official" boost release before they can accept
any changes.
>>> if the interface changes then a test failure would be an error in
>>> boost test which is beyond my scope, expertise and authority to
>>> address. So there is no point in my testing for it. If the
>>
>> Another possibility is that next version of library finally remove
>> some deprecated interfaces and you aren't aware of this until you
>> start testing against new version.
>
> Again - this is the problem of interface change. Deprecation is taken
> far too lightly by boost developers. I realize it makes the developer's
> job a lot easier - but it creates a big problem for library users. It
> basically requires that users constantly go back and fiddle with thier
> applications which has a large economic cost. Interface enhancement
> is OK - Interface enhancement is a big problem. Imagin the chaos
> if someone decides - oh std::list could have been better designed -
> we're going to fix it. The correct fix is to make a new interface
> (say std::xlist or whatever). If you want your software to last
> a generation, you can't be deprecating stuff every year.
As a general sentiment I agree. And I also admit I prone to this "interface
deprecation" myself. It's always a balance of supporting two (old and new)
interfaces vs. backward compatibility. I believe 2-3 years if good enough
period for some interfaces to be first declared deprecated and eventually
removed. Special situations - where original interface is incorrect and/or
prone to erroneous usage - have to be treated on cases by case level.
>>> author of Boost Test wants to use the serialization library
>>> to test his new version - of course he's free to do so.
>>>
>>> If a library that I depend upon changes its interface and/or support
>>> too frequently, I'll have to decide on case by case basis
>>> whether using the library is more trouble than it's worth.
>>
>> More complicated your library is, more dependencies it has. And more
>> chances that once in a while at least one of them is going to cause
>> you some trouble with new/updated/deprecated interfaces. Backward
>> comparitibilty has it's limits. At some point developer may be faced
>> with a choice of breaking it or missing some good opportunities.
>
> Of course this may occur from time to time. But this has to be the
> developer's choice - not boosts. Right now we have wide variation
Right. Exactly my point. Each developer should decide for oneself which
version of dependant library to use.
> of views - spirit 1.6x is still available (thank goodness) so xml
> serialization
> is still backwards compatible to older/broken compilers. Boost Test
> doesn't
> do this so it forces library developer's to make a choice. Different
Boost.Test did not removed that many interfaces. There were couple instances
were original feature design was found unsound or incorrect or inconsistent.
> developer's have made different choices - which is how its
> always going to be.
>
>>> In any case, It seems current regression testing isn't going
>>> to change much in the near future so I would expect these
>>> kinds of errors will continue to be caught by the current system.
>>
>> What kind of errors? Change in a dependency is not necessarily an
>> error. Though I may be. And then you are stuck until it's fixed.
>
> In the case where I depend on a feature which has been deprecated
> or interface which has been changed - I'm still stuck as I am
> now. This is a problem beyond my power to fix - I either have
> to adjust to it (by waiting) or work around it (by reducing
> dependence). Just as I have to now.
No. With independent library versioning you may refer and test with older
version of the component that did not make the changes. Until you have
time/resources to move on to adopt the changes in a dependency.
> My "solution" doesn't address every probem, just the one's
> mentioned above. I'm not striving for a "definitive fix"
> (which I think is a hopeless quest) but rather continuous
> incremental improvement - which is what I expect to achieve.
>
>>> Hmm - my "formalization" is to discourage libraries from
>>> changing their interface and/or support very often.
>>
>> You can't enforce freezing of development of all the libraries you
>> depend on.
>
> I agree, I have to decide which dependencies are worth maintaining.
This is not necessarily have to be "either or" decision.
>> Eventually it's always choice of either totally controlled
>> "in-house implementation" or dealing with occasional change. Also
>> "often" is very subjective term. For some people once in a few years
>> is too often. Others are OK with monthly releases.
>
> agreed.
>
>> Advantage of
>> independent versioning system is that it should allow you to develop
>> library in your own pace and don't have to keep up with every release
>> of component you depend on (with some limitations).
>
> I don't thing a more elaborate versioning system is going to achieve this.
I hope I made it clear.
> Robert Ramey