Re: [boost] boost.test regression or behavior change (was Re: Boost.lockfree)

Subject: Re: [boost] boost.test regression or behavior change (was Re: Boost.lockfree)
From: Edward Diener (eldiener_at_[hidden])
Date: 2015-10-09 02:22:27

On 10/9/2015 1:38 AM, Bjørn Roald wrote:
>
>> On 08 Oct 2015, at 22:06, Edward Diener <eldiener_at_[hidden]> wrote:
>>
>> On 10/8/2015 1:46 PM, Bjørn Roald wrote:
>>>> On 04 Oct 2015, at 14:49, Raffi Enficiaud <raffi.enficiaud_at_[hidden]> wrote:
>>>>
>>>> Le 04/10/15 13:38, John Maddock a écrit :
>>>>>
>>>>>
>>>>> On 04/10/2015 12:09, Bjorn Reese wrote:
>>>>>
>>>>> As many others have said, Boost.Test is "special" in that the majority
>>>>> of Boost's tests depend on it.  Even breakages in develop are extremely
>>>>> painful in that they effectively halt progress for any Boost library
>>>>> which uses Test for testing.
>>>
>>> This sort of problem has been discussed before on this list without any real progress.  I think a solution to this is needed to allow boost tools maintainers  (boost.test is also a tool), similar services that library maintainers enjoy.  A solution may also provide better test services for all boost developers and possibly other projects.  An idea of a possible way forward providing a test_request service at boost.org/test_request is outlined below.
>>>
>>> I would like thoughts on how useful or feasible such a service would be, these are some questions I would like to have answered;
>>>
>>> - Will library maintainers use a boost.org/test_request service?
>>> - How valuable would it be, as compared to merging to develop and waiting
>>>   for current test reports?
>>> - How much of a challenge would it be to get test runners (new and old) onboard?
>>> - How feasible is it to set up a service as outlined below based on modification
>>>   of the current system for regression testing in boost?
>>> - What alternatives exist providing same kind of, or better value to the community,
>>>   hopefully with less effort?  E.g.: can Jenkins or other such test dashboards /
>>>   frameworks easily be configured to provide the flexibility and features needed here?
>>>
>
> removed most of message, see original post.
>
>>
>> I think that what you have written is extremely valuable but I think you may underrate the need for individual developers to be able to test their library, or any other Boost library, on their local machine using a testing environment that is part of Boost. Currently that testing tool is usually either Boost.Test or lightweight test.

snipped...

> That is the idea, basically putting the control of "what to test” remotely in the hands of the individual developer, not only the bot managing the boost.org develop and master branch.  The challenge I believe is that the added flexibility will cause less structure and easily exhaust the test runner resources unless it is under some sort of control.  It may be too simple post test requests invoking compilation and testing parts of boost you do not need to test and on more runners than you need.

Could you please try to break up your messages/response into manageable 
lines for the viewing by others ?

The bot managing the boost.org 'master' and 'develop' tests does not 
currently determine what tests are there. What does determine which 
regression tests appear are individual testers who have the resources 
and the time on their computers to run a regression test.

If there were some sort of test runners on the Internet it should be 
fairly easy to setup regression tests for the major compilers on various 
platforms, as long as the testing apparatus supported a number of 
platforms and a wide variety of compilers and their versions. The 
biggest problem as I see it is co-ordination between changes to Boost 
'develop' and 'master' and the testing apparatus accessing the Boost git 
repository and periodically running regression tests. Because Boost as a 
whole is a very large number of individual libraries it would be 
wasteful to run a regression tests on all libraries every time a change 
was made to any individual library on whatever branches, presumably 
'master' and 'develop' for now, for which we want to automate regression 
testing. So some other schema would have to be created to determine how 
often regression testing would be run, for a particular environment, on 
Boost as a whole. Also when an automatic test runner actually runs 
regression tests it would need to take a snapshot of the Boost libraries 
at a given time to prevent the regression test from running its tests 
from a changing source tree.