[boost] Boost Library Testing - a modest proposal - was boost.test regression or behavior change (was Re: Boost.lockfree)

Subject: [boost] Boost Library Testing - a modest proposal - was boost.test regression or behavior change (was Re: Boost.lockfree)
From: Robert Ramey (ramey_at_[hidden])
Date: 2015-10-09 12:37:57

I believe this whole thread started from the changes in Boost.Test such 
that it can no longer support testing of C++03 compatible libraries. 
This is totally unrelated to the testing of Boost libraries.

Here is what I would like to see:

a) local testing by library developers.

Of course library developers need this in order to develop and maintain 
libraries.

Currently we have this and has worked quite well for many years. Making 
Boost.Test require C++11+ throws a monkey wrench into things for the 
libraries which use it. But that's only temporary. Libraries whose 
developers feel they need to maintain compatibility with C++98 can move 
to lightweight test with relatively little effort.

Developers who are concerned that the develop branch is a "soup" can 
easily isolate themselves from this by testing against the master branch 
of all the other libraries. The Boost modularization system with git has 
made this very simple and practicle (thank you Beman!).

So - not a problem.

b) Testing on other platforms.

We have a system which has worked pretty well for many years. Still it 
has some features that I'm not crazy about.

i) it doesn't scale well - as boost gets bigger the testing load gets 
bigger.

ii) it tests the develop branch of each library against the develop 
branch of all the other libraries - hence we have a testing "soup" where 
a test might show failure but this failure might not be related to the 
library under test but some other library. It diminishes the utility of 
the test results in tracking down problems.

iii) it relies on volunteer testers to select compilers/platforms to 
test under. So it's not exhaustive and the selection might not reflect 
that which people are actually using.

I would like to see us encourage our users to test the libaries that 
they use. This system would work in the following way.

a) A user downloads/builds boost.

b) he decides he's going to use library X, and Y

c) he runs a tool which tells him which libraries he has to test. This 
would be the result of a dependency analysis. We have tools which do 
similar dependency analysis but they would have to be slightly enhanced 
to distinguish between testing, deployment, etc. I don't think this 
would be a huge undertaking given the work that has already been done.

d) he runs the local testing setup on those libraries and their dependents.

e) he uploads the test results to a dashboard similar if not identical 
to the current one.

f) we would discourage uses from just using the boost libraries without 
runnig they're own tests. We would do this by exhortation and by 
refusing to support users who have been unwilling to run and post local 
tests.

This would give us the following:

a) a scalable testing setup which could handle a Boost containing any 
number of libraries.

b) All combinations of libraries/platforms/compilers actually being used 
would be those being tested and vice versa. We would have complete and 
efficient test coverage.

c) We would have statistics on libraries being used. Something we are 
sorely lacking now.

d) We would be encouraging better software development practices. 
Sometime ago someone posted that he had a problem but couldn't run the 
tests because "management" wouldn't allocate the time - and this was a 
critical human life safety app. He escaped before I could weedle out of 
him which company he worked.

And best of all - We're almost there !!!! we'd only need to:

a) enhance slightly the dependency tools we've crafted but aren't 
actually using.

b) develop a tool to post the local results to a common dashboard

c) enhance the current dashboard to accept these results.

Robert Ramey