[boost] Testing Problems

Subject: [boost] Testing Problems
From: Christopher Jefferson (chris_at_[hidden])
Date: 2011-05-10 04:27:35

Apologises, the following message is going to be a fairly long list of complaints. However, I wanted to post them rather than just let them disappear.

Some background: I previously worked on getting clang and libc++ working in boost, in both c++03 and c++0x mode, submitting a large number of patches to achieve that goal. I also ran regression testers for this libraries. I am broadly planning on giving that up, unless various things improve in various ways. That isn't a threat, just I don't have the man and CPU time to spare working around issues in boost.

Problems:

1) Running the boost regression tests takes unacceptably long. I would like to run tests for clang with and without libc++, with and without c++0x mode. Running those 4 sets of tests takes a huge amount of time. Various tests (and I keep meaning to try to find which ones) take huge amounts of memory, meaning it isn't reasonable to run even a single-threaded boost regression test in the background on a 4GB machine. Running tests in parallel doesn't seem to speed them up particularly, while bringing the machine even more onto it's knees.

2) The boost regression testpage ( http://www.boost.org/development/tests/trunk/developer/summary.html ) is fairly useless, as it doesn't say when a particular test failed. This means I just have to try to memorise the current state of all the compilers I am interested in, and look for new fails by eyeball. Much more useful would be changing the 'fail' to the most recent svn version at which a new fail occurred?

3) I always find trac hard to use. I usually want to do 2 things:
a) Find all open bugs attached to package X
b) List all bugs which I filed, or have have submitted a message to.

I find both of these easy to achieve with bugzilla, and not with trac.

4) Getting patches, in particular very simple ones (like warning suppression) can take multiple prods and can languish for a long time in trac.

I'm not sure what the best solutions to all of these problems are. Obviously they could each be tackled but several of them would involve changes to many different boost libraries, which is something I find can be like pulling teeth (some maintainers are extremely good, I certainly don't want to tar everyone with one brush!)

On practical note, is it time for boost to move to officially supported compilers for each release? These could easily be read off the testing page (no test = not offically supported), and the packages which do not pass testing could be briefly listed in the release notes. This might have the advantage of encouraging more people to run tests, if they want their compiler to be offically supported.

Apologises for the lengthy rant,

Chris Jefferson