[boost] BoostCon07 testing sprint, results.

From: Rene Rivera (grafikrobot_at_[hidden])
Date: 2007-05-20 18:27:54

First...

Thanks to all those who made it to the testing session at BoostCon,
especially Victor for the beer :-) Even though I was dead tired by the
end of it, it was a lot of fun.

The rest...

I'll be posting the slides and notes from the session at some near
future (when it's clear where they get posted). Don't worry it's nothing
exciting ;-)

The immediate result from the session is that the current system is very
brittle. In particular Thomas identified getting reliable testing
results as the most imminent failure point. Unfortunately we didn't get
to discussing longer, and hence conclusions, as to the longer term
issues. The plan for the near future is to try out the Dart testing
results system <http://www.na-mic.org/Wiki/index.php/Dart2Summary>. We
are attacking that task from three sides:

* Doug & Troy, are going to do a test case of using CMake+CTest which
support posting results to Dart.

* Noel and myself, are going to make changes to bjam and Boost.Build to
support posting results to Dart directly.

* I volunteered to make changes to the regression Python scripts to
allow posting current results to Dart in addition to posting them to
Meta-Comm. I finished doing those changes last night, and the new
scripts are in CVS-HEAD. Now, by default, testers doing HEAD regression
runs will also post to the Dart server at <http://beta.boost.org:8081/>.
Take this also as a notice to testers...

> If you are testing CVS-HEAD, please update your regression.py and/or
> collect_and_upload_logs.py scripts.

* Everyone else is hereby drafted to look at the Dart results, when 
available, and comment on the system.

There was also discussion about using BuildBot for the front half of the 
testing problem, controlling testing resources. Sorry if I can't 
remember the persons name, but there was one person and myself willing 
to look at setting up and developing a test use case for BuildBot.

The future...

Since we didn't get to discuss much past the above topics, I'd like it 
if we continue the discussion in the testing list. Topics to still discuss:

* Other possible immediate solutions than the ones above. In particular 
Stefans mentioned QMTest before the conference, so it would be nice to 
get someone to work on a test use case for that.

* The longer term question of how to get more testing resources and how 
to manage them is still up in the air.

Finally, I'll set up a section in the new wiki to keep track of testing 
information as needed.

PS. I've avoided bringing up the subject of CMake vs. Boost.Build, 
mostly because it's a subject with personal ramifications, and partly 
because it's mostly a topic a separate topic. Posts about that topic are 
forthcoming.

-- 
-- Grafik - Don't Assume Anything
-- Redshift Software, Inc. - http://redshift-software.com
-- rrivera/acm.org - grafik/redshift-software.com
-- 102708583/icq - grafikrobot/aim - grafikrobot/yahoo