Re: [boost] Regression test / compiler status progress

From: Beman Dawes (bdawes_at_[hidden])
Date: 2002-07-11 21:40:37

At 03:51 PM 7/11/2002, William E. Kempf wrote:

 >> But step back a bit.  I quickly scanned back over this thread, and 
didn't
 >> see anyone explaining the benefits from generating XML as they would
 >apply
 >> to regression.  In other words, if the sole use is to look at the 
output
 >> with a web browser, why bother?
 >
 >I gave one example, to view the data in a web browser in a different
 >format.
 >To make this a little more concrete, let's think about a compiler vendor,
 >say Comeau C++.  For marketing purposes it would be great if they could 
run
 >their own regression tests (possibly using more platforms and backends 
than
 >what we are using) and publish reports showing how well they can handle
 >Boost code (or even some other library).  With the hard coded HTML output
 >they'll have to step through several hoops to get an accurate report that
 >shows only the data they find pertinent and in a format appropriate for
 >their marketing.
 >
 >Or, they could use the XML generated by Boost, but combine the output 
from
 >the various platforms into one report about their compiler, something our
 >reports don't do for compilers that work on multiple platforms.  (That's 
a
 >format that might be useful even for us.)
 >
 >Or, imagine a compiler vendor that's working on their C++ conformance and
 >using Boost as a test bed.  Instead of an HTML report they could use the
 >XML
 >to generate bug reports with what ever bug tracking software they happen 
to
 >use in an automated build process.
 >
 >Theoretically (not really suggesting this, just brain storming) the Boost
 >release process could even make use of this by e-mailing the owners of
 >specific libraries about which tests are failing on what
 >compilers/platforms.  An automated script could even compare the current
 >results against previous results (pulled from CVS by the tag used for the
 >previous release) to pinpoint things that are breaking that never used 
to.
 >Once the work of creating the scripts was completed this automation could
 >simplify your job as the administrator of Boost releases.
 >
 >I'm sure there are other things that people could think of.  I don't 
think
 >that Boost themselves will have a need beyond generating an HTML page for
 >the web (well, again, maybe some automation in releases), but Boost users
 >would benefit in many ways from a regression test system that created XML
 >files instead... especially if that system can be reused in their own
 >development efforts.

Those sound like reasonable possibilities, as do the suggestions from Rob 
Stewart and Peter Dimov.

We've been experimenting with generated diffs, and with automatic emails to 
developers when some test event happens, but have just been whacking 
together little scripts.  Not very portable or maintainable.

Meanwhile, for the regression test, I've decided to pull together 
information now in several different files and formats into a single new 
file, and need a format for that.  Since I'm starting from scratch as far 
as the format goes, it shouldn't be much more difficult to use an XML 
format. Usually I would describe such a format in EBNF, so I'll have to 
figure out how to describe it in the XML equivalent.

Also, Dave Abrahams has been pushing me to generate a specific set of 
requirements for regression testing under Boost.Build V2, and the XML specs 
would serve to communicate what is desired in an unambiguous fashion.

Thanks for comments on this; it really helps to hear others perspectives 
and ideas.

--Beman