RE: [boost] MPL regression tests?

From: Beman Dawes (bdawes_at_[hidden])
Date: 2003-05-30 10:43:45

At 07:15 AM 5/30/2003, Aleksey Gurtovoy wrote:

 >IMO it's worth to step back and try to answer a couple of "big picture"
 >questions:

Yes, that's a good idea.
 >
 >1) What are the target audiences for the regression test results?
 >2) What kind of information these audiences are looking to find in
 >there?
 >
 >Answering these is actually simple - first, as usual, there are two
 >primary audiences here - boost users and boost developers. Now, IMO,
 >when going to the regression results page, these two groups are looking
 >for the answers on quite different questions.

Agreed.

 >I would argue that for a user _the_ dominant motive to study the
 >regression results is to find out whether a particular library works on
 >a particular compiler(s) she uses. What's tricky about it is that often
 >"works" doesn't necessarily equals to "all tests are passing". It's
 >quite common for a library to fail a few corner-case tests, tests of
 >seldom-used functionality, or advanced functionality that demand a high
 >standard conformance, and yet in practice be perfectly usable with many
 >of those compilers. As a matter of fact, if you analyze the current
 >compiler status table for, let's say MSVC 6.5, you will see that _most_
 >of the boost libraries fall into this "works with caveats" category.
 >Well, except that if you are indeed a user, you don't know that because
 >when looking at the particular tests' failures you have no idea whether
 >these are show stoppers or not, and if not, what do the failures
 >_mean_, in user-level terms.

It used to be that regression test failures fell in three categories:

1) Bug in the Boost code.

2) Bug in the compiler, will be fixed in their next release or service pack 
if we can get the vendor's attention (no always easy.)

3) Bug in the compiler, caused by some major internal compliance problem 
not likely to be fixed in the next release.

Category (3) is going away. For a bunch of the compilers Boosters seem to 
care about, (3) has already gone away. That gives us more time to 
concentrate on (1) and (2).

I'm not exactly sure how that affects the end user, except that more 
libraries should be unconditionally passing all tests on any given 
compiler. Maybe we shouldn't worry too much about shades of passing 
conditionally.

 >Now, that was a user position. If you wearing the developer's hat,
 >though, you are not really interested whether a particular library
 >works on a certain compiler; or, rather, you already know it because
 >you are the library author. Instead, here, the primary reason to check
 >regressions results is to make sure that none of the recent changes in
 >the CVS lead to, well, regression in the library's functionality (or
 >better yet, to be automatically notified if this happens).

Automatic notification is do-able. The notify/don't-notify option for each 
library can be checked into CVS, so each library can make their own 
decision as to whether or not to get notifications.

 > The failures
 >that are known are not nearly as interesting to you as a change in the
 >failures pattern. And just like the user, you don't want to gather this
 >information from pieces. Basically, you want the same field of
 >green/red cells as a user, one row per library, only in this case green
 >would mean "nothing's broken" (comparing to expected failures picture),
 >and red would mean "somebody broke something". Ideally, red on the
 >developers' report would be a quite rare event.
 >
 >Well, anyway, that was my take. I haven't thought through how exactly
 >something like this can be implemented so it's both easy to setup and
 >maintain (per library, especially the user-oriented report), and yet
 >indeed satisfactorily informative.

I've been thinking about those issues for awhile, and plan to add 
historical information to the .xml file maintained for each test. That will 
make change reporting easier, more accurate, and portable to all testing 
platforms.

 > We plan to use MPL as a test-bed for
 >these ideas to see how things work out.

Good!

--Beman