Re: [Boost-cmake] Big picture?

From: Doug Gregor (doug.gregor_at_[hidden])
Date: 2008-05-21 10:59:49

On Wed, May 21, 2008 at 10:28 AM, Beman Dawes <bdawes_at_[hidden]> wrote:
> On Wed, May 21, 2008 at 8:49 AM, Doug Gregor <doug.gregor_at_[hidden]> wrote:
> Dave has also been talking about the need for overall measures of what he
> calls the health of Boost. To me, that should include QA testing, such as
> the Boost inspect program, as part of the regular testing and reporting
> system. So I'd like to see a library marked as failing if it is getting
> inspect failures. That would be a big incentive to expand the inspect
> program to cover more areas, such as documentation probes.

We could  do this by automatically creating a test per library that
runs inspect on the library's sources. VTK actually does this, and
it's quite helpful. Once the libraries are modular, it'll be easy to
just point inspect at the base of that library's sources.

I've added this to the "CMake TODO" list on the Trac.

>>
>> > Who will be responsible for the scripts that testers rely on? Rene?
>>
>> I'm guessing Troy or Dave. Troy wrote the original tester scripts for
>> the CMake-based build system.
>
> My concern about scripts are ease-of-use and robustness. Before Rene started
> working on the scripts last fall, it took several (and sometimes a lot more)
> emails back and forth to get a new tester up and running. Even then testing
> often stalled for long periods, and more emails went back and forth, because
> something broke. If there were changes to testing setup or configuration,
> more emails and more delays resulted.

Understood.

>> > Is the plan to start small with a few libraries and testers, and then
>> > gradually roll-out to all libraries and all testers?
>>
>> All libraries need to build and test so that we can verify that the
>> build system is working properly.  Plus, we would like to be able to
>> build binary installers with CMake sooner, and not wait for all of the
>> regression-testing work to be completed.
>
> Yes, that would be helpful. Building daily release snapshots has been very
> helpful, and the logical conclusion would be to both build and test at least
> one binary installer as part of each snapshot.

Automatically building the binary installers is easy; testing them is
a little tougher, because some of them are graphical and might require
some "clicking". Perhaps they can still be driven from the command
line? I'm not sure.

Either way, I have a preliminary implementation of testing Boost
against an installed tree. Essentially, one does a build+install
(without tests), then a separate build of the regression tests using
the installed headers and installed libraries. This is the eventual
goal for our testing environment, since it mimics the way a user would
use Boost. Adding the extra step of building a binary installer and
installing that for testing... that would be even better.

> Does cpack also build 7zip archives? They are becoming more popular, and
> Boost has been providing them for several releases.

CPack does not build 7zip archives. We've only been using 7zip
archives for Boost source-code releases, right? We can create those
without even involving CPack... binary releases using 7zip would
require a bit more effort, and possibly some CMake hacking.

>> > Are any metrics being collected? I'm interested in time and bandwidth
>> > utilization for both test clients and the central server, since these
>> > are
>> > limiting bottlenecks with the current mechanism.
>>
>> CTest collects some statistics for compile-time (in aggregate) and
>> run-time of individual tests. It's in the XML, and we could certainly
>> report that back to Bitten.
>
> What I'm interested in (as a release manager concerned with having enough
> testers for good coverage) are some overall measures of what kinds of
> demands we are placing on testers and servers. For example, how much time
> (wall clock? CPU?) is the tester using per day? How many bytes received?
> Transmitted? Similar for servers. No need to beat these to death; just three
> or four overall indications of resources consumed. Useful surrogates if
> those particular numbers aren't available.

Okay.

  - Doug