Re: [Boost-cmake] CTest and logfile scraping

From: troy d. straszheim (troy_at_[hidden])
Date: 2008-06-01 16:49:12

Hey Beman,

Beman Dawes wrote:
> 
> I've been much happier with the current Boost regression reporting since it
> started reporting the SVN revision number. I'd hate to go back to a system
> that didn't report that.
> 

No question about it, this kind of meta-information is crucial.
The snowblower (that php thing I mentioned which has now been abandoned)
displayed the urls of each svn:external.  It was nice to be able to see
e.g. that version X of component/library A was all-green
against version Y of library B.  We might want to keep that additional
svn:externals stuff in mind should boost get more 'componentized' in
the future...

Another metadata for-instance:  a tester should be able to include arbitrary
'notes' about the configuration of the platform/build, both to clarify what the
build results mean and to get credit for their contribution.

Luckily, once you've avoided the log-scraping step, the data path becomes
fairly clean and getting this stuff up to the server becomes quite easy.

>>> 5.  See the actual commands executed to run certain builds.  One often
>>>     wants to do this when chasing build misconfigurations: what were
>>>     the flags this lib was built with?
>> Yes, this is really, really important information whenever something
>> fails, and CTest/Dart/CDash don't do a good job of handling this.
> 
> FWIW, I really depend on jbam's -d2 option to track down misconfigurations.
> 

If you can diagnose and fix a problem without logging in to a machine and running
a build, (or in boost's case, writing a email to the tester saying
'try this'), you're saving tons of time.  You'd like to have exactly that
-d2 information reported and available on the test-reporting website.

>> [snip log-scraping issues, solution]
>>> On integration into cmake:
>>>
>>> - CMake already does something similar:  Think
>>>   CMAKE_VERBOSE_MAKEFILES and the toggleable fancy colorization
>>>   and percent-completed display.
>> Right. Note that this stuff does not work in Visual Studio, which may
>> be an issue. I don't understand the Visual Studio model well enough to
>> have a good sense of whether something similar is possible. I guess in
>> the worst case, we have some limitations in Visual Studio or ask
>> regression testers to use NMAKE
> 
> 
> Even though Visual Studio has been my preferred development platform for
> many years, I'd prefer NMAKE for regression tests. I run them in the
> background while doing other work, and don't want Visual Studio popping up
> on the screen.

We'll have to have a look at this.

Best,

-t