Re: [boost] regression test feature requests

From: David Abrahams (abrahams_at_[hidden])
Date: 2001-01-22 20:45:26

Thanks, Jens!

A couple more problems:
1. Why am I forced to specify an output file if I'm just running one test?
In that case, the results at stdout are perfectly informative; I don't
really need an .html result. Could we just skip it?
2. Trying to specify --config and --compiler seems to choke the test system
on my machine:

------
cd c:/boost/status/
regression --config compiler.cfg --compiler vc --output foo.html run
libs/graph/test/graph.cpp
***  ***

** Microsoft Visual C++
cl /Feboosttmp.exe /nologo /Zm400 /MDd /W3 /GR /GX /Zi /Od /GZ /I".." /D
"WIN32" /D "_DEBUG" /D "_MBCS" /D "_CONSOLE" ../ /link /incremental:no
user32.lib
Command line warning D4024 : unrecognized source file type '../', object
file assumed
LINK : fatal error LNK1181: cannot open input file "../.obj"
Fail
--- but ----
cd c:/boost/status/
regression --compiler vc --output foo.html run libs/graph/test/graph.cpp
*** libs/graph/test/graph.cpp ***

** Microsoft Visual C++
cl /Feboosttmp.exe /nologo /Zm400 /MDd /W3 /GR /GX /Zi /Od /GZ /I".." /D
"WIN32" /D "_DEBUG" /D "_MBCS" /D "_CONSOLE" ../libs/graph/test/graph.cpp
/link /incremental:no user32.lib
graph.cpp
boosttmp.exe
Pass
----and ----
cd c:/boost/status/
regression --config compiler.cfg --output foo.html run
libs/graph/test/graph.cpp
*** libs/graph/test/graph.cpp ***

** Borland C++ 5.5.1
bcc32 -eboosttmp.exe -I.. -j10 -q ../libs/graph/test/graph.cpp
'bcc32' is not recognized as an internal or external command,
operable program or batch file.
Fail

** Metrowerks CodeWarrior
mwcc -o boosttmp.exe -maxerrors 10 -cwd source -I- -I..
../libs/graph/test/graph.cpp
boosttmp.exe
Pass

** Microsoft Visual C++
cl /Feboosttmp.exe /nologo /Zm400 /MDd /W3 /GR /GX /Zi /Od /GZ /I".." /D
"WIN32" /D "_DEBUG" /D "_MBCS" /D "_CONSOLE" ../libs/graph/test/graph.cpp
/link /incremental:no user32.lib
graph.cpp
boosttmp.exe
Pass

** Microsoft Visual C++ with STLport
cl /Feboosttmp.exe /nologo /Zm400 /MDd /W3 /GR /GX /Zi /Od /GZ /I
"c:/prj/stlport-4.0/stlport" /I".." /D "__STL_DEBUG" /D "WIN32" /D "_DEBUG"
/D "_MBCS" /D "_CONSOLE" ../libs/graph/test/graph.cpp /link /incremental:no
/libpath:c:/prj/stlport-4.0\lib user32.lib
graph.cpp
boosttmp.exe
Fail

** GCC under cygwin
g++ -o boosttmp.exe -ftemplate-depth-30 -I.. ../libs/graph/test/graph.cpp
boosttmp.exe
Pass

** Intel C++ 5.0
icl /Feboosttmp.exe /nologo /Qwn5 /Zm400 /MDd /W3 /GR /GX /Zi /Od /GZ /I".."
/D "WIN32" /D "_DEBUG" /D "_MBCS" /D "_CONSOLE" ../libs/graph/test/graph.cpp
/link user32.lib
'icl' is not recognized as an internal or external command,
operable program or batch file.
Fail

----- Original Message -----
From: "Jens Maurer" <Jens.Maurer_at_[hidden]>
To: <boost_at_[hidden]>
Sent: Monday, January 22, 2001 6:03 PM
Subject: Re: [boost] regression test feature requests

> David Abrahams wrote:
> > 1. There's really too much output to get a quick, clear view of what
might
> > have been broken. What I really want is to have the testing system
compare
> > the results I'm getting with the previously-checked-in state and tell me
how
> > many tests I've broken (and what I've fixed!)
>
> Done.  There's a new option "--diff" to the regression test program
> which reads the current cs-XXX.html file before overwriting it.
> Output style for differences is similar to Beman's script; please adjust
> to taste. The difference mechanism can cope with new or removed tests,
> but it cannot cope with new or re-ordered compilers.  In the latter cases,
> it is advisable to not use "--diff" to get a fresh cs-XXX.html file.
>
> We should check in only non-highlighted versions; i.e. the highlight
> feature is just some internal debugging aid for the individual developer.
>
> > 2. Programs which fail at runtime typically give me no feedback about
why
> > they've failed,
>
> Do you mean "core dump vs. normal failure detected by application"?
> system() is not an adequate interface to give you that amount of
information.
> For Unix, I know how to get you the information, but I've no idea about
> Windows.
>
> To retrieve more information about application-level test failures
> needs a test protocol and a test framework.  Otherwise, just have a look
> at the compiler (application output) message log.
>
> Is it ok to revive some sort of annotation such as a footnote indicating
> compile, link, or run failure for the failed tests?
>
> >  and I don't know how to get a test to run under the
> > debugger.
>
> Have your test recompile with
>   regression -o /dev/null --compiler XXX compile-link the_broken_test
> and fire up your debugger on the resulting boosttmp.exe.
>
> Jens Maurer
>
>
>
>