|
|
Boost Testing : |
From: Carl Daniel (cpdaniel_at_[hidden])
Date: 2005-03-15 10:28:02
Aleksey Gurtovoy wrote:
> Carl Daniel writes:
>> I'd like to get my regression-test-running automated. Currently, a
>> number of tests fail with runtime dialogs or JIT debugger dialogs.
>> Using --monitored on regression.py helps somewhat, but it's not
> ^^^^^^^^
>> really a solution.
>
> Meaning you've encountered a situation when it didn't help? Just
> curious, because to our best knowledge, while not without serious
> usability problems (which I pointed out offline earlier), it does its
> job after all. We've been running unsupervised Win32 regressions with
> it 24/7 for almost 2 years.
When running tests interactively with --monitored, it successfully kills the
errant processes, but the JIT debugger dialog is produced by a different
process, so at the end of the run I have a screen full of "do you want to
debug" dialogs that I have to close.
>
>>
>> So, what other complaints are there about build monitor now? What
>> desired features are not supplied by the current implementation?
>
> To my best knowledge, there are two problems with the current
> implementation:
>
> 1) It's too greedy in killing processes; specifically, it doesn't
> conform to the "ideal" spec, which is:
Yep, I've seen that.
>
> "Regression tests' monitoring tool should terminate any process
> that:
>
> - Is spawned by 'bjam.exe' executable (or is the executable
> itself).
> - It has been running for more than N minutes _or_ showed an
> error dialog window.
> "
>
> 2) It doesn't handle the case of missing DLL dependencies; on Win32
> this results in a error dialog "DLL such and such is not found",
> which stalls futher running of tests.
Haven't run into that, but I can see that it wouldn't help there.
> The use cases that led us to the above formulation of the desired
> behavior were:
>
> - So-called "access violations"
> - In bjam
> - During compilation, linking, and running of tests
>
> - Error dialogs shown by C++ runtime support code
>
> - Processes hanging/running too long:
> - Tests
> - Compilers/linkers
>
>
> Issue #2, which we discovered later, obviously adds this one:
>
> - Other error dialogs shown by operating system (e.g. missing DLL
> dependencies).
A couple of things that will help (I haven't yet tried to find the
appropriate sources to see what's in there already). The main() for tests
(and maybe bjam itself) should be written something like:
int main(/* params */)
{
// prevent CRT error dialogs
_set_error_mode(_OUT_TO_STDERR);
// prevent JIT debug dialogs
__try
{
// previous contents of main()
}
__except(EXCEPTION_EXECUTE_HANDLER)
{
// Courtesy stop in the debugger, if present
if (::IsDebuggerPresent())
::DebugBreak();
// Paranoid - bail out of the process rather than attempting
// any orderly shutdown
::TerminateProcess(::GetCurrentProcess(),255); // or whatever exit
code
}
}
with that code in place in the test programs' main(), there should be no
more dialogs from the CRT/Debugger.
To handle all cases - like missing DLLs, as well as handle stuck programs,
the best solution may be to write a custom debugger/lancher that's used as a
trampoline for launching all child processes of bjam (or at least the tests,
compiler and linker).
so, rather than invoke cl.exe <arguments>, bjam would invoke monitor
<timeout> cl.exe <arguments>, or similar.
A debugger/launcher like that is not a difficult program to write. Such a
debugger could also hook any number of Windows API functions (CreateDialog,
etc) to prevent the subject process from creating any kind of GUI. A
debugger can intervene in the DLL loading process as well (I believe - I
haven't verified that).
One downside to the debugger solution is that the test runner would be
required to have SeDebugPrivilege. I would expect that nearly all (if not
all) developers (and their regression test suites) are running with
SeDebugPrivilege already, since you cannot debug without it.
I'm willing to take on writing that debugger/launcher if it sounds like it
could help.
-cd