[boost] [test] handling fatal errors

From: Gennadiy Rozental (gennadiy.rozental_at_[hidden])
Date: 2002-12-01 12:28:39

Hi, everybody

   I would like to bring your attention to some issues related to handling
by Boost.Test errors causing program to terminate in regular case. It could
be fatal error like SIGSEGV is caught or division by zero or assert
statement. Unfortunately Boost.Test a bit inconsistent in it's handling
logic. And in any case not very helpful in some cases. Seeing "memory access
violation" does not give enough information. And current logic would not
stop/produce core at the point of the error. On the other hand it may not be
convenient for regression testing.

   So let list three models of how developer used to try to locate/prevent
the errors, with respective developers preferences:

[a] Working in GUI/command line debugger
     In this case the best would be to stop exactly in the point of the
fatal failure or produce a core.
[b] Adding trace statements, running program and analyze output.
     In this case it good enough to see output together with terminating
message from Boost.Test. On the other hand to locate the exact point of
failure may require several iteration of adding trace statements and
recompilation. It would be helpful to see as much program flow info as
possible. Boost.Test provide the means for this in a form of
BOOST_CHECKPOINT tool. But it still need to be placed in manually.
[c] Running regression tests on regular basis by some automation scripts.
    In this case developer would be interested in much more coverage as
possible. IOW one would prefer to see total statistics on all test cases
(which one passes/which one failing) even if one of them causing assert to
fire.

Now let's list conditions causing program to crash and how Boost.Test
handling them now:

1. System fatal errors (SIGSEGV,SIGBUS or some of the MS exceptions)
   Caught by execution monitor and will cause testing to abort.
2. SIGABRT initiated by abort
   Doesn't caught by Boost.Test at the moment
3. System errors (SIGFPE, SIGTRAP or some of the MS exceptions)
   Caught by execution monitor and will not cause testing to abort.
4. assert statements
   In all implementations that I found it's implemented by printing error
message into stderr and calling abort() effectively bringing us to the case
2.
5. _ASSERT statements
   MS provide a way to set a debug hook that will allow to handle these
assert statements. I do not know about others supporting this macro.

Well now what I think could be done:

I.  I could introduce parameter in execution monitor that will manage
whether or not it should try to catch "fatal" exceptions. We will need
environment variable to set it value at runtime Program/Test Execution
Monitor or Minimal testing. I will also support CLA in Unit Test Framework.
II.   I could introduce report hook for _ASSERT handling on MS using
_CrtSetReportHook.
III.  I could try to catch SIGABRT signals on UNIX systems and introduce
hook for abort on MS using _CrtSetReportHook. The question is whether it's
fatal error or not. I inclining to think that we could try to continue.
IV. At first I though that it would be convenient for the user in a model
[b] to set the last passed assertion text as a checkpoint. So that user will
see failed assertions and if last assertion before fatal failure passed it
will see it's text. But then I realized that user could always set report
level to all and will see all the messages anyway. I will specifically
remark this in docs.

Comments?

Gennadiy.