Re: unit test framework

From: Gennadiy E. Rozental (rogeeff_at_[hidden])
Date: 2001-10-23 03:00:16

--- In boost_at_y..., "Mike Attili" <yahoomail_at_a...> wrote:
> > From: Beman Dawes [mailto:bdawes_at_a...]
> > Subject: RE: [boost] unit test framework
> >
> > We really need to decide what dependencies we are willing to 
tolerate in
> > the unit test library.
> 
> The unit test tools in CVS only make use of <cstdlib> and <config>.
> Gennadiy's changes add <utility>, <smart_ptr>, and <progress>.
> I've added <regex> locally and have thought about using <thread>, 
among
> others. That's clearly not consistent with the needs of the Boost
> Internal Regression Test Suite.

I am hoping you will allow me to use noncopyable and scope_ptr (and 
let's hope thay will live separate home soon). Progress does not 
seems to use any "advanced" features of C++ s oshould work everywhere.
As one of the way to manage complex advanced additions to framework I 
see a separation them from framework core into standalone library. 
Tests included into regression suite should try to use only core 
functionallity. But online user can try to use advanced tools. In 
your case regex is used to prepare a list of test to run (though I 
think it could be done with less powerful weapon). This is obviosly 
does not required for automatic regression testing and can be linked 
in with your local project. Multithreaded unit Test require special 
attention and addiitional dscussions.

> 
> > If we want fuller functionality which comes at the cost of 
dependencies,
> > that may well mean that test tools can't use unit test.  
Otherwise,
> > dependency on regex would mean we can't test regex itself, and if 
regex
> > fails for a platform, we can't test on that platform at all.
> 
> If I understand correctly, 'unit test tools' is intended to be a 
layer on
> top of the 'test tools' and 'execution tools'. But, while trying to 
figure
> out what the trade-off's are, I realized that I don't have a clear 
picture
> of where the 'unit test tools' fit relative to the 'test tools'.
> Specifically, if I'm writing tests for a new C++ library (Boost or
> otherwise)
> when should I restrict myself only to the facilities provided 
by 'test
> tools'
> versus those provided by 'unit test tools'?
> 
> My apologies if I've missed this in the docs.

Unit test tools supposed to be used for UNIT testing, while notion of 
test tools is refered to the case when you need to test whole 
program. One example could be like this: you have an existent program 
that crash for some reason. You use cpp_main and be able to find and 
fix a place of crash. But still program seems to fail to do what is 
supposed to do. You use now test_main and start adding check 
statements all over the place. Finnaly you are able to analize the 
program behavior. There may be some other scenarios. Though I agree 
that using of test tools for new development now looks like using 
unit test tools with 1 test case only.

> 
> I should say that my specific interest in the unit test framework 
is for
> very short design-implement-test cycles (or what the 'agile' folks 
call test
> first design). 
> So, in addition to the requirements for regression testing,
> I'm also interested in:
>   Making tests as simple to write as possible;
>   Getting very fast feedback
>     (thus <regex> to temporarily reduce the scope of testing);
>   Integrating tests into development tools;
>   Detecting resource leaks;
>   GUI testing;
>   Thread testing

So, hoppefully you will like new error format that in combination 
with MSVC Post-Build step integration gives you an ability to see run-
time errors like compile-time one. You can even jump to the error 
location using F4. I am planning to add the same feature for other 
GUI (like XEmacs), though I will need a help, because I do not know 
how it get an errors positions from the compiler output. Also it 
would be interesting to know how it works with other Windows base 
compilers/GUI.

> 
> Maybe Boost isn't the right venue (since it's not exactly a library 
and I'd
> be surprised if any test tools were considered for standardization) 
but with
> such a great foundation I'd like to leverage as much as possible. 
And the
> easier it is test, the better our end products.
> Looking forward to helping any way that I can...
> 
>     Mike Attili
>     Amaxo, Inc.

Gennadiy.