Re: [boost] Google SOC 2007, Boost.Test Frontend

From: Stefan Seefeld (seefeld_at_[hidden])
Date: 2007-03-21 20:23:37

Younes M wrote:
> First, thank you all for the comments.
> 
> On 3/21/07, Stefan Seefeld <seefeld_at_[hidden]> wrote:
>> There are a number of things I can think of that would make running boost tests
>> more useful and convenient. Among them:
>>
>> * An easy way to introspect the test suite, i.e. display all tests, with
>>   metadata.
> 
> I agree. I think a big benefit in a GUI frontend will be to display
> results in a more digestible manner than the current text output
> allows. For example:

Actually, I don't think the issue here is GUI vs. CLI. Instead, it's
about how robust and scalable the testing harness is. Think of it as
a multi-tier design, where the UI is just a simple 'frontend' layer.
Some layer underneath proides an API that lets you query about what
tests exist (matching some suitable criteria, such as name pattern matching,
or filtering per annotations), together with metadata.

That, together with other queries such as 'give me all platforms this
test is expected to fail on' would be very valuable for the release
process.
(All this querying doesn't involve actually running any tests.)

>> * An easy way to run subsets of tests.
>>
> 
> On the current Open Issues page selectively running test cases by name
> is mentioned, which I think fits into this. I've included it in my
> list of running ideas above.

Right, but it may also include sub-suites. To push a little further
(and deviate from the topic only a little bit), it would be good to
parametrize sub-testsuites differently. For example, the boost.python
tests may be run against different python versions, while that particular
parameter is entirely meaningless for, say, boost.filesystem.

>> * Enhanced test run annotations, to allow report generation to build a more
>>   meaningful test report (e.g. fold multiple equivalent test runs into one,
>>   only consider test runs from a given time window, or associated with a given
>>   revision, etc.)
> 
> One issue I forsee is in synchronizing the unit test with the GUI. As
> it stands, I had only considered using the output of Boost.Test to
> generate reports, but this means that the reports can get stale until
> you re-run. This also complicates the statistics across reports idea I
> had above, since it makes little sense to consider such a thing when
> the unit test changes considerably.

I'm still not thinking of the GUI as something with an internal state.
It's just a frontend to the rest of the harness, and thus, there isn't
anything to synchronize. The only thing that has a timestamp (or revision)
is the code, as well as a particular test run.

> I think your idea, if I've understood it correctly, of taking into
> account revisions of the test suite would work towards solving such
> issues. I'm not sure how to go about detecting/delineating revisions
> however, but I'll give it some thought.

Once boost is hosted on subversion, each checked-out source tree has a
single revision. Thus, you can label a test run with such a revision
and can immediately see whether it corresponds to a particular working
copy or not.

>> * Support for running tests in parallel, or even on distributed hardware, if
>>   available.
>>
>> * Cross-testing support (i.e. compiling with a cross-compiler toolchain on host A
>>   for target B, then uploading the test executables to B and run it there)
> 
> I must admit that I don't usually find myself cross-compiling or
> running on distributed hardware, so I might not appreciate some of the
> issues and requirements involved.

Distributed hardware here simply means that you may have got a compile farm
or other means to parallelize a test run, yielding significant speedups
(some boost testers report that a non-incremental run of the boost tests
takes *days* to complete !)

Cross-testing is useful for example when developing for embedded platforms,
where the machine hosting the development environment isn't the one running
the compiled applications (and thus, tests). We are using QMTest in-house
to cross-test GCC and various libs, such as libstdc++, for numerous platforms.

Regards,
                Stefan

-- 
      ...ich hab' noch einen Koffer in Berlin...