Re: [boost] Google SOC 2007, Boost.Test Frontend

From: Gennadiy Rozental (gennadiy.rozental_at_[hidden])
Date: 2007-03-22 01:02:22

"Stefan Seefeld" <seefeld_at_[hidden]> wrote in message 
news:4601CC89.2030506_at_sympatico.ca...
> Younes M wrote:
>> First, thank you all for the comments.
>>
>> On 3/21/07, Stefan Seefeld <seefeld_at_[hidden]> wrote:
>>> There are a number of things I can think of that would make running 
>>> boost tests
>>> more useful and convenient. Among them:
>>>
>>> * An easy way to introspect the test suite, i.e. display all tests, with
>>>   metadata.
>>
>> I agree. I think a big benefit in a GUI frontend will be to display
>> results in a more digestible manner than the current text output
>> allows. For example:
>
> Actually, I don't think the issue here is GUI vs. CLI. Instead, it's
> about how robust and scalable the testing harness is. Think of it as
> a multi-tier design, where the UI is just a simple 'frontend' layer.
> Some layer underneath proides an API that lets you query about what
> tests exist (matching some suitable criteria, such as name pattern 
> matching,
> or filtering per annotations), together with metadata.

Boost.Test UTF itself provides this information already (using test tree 
travercing interfaces). We may consider adding some simpler, strait to the 
point interfaces, but even now you could get all you need.

> That, together with other queries such as 'give me all platforms this
> test is expected to fail on' would be very valuable for the release
> process.

Umm. I am not sure how do you plan to maintain this information.

> (All this querying doesn't involve actually running any tests.)
>
>>> * An easy way to run subsets of tests.
>>>
>>
>> On the current Open Issues page selectively running test cases by name
>> is mentioned, which I think fits into this. I've included it in my
>> list of running ideas above.
>
> Right, but it may also include sub-suites. To push a little further
> (and deviate from the topic only a little bit), it would be good to
> parametrize sub-testsuites differently. For example, the boost.python
> tests may be run against different python versions, while that particular
> parameter is entirely meaningless for, say, boost.filesystem.

What do you mean by "parametrize"?

>>> * Enhanced test run annotations, to allow report generation to build a 
>>> more
>>>   meaningful test report (e.g. fold multiple equivalent test runs into 
>>> one,
>>>   only consider test runs from a given time window, or associated with a 
>>> given
>>>   revision, etc.)
>>
>> One issue I forsee is in synchronizing the unit test with the GUI. As
>> it stands, I had only considered using the output of Boost.Test to
>> generate reports, but this means that the reports can get stale until
>> you re-run. This also complicates the statistics across reports idea I
>> had above, since it makes little sense to consider such a thing when
>> the unit test changes considerably.
>
> I'm still not thinking of the GUI as something with an internal state.
> It's just a frontend to the rest of the harness, and thus, there isn't
> anything to synchronize. The only thing that has a timestamp (or revision)
> is the code, as well as a particular test run.

Yes. That's correct way to look into this. test runner should employ 
exisiting Boost.Test interfaces to implement it's tasks.

>> I think your idea, if I've understood it correctly, of taking into
>> account revisions of the test suite would work towards solving such
>> issues. I'm not sure how to go about detecting/delineating revisions
>> however, but I'll give it some thought.
>
> Once boost is hosted on subversion, each checked-out source tree has a
> single revision. Thus, you can label a test run with such a revision
> and can immediately see whether it corresponds to a particular working
> copy or not.

As I argued in other post, IMO this is not part od this project. At least 
not in a first draft. It maybe added later on as an addon.

Gennadiy