Re: [boost] Testing and toolsets

Subject: Re: [boost] Testing and toolsets
From: Rene Rivera (grafikrobot_at_[hidden])
Date: 2015-01-19 12:29:31

Consider this my view as testing manager..

On Mon, Jan 19, 2015 at 8:14 AM, Andrzej Krzemienski <akrzemi1_at_[hidden]>
wrote:

> Hi Everyone,
> Could anyone give me some information on how regression test matrix is
> supposed to work? I can see that when testers run their tests they specify
> a toolset; this toolset is reported in test matrix, and based on this
> toolset name failures in tests can be labeled as expected. Did I get it
> right?
>

Correct.

However, the fact that a tester turns C++11 support or not, is not
> necessarily reflected in the toolset name (some testers indicate it in the
> toolset name, others don't). On the other hand C++11 being turned on or off
> can affect the test results. In that case the toolset name alone is not
> sufficient to make a decision whether I expect a failure or not.
>

Right.

Is there some policy for addressing this problem? Am I the only person to
> be affected?
>

My view is that you should not use the toolset name as a basis for markup.
In an ideal world the status markup would entirely go away. As it's an
inferior mechanism for dealing with expected test variance. Instead please
use the config method as previously (and below) mentioned.

-- 
-- Rene Rivera
-- Grafik - Don't Assume Anything
-- Robot Dreams - http://robot-dreams.net
-- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail