Re: [boost] Testing and toolsets

Subject: Re: [boost] Testing and toolsets
From: Andrzej Krzemienski (akrzemi1_at_[hidden])
Date: 2015-01-19 14:35:25

2015-01-19 18:29 GMT+01:00 Rene Rivera <grafikrobot_at_[hidden]>:

> Consider this my view as testing manager..
>
> On Mon, Jan 19, 2015 at 8:14 AM, Andrzej Krzemienski <akrzemi1_at_[hidden]>
> wrote:
>
> > Hi Everyone,
> > Could anyone give me some information on how regression test matrix is
> > supposed to work? I can see that when testers run their tests they
> specify
> > a toolset; this toolset is reported in test matrix, and based on this
> > toolset name failures in tests can be labeled as expected. Did I get it
> > right?
> >
>
> Correct.
>
> However, the fact that a tester turns C++11 support or not, is not
> > necessarily reflected in the toolset name (some testers indicate it in
> the
> > toolset name, others don't). On the other hand C++11 being turned on or
> off
> > can affect the test results. In that case the toolset name alone is not
> > sufficient to make a decision whether I expect a failure or not.
> >
>
> Right.
>
> Is there some policy for addressing this problem? Am I the only person to
> > be affected?
> >
>
> My view is that you should not use the toolset name as a basis for markup.
> In an ideal world the status markup would entirely go away. As it's an
> inferior mechanism for dealing with expected test variance. Instead please
> use the config method as previously (and below) mentioned.

I am sorry, I may be thinking slowly, but what config method are you
referring to?

Regards,
&rzej