Re: [Boost-users] --run_test and using this to run a subset of Testcases.

Subject: Re: [Boost-users] --run_test and using this to run a subset of Testcases.
From: Richard (legalize+jeeves_at_[hidden])
Date: 2013-12-29 01:27:12

[Please do not mail me a copy of your followup]

boost-users_at_[hidden] spake the secret code
<OF52433783.7EEA0E88-ONC1257C1B.001CA2A2-C1257C1B.001CB1FB_at_[hidden]> thusly:

>First off, is there a possibility to do a negative selection. 
>i.e. tell boost.test to NOT run certain Testcases/testsuites? 

There is only positive specification, although it does have a wildcard
facility.

>Further is it possible to run sets of Testcases(which may reside in 
>different testsuites)? 

Yes, you can specify multiple sets of test cases by using commas and
identify suite levels using slash like a pathname.

However, if you have your tests arranged into suites and you use a
comma, things don't bind the way you'd expect because the matcher
splits on slash first and then on comma.  So commas don't separate
completely seperate test specifications, they separate names at the
same level of the test suite hierarchy.

>Assume following test tree: 
>MASTER 
>suite1 
>        case1 
>        case2 
>
>suite2 
>        testcase1 
>        testcase2 
>suite3 
>        subsuite 
>                testcase1 
>
>Now lets say I wanna run testcase case1, testcase2 and the subsuite 
>testsuite. 

I made a dummy test application with suites and test cases arranged as
above and got this:

> ConsoleApplication1.exe --report_level=detailed --run_test=*/case1,testcase2,subsuite
Running 3 test cases...

Test suite "Master Test Suite" passed with:
  3 assertions out of 3 passed
  3 test cases out of 3 passed

  Test suite "suite1" passed with:
    1 assertion out of 1 passed
    1 test case out of 1 passed

    Test case "case1" passed with:
      1 assertion out of 1 passed

  Test suite "suite2" passed with:
    1 assertion out of 1 passed
    1 test case out of 1 passed

    Test case "testcase2" passed with:
      1 assertion out of 1 passed

  Test suite "suite3" passed with:
    1 assertion out of 1 passed
    1 test case out of 1 passed

    Test suite "subsuite" passed with:
      1 assertion out of 1 passed
      1 test case out of 1 passed

      Test case "testcase1" passed with:
        1 assertion out of 1 passed

However, things aren't as wonderful as this implies.  Suppose I add
another test case called "case1" to "subsuite" and rerun the same
command:

> ConsoleApplication1.exe --report_level=detailed --run_test=*/case1,testcase2,subsuite
Running 4 test cases...

Test suite "Master Test Suite" passed with:
  4 assertions out of 4 passed
  4 test cases out of 4 passed

  Test suite "suite1" passed with:
    1 assertion out of 1 passed
    1 test case out of 1 passed

    Test case "case1" passed with:
      1 assertion out of 1 passed

  Test suite "suite2" passed with:
    1 assertion out of 1 passed
    1 test case out of 1 passed

    Test case "testcase2" passed with:
      1 assertion out of 1 passed

  Test suite "suite3" passed with:
    2 assertions out of 2 passed
    2 test cases out of 2 passed

    Test suite "subsuite" passed with:
      2 assertions out of 2 passed
      2 test cases out of 2 passed

      Test case "testcase1" passed with:
        1 assertion out of 1 passed

      Test case "case1" passed with:
        1 assertion out of 1 passed

So you see the problem has to do with the backwards way in which slash and
comma are used to parse the argument value.  You'd expect that comma
would specify a whole new test specification and not a variant at the
level expressed between slashes.  Since you have your tests in suites,
you must specify the slash or your filter won't match any tests.

There is another option which may be appealing if you have to run this
subset long-term and that is to simply link the interesting subset
into it's own executable and run everything in that executable.
-- 
"The Direct3D Graphics Pipeline" free book <http://tinyurl.com/d3d-pipeline>
     The Computer Graphics Museum <http://computergraphicsmuseum.org>
         The Terminals Wiki <http://terminals.classiccmp.org>
  Legalize Adulthood! (my blog) <http://legalizeadulthood.wordpress.com>