Re: [boost] [build] Running 1 per N tests

Subject: Re: [boost] [build] Running 1 per N tests
From: Steven Watanabe (watanabesj_at_[hidden])
Date: 2016-06-29 12:54:19

AMDG

On 06/29/2016 04:34 AM, Adam Wulkiewicz wrote:
> Hi,
> 
> At Boost.Geometry we're using CircleCI for testing. They allow to run
> the tests concurently on N virtual machines. I'd like to alter my script
> to automatically redistribute the tests evenly across all of the
> machines, i.e. to run 1/N of the tests on each machine. The problem is
> that at Boost.Geometry we have a lot of nested project directories, i.e.
> b2 executed for some upper directory runs the tests in the test-suite in
> this directory and then follows nested projects defined by build-project
> directives, so runs the tests recursively. So e.g. if b2 was executed on
> some virtual machine for the most-top-level directory all of the tests
> would be built anyway.
> 
> Do you have an idea how this could be solved without altering the tests
> structure?
> 
> Is it currently somehow possible (though AFAIK it isn't) or would be
> possible to implement options allowing to e.g.:
> - run b2 only for a certain directory/Jamfile, e.g. --non-recursive

Boost.Build only builds subdirectories when
you explicitly tell it to.

> - ignore arbitrary Jamfile directive, e.g. --ignore-build-project

if ! ( --ignore-build-project in [ modules.peek : ARGV ] )
{
  build-project subdir1 ;
}

> - run only 1 per N tests, an option used together with -jX but running
> tests only for one thread, e.g. b2 -i2 -j4 running only 1 per 4 commands
> starting from 2nd command
> ?
> 

Making something like this work would be really
complex, for a number of reasons:
- The order in which targets are updated is non-deterministic
  for parallel builds.
- Simply partitioning targets won't work without
  a shared filesystem, because some targets depend
  on others.  The partitioning needs to follow the
  dependency graph and some targets (such as libraries)
  may need to be included in more than one subset.
- The partition algorithm cannot take into account
  how long each test takes to run, so it may still
  be suboptimal.

> The last option would be the most convenient for me. The tests would be
> divided as evenly as possible. It wouldn't require to use some
> workarounds i.e. traversing test directories recursively and running b2
> manually for them which could result in low-quality redistribution, e.g.
> if some directory contained many tests and other small number of tests.
> 

  It's actually possible to do something like this
programatically at the virtual target layer.
Basically, you can write a custom target
that constructs a list of test targets and
then filters the ones that you want.

.group = [ MATCH --test-group=(.*) : [ modules.peek : ARGV ] ] ;
rule filter-targets ( project name : property-set : sources * )
{
  # sources should be a list of test targets
  # return a subset of $(sources)
}

generate all-tests
 : # list of tests goes here
 : <generating-rule>@filter-targets
 ;

Note: You can include test-suites and projects in
the list of tests passed to generate.  (If you
use a project, just be careful to avoid recursion.)
filter-targets will be called with an expanded
list of the individual test-cases.

In Christ,
Steven Watanabe