Re: [boost] [build] Running 1 per N tests

Subject: Re: [boost] [build] Running 1 per N tests
From: Adam Wulkiewicz (adam.wulkiewicz_at_[hidden])
Date: 2016-06-30 19:10:53

Steven Watanabe wrote:
> AMDG
>
> On 06/29/2016 04:34 AM, Adam Wulkiewicz wrote:
>> Hi,
>>
>> At Boost.Geometry we're using CircleCI for testing. They allow to run
>> the tests concurently on N virtual machines. I'd like to alter my script
>> to automatically redistribute the tests evenly across all of the
>> machines, i.e. to run 1/N of the tests on each machine. The problem is
>> that at Boost.Geometry we have a lot of nested project directories, i.e.
>> b2 executed for some upper directory runs the tests in the test-suite in
>> this directory and then follows nested projects defined by build-project
>> directives, so runs the tests recursively. So e.g. if b2 was executed on
>> some virtual machine for the most-top-level directory all of the tests
>> would be built anyway.
>>
>> Do you have an idea how this could be solved without altering the tests
>> structure?
>>
>> Is it currently somehow possible (though AFAIK it isn't) or would be
>> possible to implement options allowing to e.g.:
>> - run b2 only for a certain directory/Jamfile, e.g. --non-recursive
> Boost.Build only builds subdirectories when
> you explicitly tell it to.
>
>> - ignore arbitrary Jamfile directive, e.g. --ignore-build-project
> if ! ( --ignore-build-project in [ modules.peek : ARGV ] )
> {
>    build-project subdir1 ;
> }
>
>> - run only 1 per N tests, an option used together with -jX but running
>> tests only for one thread, e.g. b2 -i2 -j4 running only 1 per 4 commands
>> starting from 2nd command
>> ?
>>
> Making something like this work would be really
> complex, for a number of reasons:
> - The order in which targets are updated is non-deterministic
>    for parallel builds.
> - Simply partitioning targets won't work without
>    a shared filesystem, because some targets depend
>    on others.  The partitioning needs to follow the
>    dependency graph and some targets (such as libraries)
>    may need to be included in more than one subset.
> - The partition algorithm cannot take into account
>    how long each test takes to run, so it may still
>    be suboptimal.
>
>> The last option would be the most convenient for me. The tests would be
>> divided as evenly as possible. It wouldn't require to use some
>> workarounds i.e. traversing test directories recursively and running b2
>> manually for them which could result in low-quality redistribution, e.g.
>> if some directory contained many tests and other small number of tests.
>>
>    It's actually possible to do something like this
> programatically at the virtual target layer.
> Basically, you can write a custom target
> that constructs a list of test targets and
> then filters the ones that you want.
>
> .group = [ MATCH --test-group=(.*) : [ modules.peek : ARGV ] ] ;
> rule filter-targets ( project name : property-set : sources * )
> {
>    # sources should be a list of test targets
>    # return a subset of $(sources)
> }
>
> generate all-tests
>   : # list of tests goes here
>   : <generating-rule>@filter-targets
>   ;
>
> Note: You can include test-suites and projects in
> the list of tests passed to generate.  (If you
> use a project, just be careful to avoid recursion.)
> filter-targets will be called with an expanded
> list of the individual test-cases.

Thanks for the help!

Adam