Re: [Boost-users] [Test] can tests be run in parallel?

Subject: Re: [Boost-users] [Test] can tests be run in parallel?
From: Ovanes Markarian (om_boost_at_[hidden])
Date: 2011-06-10 13:19:21

Hi!

On Fri, Jun 10, 2011 at 5:13 PM, Gennadiy Rozental <rogeeff_at_[hidden]>wrote:

> Ovanes Markarian <om_boost <at> keywallet.com> writes:
> > All in all it sound as a huge amount of work ;)
>
> 1. Frankly I thought more in terms of running test cases in multiple
> processes.
> Either on a local host or in a cloud. Running in multiple threads have some
> small advantages (like no need to serialize extras test case data and some
> existing infrastructure), but IMO it breaks most important restriction of
> unit
> testing - all test units needs to be run independently (unless your classes
> are
> specially prepared for running in MT environment). Running in multi-process
> scenario is much more close to running test cases serially. In addition we
> do
> not need to worry about syncing access Boost.Test internals.
>
I might agree, but it heavily depends on the target platform of test
framework. It is really more clean to have the process based context for TC
execution, but on many embedded platforms you simply do not have processes,
but a single image (which includes your app, operating system etc). Usually
such image is started by the bootloader and executes multiple tasks. Tasks
might be seen in this context as thread equivalents (because each task owns
a separate stack, but they all run within the same address space).
Therefore, it is still beneficial to have a test app which can use
multi-core ability of the embedded target to execute much faster. Because in
some complex systems (e.g. telecoms ) there might be some thousands of TCs.
Running them parallel on a multi-core processor might greatly reduce CI
times. On the other hand it might be simply impossible or very difficult to
produce and run per core images (instead of per processor). I see these
major embedded systems as a C++ niche. Therefore there is a need of some
easily portable task layer for parallel TC execution from within one process
(or let's say execution image).

>
> 2. Managing dependency should not be a problem IMO. Boost.Test has rather
> simple
> model and running test units in parallel should not complicate it much.
>
But how do you see the dependencies in the tcs? Does it make sence to run a
dependent tc if the main tc already failed? There must be some split, which
splits tcs and considers their dependency graphs. What happens now if e.g.
there is a cyclic dependency?

> 3. I do not see, why we need to parallelize on test suites level. IMO it
> should
> be on test case level.
>
Isn't it so that TC and TestSuite have a common base class, which is
additionally is responsible for dependency management? If so this class
should be used as parallelization unit ;)

>
> 4. I do not see a problem with dependence on either Boost.Thread or Intel
> TBB
> (or anything else for that matter). In my design the logic of distribution
> resides in standalone runner application and we can have number of
> different
> runners with different dependencies. There is only some generic
> "distribution
> support logic which will reside in main library.
>
Actually, when I wrote my previous post I was thinking about, but than it is
some kind of a spin off, which in case of dependency upon TBB will require
additional lib to be installed and will not build with the whole boost?

> And it does indeed non trivial piece of work. So if there are any
> volunteers...
> I'll be happy to help with general design and some core infrastructure
> changes
>
We could try that ;) Sounds really interesting to me.

>
> Gennadiy

Regards,
Ovanes