Re: Streamlining benchmarking process

Subject: Re: Streamlining benchmarking process
From: Mateusz Loskot (mateusz_at_[hidden])
Date: 2019-05-10 10:30:51

On Thu, 9 May 2019 at 14:45, stefan <stefan_at_[hidden]> wrote:
>
> But before diving into a tools discussion, let's quickly collect
> requirements that any tool(s) we agree on needs to meet. Here is my list
> of use-cases. Feel free to augment and complement:
>
> * We should define one benchmark per algorithm, parametrized around
> various axes, such as value- and layout types (pixels, channels, etc.)
> to make it easy to compare different instances of the same algorithm.
>
> * Likewise, we should define benchmarks to be able to run over a range
> of (image-) sizes, as performance will vary greatly on that (and depend
> on the hardware we run on, including but not limited to cache sizes).
>
> * It should be possible to run a single benchmark instance, and produce
> a benchmark result file, containing a table (list of (size,time) pairs).
>
> * It should then be possible to take multiple such files as input, and
> produce a comparative chart.
>
> * It should be possible to implement benchmarks for a given algorithm,
> using external (non Boost.GIL) implementations, to compare Boost.GIL to
> other libraries (OpenCV, for example).
>
> * It should also be possible to later add additional implementations to
> Boost.GIL, and thus augment the parameter space (last summer I mentored
> a Boost.uBLAS project that added GPU support using OpenCL backends, so
> the ability to compare a GPU-based implementation with a host-only
> implementation, especially over a range of problem sizes, was extremely
> useful.

I've forgot to add one or two to this list:

* Installation of benchmark framework should be easy and
  and CPU time friendly on Debian-based Linux, Windows and MacOS.
  On Windows, we can live with deployment using conan or vcpkg.
  On MacOS, I'm not user of this OS myself, but I think brew is the way to go.
  Shortly, we need to think of usability of the framework on CI-s.

* Configuration of the framework should be build system agnostic
  or at least usable with Boost.Build and CMake

If google/benchmark fulfils the requirements listed by Stefan
and the two of mine, then great. It is a well-recognised solid
future-proof framework with active community of contributors.
I'm happy to use it.

Best regards,
-- 
Mateusz Loskot, http://mateusz.loskot.net