Re: Streamlining benchmarking process

Subject: Re: Streamlining benchmarking process
From: stefan (stefan_at_[hidden])
Date: 2019-05-09 21:32:53

On 2019-05-09 5:11 p.m., Mateusz Loskot wrote:
> Now, I'm wondering this:
> if we maintain benchmark structure similar to tests e.g.
> - test/algo1.cpp, benchmark/algo1.cpp
> - test/algo2.cpp, benchmark/algo2.cpp
> could we define build config that allows to build
> 1) exe per .cpp - useful to bisect, find regressions
> 2) exe from multiple algorithms, related to the same problem of course,
> Â  with one is used as baseline - comparative benchmarking

I'm in favor of 1). Whether that's truly one algorithm or not, however, 
depends on whether the same code is parametrized to yield multiple 
implementations.

In https://github.com/boostorg/ublas/blob/develop/benchmarks/mm_prod.cpp 
you can see how a single compilation unit / executable can represent the 
same algorithm for multiple (template) parameters - value-types, or in 
our case, pixel types or other compile-time parameters.

Even if we combine multiple algorithms into a single executable, I think 
an important requirement is to be able to perform benchmark runs on 
algorithms individually, as that allows to generate comparison charts 
freely. And it scales better as we add more variants.

Stefan

--
       ...ich hab' noch einen Koffer in Berlin...