Re: [boost] Checking interest in stacktrace library

Subject: Re: [boost] Checking interest in stacktrace library
From: Niall Douglas (s_sourceforge_at_[hidden])
Date: 2016-06-19 14:55:29

On 18 Jun 2016 at 19:42, Gennadiy Rozental wrote:

> > I think that'd be great - even more so if you could integrate this with 
> > boost.test, so I could state an expectation of what my function calls. 
> > I.e. something like that:
> > 
> > void f(int) {g(); h(); g();};
> > 
> > BOOST_CHECK_CALLS(f(42), g, h, g);
> 
> This is specific case for interaction based testing which is usually being 
> done based on mocks. We would have very hard time to present something more 
> or less generic without mocks.
> 
> That being said, I'd be willing to work closely with anyone interested in 
> bringing stacktrace to boost and integrate it into Boost.Test. It would be 
> very useful to report stacks at the point of unexpected exceptions or fatal 
> errors.

I have some buns in the oven on that front: 
https://github.com/ned14/boost.kerneltest. It's the "other half" of 
automated testing, the part unit test frameworks or dependency 
injection don't usually cover: the Monte Carlo proof of correctness 
part.

Longer run i.e. not in 2016 it'll use LLVM to trace every edge of 
execution and flow of data between the public facing library API and 
any external functions/syscalls it makes. It will then automatically 
permute the inputs and effects from external functions/syscalls to 
formally prove a piece of C++ behaves as specified. Trunk clang and 
trunk GCC is currently required for the requisite compiler and 
runtime support, hence I won't land this till 2017 at the earliest, 
though I have a cool proof of concept test here locally which even 
works right in the MSVC ABI (well done clang!).

KernelTest is a natural fit for other tricks too, like running 
libfuzzer, the AFL, all the sanitisers, valgrind etc. It multithreads 
and multiprocesses naturally, so you can fire all your CPUs at the 
problem for the many hours it takes to cover every permutation.

You can see an example of usage in AFIO v2's very first unit test: 
https://github.com/ned14/boost.afio/blob/master/test/tests/file_handle
_create_close/runner.cpp with an example of a test kernel at 
https://github.com/ned14/boost.afio/blob/master/test/tests/file_handle
_create_close/kernel_async_file_handle.cpp.hpp. As you'll see, the 
simplest use case is to supply a kernel, an array of outcomes, kernel 
parameters, preconditions and postconditions. KernelTest will iterate 
the kernel and check everything is correct, notifying your unit test 
framework of choice of any failures. The new fast ringbuffer logging 
system will trace paths of execution in the kernel, optionally 
logging stacktraces as it goes, to aid future permutation solvers. I 
have yet to implement disk persistence for the logging ringbuffer, 
it's a todo.

This stuff only started working for the first time the Thursday just 
past as I've wrestling with ICEs in clang or VS2015. I've been 
working on it exclusively since the ACCU conference in April as it 
would be embarassing to present AFIO v2 at CppCon without any unit 
test suite as I had to do at ACCU.

I would stress in the strongest possible terms that nobody should 
remotely consider using any of this stuff any time in 2016, wait till 
2017 or probably 2018. But interested people may wish to star my 
github repos and watch out for conference videos, it's highly likely 
KernelTest + the monadic outcomes at its very core will make an 
appearance during the 2017 conference circuit.

Niall

-- 
ned Productions Limited Consulting
http://www.nedproductions.biz/ 
http://ie.linkedin.com/in/nialldouglas/