Re: Error handling benchmarks: LEAF, Boost Outcome

From: Samuel Neves (samuel.c.p.neves_at_[hidden])
Date: 2019-11-28 17:18:20

Hi,

Maybe I'm missing the point here, but this benchmark is using
outcome::std_outcome<T, EC, EP = std::exception_ptr> in a
no-exceptions scenario, in which case it will always carry around a
std::exception_ptr for no good reason. This is, in fact, the main
source of the slowdown between the two cases in the short error
category.

Applying the following patch to deep_stack_outcome.cpp

-#include <boost/outcome/std_outcome.hpp>
+#include <boost/outcome/std_result.hpp>
 #include <boost/outcome/try.hpp>

 template <class T, class EC>
-using result = outcome::std_outcome<T, EC>;
+using result = outcome::std_result<T, EC, outcome::policy::terminate>;

and increasing the iteration count to 100000, to drown out the noise a
little more, I get (with `clang++-9 -O3 -fno-exceptions -DNDEBUG`)

$ ./a.out
100000 iterations, call depth 32, sizeof(e_heavy_payload) = 4096
LEAF
                |                    | Function | Error | Elapsed
Error type      | At each level      | inlining | rate  |    (μs)
----------------|--------------------|----------|-------|--------
e_error_code    | LEAF_AUTO          | Disabled |    2% |   14253
e_error_code    | LEAF_AUTO          | Enabled  |    2% |    1826
e_error_code    | try_handle_some    | Disabled |    2% |   18333
e_error_code    | try_handle_some    | Enabled  |    2% |    9499
e_error_code    | LEAF_AUTO          | Disabled |   50% |   14645
e_error_code    | LEAF_AUTO          | Enabled  |   50% |    3179
e_error_code    | try_handle_some    | Disabled |   50% |   21092
e_error_code    | try_handle_some    | Enabled  |   50% |   12741
e_error_code    | LEAF_AUTO          | Disabled |   98% |   14353
e_error_code    | LEAF_AUTO          | Enabled  |   98% |    3345
e_error_code    | try_handle_some    | Disabled |   98% |   23413
e_error_code    | try_handle_some    | Enabled  |   98% |   13574
----------------|--------------------|----------|-------|--------
e_system_error  | LEAF_AUTO          | Disabled |    2% |   13347
e_system_error  | LEAF_AUTO          | Enabled  |    2% |    2198
e_system_error  | try_handle_some    | Disabled |    2% |   18138
e_system_error  | try_handle_some    | Enabled  |    2% |   11177
e_system_error  | LEAF_AUTO          | Disabled |   50% |   13788
e_system_error  | LEAF_AUTO          | Enabled  |   50% |    3580
e_system_error  | try_handle_some    | Disabled |   50% |   22136
e_system_error  | try_handle_some    | Enabled  |   50% |   14479
e_system_error  | LEAF_AUTO          | Disabled |   98% |   13178
e_system_error  | LEAF_AUTO          | Enabled  |   98% |    3664
e_system_error  | try_handle_some    | Disabled |   98% |   23790
e_system_error  | try_handle_some    | Enabled  |   98% |   15512
----------------|--------------------|----------|-------|--------
e_heavy_payload | LEAF_AUTO          | Disabled |    2% |   14862
e_heavy_payload | LEAF_AUTO          | Enabled  |    2% |    1941
e_heavy_payload | try_handle_some    | Disabled |    2% |   18502
e_heavy_payload | try_handle_some    | Enabled  |    2% |   11352
e_heavy_payload | LEAF_AUTO          | Disabled |   50% |   20644
e_heavy_payload | LEAF_AUTO          | Enabled  |   50% |   11350
e_heavy_payload | try_handle_some    | Disabled |   50% |   27156
e_heavy_payload | try_handle_some    | Enabled  |   50% |   19173
e_heavy_payload | LEAF_AUTO          | Disabled |   98% |   25337
e_heavy_payload | LEAF_AUTO          | Enabled  |   98% |   18961
e_heavy_payload | try_handle_some    | Disabled |   98% |   34474
e_heavy_payload | try_handle_some    | Enabled  |   98% |   26510

$ ./a.out
100000 iterations, call depth 32, sizeof(e_heavy_payload) = 4096
Outcome
                |                    | Function | Error | Elapsed
Error type      | At each level      | inlining | rate  |    (μs)
----------------|--------------------|----------|-------|--------
e_error_code    | OUTCOME_TRY        | Disabled |    2% |    9501
e_error_code    | OUTCOME_TRY        | Enabled  |    2% |     890
e_error_code    | Handle some errors | Disabled |    2% |    9873
e_error_code    | Handle some errors | Enabled  |    2% |     885
e_error_code    | OUTCOME_TRY        | Disabled |   50% |    9965
e_error_code    | OUTCOME_TRY        | Enabled  |   50% |    2063
e_error_code    | Handle some errors | Disabled |   50% |   11080
e_error_code    | Handle some errors | Enabled  |   50% |    2490
e_error_code    | OUTCOME_TRY        | Disabled |   98% |    7456
e_error_code    | OUTCOME_TRY        | Enabled  |   98% |    1669
e_error_code    | Handle some errors | Disabled |   98% |   10620
e_error_code    | Handle some errors | Enabled  |   98% |    2465
----------------|--------------------|----------|-------|--------
e_system_error  | OUTCOME_TRY        | Disabled |    2% |   15769
e_system_error  | OUTCOME_TRY        | Enabled  |    2% |   14504
e_system_error  | Handle some errors | Disabled |    2% |   16339
e_system_error  | Handle some errors | Enabled  |    2% |   14581
e_system_error  | OUTCOME_TRY        | Disabled |   50% |   20917
e_system_error  | OUTCOME_TRY        | Enabled  |   50% |   17556
e_system_error  | Handle some errors | Disabled |   50% |   21019
e_system_error  | Handle some errors | Enabled  |   50% |   18024
e_system_error  | OUTCOME_TRY        | Disabled |   98% |   24544
e_system_error  | OUTCOME_TRY        | Enabled  |   98% |   19690
e_system_error  | Handle some errors | Disabled |   98% |   23214
e_system_error  | Handle some errors | Enabled  |   98% |   20386
----------------|--------------------|----------|-------|--------
e_heavy_payload | OUTCOME_TRY        | Disabled |    2% |  415474
e_heavy_payload | OUTCOME_TRY        | Enabled  |    2% |  125406
e_heavy_payload | Handle some errors | Disabled |    2% |  416026
e_heavy_payload | Handle some errors | Enabled  |    2% |  126080
e_heavy_payload | OUTCOME_TRY        | Disabled |   50% |  385746
e_heavy_payload | OUTCOME_TRY        | Enabled  |   50% |  142900
e_heavy_payload | Handle some errors | Disabled |   50% |  385494
e_heavy_payload | Handle some errors | Enabled  |   50% |  140713
e_heavy_payload | OUTCOME_TRY        | Disabled |   98% |  348241
e_heavy_payload | OUTCOME_TRY        | Enabled  |   98% |  157487
e_heavy_payload | Handle some errors | Disabled |   98% |  350570
e_heavy_payload | Handle some errors | Enabled  |   98% |  156954

On Wed, Nov 27, 2019 at 9:55 PM Emil Dotchevski via Boost
<boost_at_[hidden]> wrote:
>
> On Wed, Nov 27, 2019 at 4:47 AM Niall Douglas via Boost <
> boost_at_[hidden]> wrote:
> >
> > On 27/11/2019 00:01, Emil Dotchevski via Boost wrote:
> > > I wrote a benchmark program for LEAF and Boost Outcome. Here are the
> > > results:
> > >
> > > https://github.com/zajo/leaf/blob/master/benchmark/benchmark.md
> >
> > These results seem reasonable if you are calling 32 functions deep, and
> > those functions do no work.
> >
> > Returning large sized objects where you cause RVO to be disabled is by
> > definition a benchmark of memcpy().
> >
> > In real world code, the C++ compiler works very hard to avoid calling
> > deep stacks of small functions. It's rare in the real world. Returning
> > large sized objects from functions also tends to be rare in the real
> world.
>
> Interestingly, some of the feedback I got is that the call to rand()
> contaminates the results since it isn't free. I tend to agree with that,
> since the point of a benchmark is to amplify the impact of a system so its
> performance can be evaluated.
>
> Communicating large sized error objects does not cause RVO to be disabled
> with LEAF. It is designed with a strong bias towards the most common use
> case, where callers check for, but do not handle errors.
>
> If the caller is only going to check for failures and forward them to its
> caller, moving error objects one stack frame at a time adds overhead.
> Besides, even though large sized objects are not common, the need to
> communicate several error objects is. It makes no sense to try to bundle
> all that in a return value and hope for the best from the optimizer, given
> that most likely the immediate caller does not handle errors and therefore
> will not access anything other than the discriminant.
>
> To clarify, LEAF also needs to move error objects, including large sized
> error objects up the call chain, but they are moved only to (and between)
> error-handling stack frames, skipping all intermediate check-only levels.
> The benchmark is actually a bit unfair to LEAF in this regard, since the
> "handle some" case includes handling errors at every 4th function call,
> which is excessive in my experience (the "check-only" case does handle
> errors at the top).
>
> It is true that compilers avoid calling deep stacks of small functions,
> which is why the benchmark includes the inline vs. no_inline dimension. The
> simplicity of leaf::result<T> makes it extremely friendly to the optimizer,
> including when inlining is possible. I updated the benchmark paper to show
> generated code:
> https://github.com/zajo/leaf/blob/master/benchmark/benchmark.md#show-me-the-code
> .
>
> > But in truth, when I test the new implementation in an existing
> > Outcome-based codebase, I find no statistically observable difference.
> > If you're doing any real work at all, Outcome can be 10x less efficient
> > and it gets lost by other more dominant work.
>
> This is true for error handling in general. The most important function of
> an error handling library is to allow users to easily communicate any and
> all error objects to error handling contexts where they're needed. That it
> can be done efficiently is an added bonus.
>
> Emil
>
> _______________________________________________
> Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost