Re: [boost] New library proposal: Autodiff

Subject: Re: [boost] New library proposal: Autodiff
From: Matt Pulver (mpulver_at_[hidden])
Date: 2018-12-20 13:46:10

On Thu, Dec 20, 2018 at 1:34 AM Raffi Enficiaud via Boost <
boost_at_[hidden]> wrote:

> it supports only compilation time expressions, is that correct? Would
> it be possible to construct expressions at runtime and then call the
> autodiff on that expression? I believe this would make the library
> extremely useful and comparable to whatever tensorflow or caffe have
>

Let's say one is interested in calculating the first 3 derivatives of a
function f(x) with respect to x, and for a particular computation, the
value of x is 10. The 3 must be known at compile-time, but the 10 can be
decided at runtime:

autodiff::variable<double,3> x(10);

Within the function body f, say x is squared: x*x. This operation must call
the overloaded operator*() defined in autodiff.hpp. Similarly when calling
basic mathematical functions such as exp(x) or tan(x) and compositions of
such operations/function they must call those defined in autodiff.hpp. (You
can also call boost's math functions whose parameters are template
variables, such as tgamma(), but you will not get very good results on the
derivatives since these functions were written to only approximate the
value itself and not the derivatives. Fortunately it is quite easy to add
functions whose derivatives are known to autodiff.hpp.) To answer your
question, as long as the runtime-constructed expressions satisfy this, then
it should work.

> - does it handle vector/arrays/matrices already? It happens often that
> we have a vector function returning eg. an array, and we want the
> differential wrt. one element of that array. Same for matrices.
>

It does not directly included matrix multiplication, if that is what you
are asking. However if each element of a matrix can be an
autodiff::variable<> then the existing matrix multiplication logic should
use the overloaded autodiff operator*() and it should all work. For
example, calls are make to std::inner_product() in which the internal
multiplication is calling the overloaded autodiff operator*().

Matt