Re: [boost] Ann: Statistical distributions / Mathematical Special Functions

From: John Maddock (john_at_[hidden])
Date: 2006-10-08 13:39:07

Janek Kozicki wrote:

> My conculusion is that maybe I'm wrong to say that it is the shape
> only (although I belive one of the papers I read said exactly that).
> But then - why people are not using a simple polymonial for that?

Right after I posted, I thought "why not use a rational approximation", or 
better a polynomial approximation: the derivatives of the latter are of 
course trivial.

I did a quick Google and found 
http://lib.tkk.fi/Diss/2005/isbn9512275279/article8.pdf#search=%22sigmoid%20function%20polynomial%20approximation%22

Which is basically about implementing the sigmoid in hardware, but Eq15 
suggests that:

1 - 0.5 * (1 - (0.25 * x)^2)

is a viable approximation.

If ldexp is faster than a multiply then the multiplication by powers of 2 
could be replaced with ldexp.

> Oh, I know why.... its integral is this:
>
>  2*atan((2*x-1)/sqrt(7))/sqrt(7)
>
> (this is a sigmoid function)
>
> I belive that backpropagation will work correctly when the
> derivatives/integrals are exact.
>
> Neural networks use both: the sigmoid and it's derivative (a bell
> curve), and they both need to be fast.
>
> Oh, and I found another reason right here in wikipedia :)
>
> "
> A reason for sigmoid popularity in neural networks is because the
> sigmoid function satisfies this property:
>
> sigmoid_derivative(x) = sigmoid(x)*(1-sigmoid(x))
> "
>
> So I was wrong saying that it's only the shape that is important :)
>
>
> So, are you going to add those?
>
> - sigmoid,
> - sigmoid_derivative,
> - double_sigmoid,
> - double_sigmoid_derivative
>
> The derivative of double sigmoid can be done as a connection of
> two sigmoid_derivatives :)

No, I believe it's outside the scope of the current library: providing 
support for the statistical code.

Or to put it another way: there's too much else to do!

> Another question: how about Fast Fourier Transform? is it by
> definition already out of the scope of your library, or maybe not?

Yes, definitely outside the scope, and also patent protected I believe.

John.