
Boost : 
From: John Maddock (john_at_[hidden])
Date: 20061008 13:39:07
Janek Kozicki wrote:
> My conculusion is that maybe I'm wrong to say that it is the shape
> only (although I belive one of the papers I read said exactly that).
> But then  why people are not using a simple polymonial for that?
Right after I posted, I thought "why not use a rational approximation", or
better a polynomial approximation: the derivatives of the latter are of
course trivial.
I did a quick Google and found
http://lib.tkk.fi/Diss/2005/isbn9512275279/article8.pdf#search=%22sigmoid%20function%20polynomial%20approximation%22
Which is basically about implementing the sigmoid in hardware, but Eq15
suggests that:
1  0.5 * (1  (0.25 * x)^2)
is a viable approximation.
If ldexp is faster than a multiply then the multiplication by powers of 2
could be replaced with ldexp.
> Oh, I know why.... its integral is this:
>
> 2*atan((2*x1)/sqrt(7))/sqrt(7)
>
> (this is a sigmoid function)
>
> I belive that backpropagation will work correctly when the
> derivatives/integrals are exact.
>
> Neural networks use both: the sigmoid and it's derivative (a bell
> curve), and they both need to be fast.
>
> Oh, and I found another reason right here in wikipedia :)
>
> "
> A reason for sigmoid popularity in neural networks is because the
> sigmoid function satisfies this property:
>
> sigmoid_derivative(x) = sigmoid(x)*(1sigmoid(x))
> "
>
> So I was wrong saying that it's only the shape that is important :)
>
>
> So, are you going to add those?
>
>  sigmoid,
>  sigmoid_derivative,
>  double_sigmoid,
>  double_sigmoid_derivative
>
> The derivative of double sigmoid can be done as a connection of
> two sigmoid_derivatives :)
No, I believe it's outside the scope of the current library: providing
support for the statistical code.
Or to put it another way: there's too much else to do!
> Another question: how about Fast Fourier Transform? is it by
> definition already out of the scope of your library, or maybe not?
Yes, definitely outside the scope, and also patent protected I believe.
John.
Boost list run by bdawes at acm.org, gregod at cs.rpi.edu, cpdaniel at pacbell.net, john at johnmaddock.co.uk