From: John Maddock (john_at_[hidden])
Date: 2006-10-08 13:39:07
Janek Kozicki wrote:
> My conculusion is that maybe I'm wrong to say that it is the shape
> only (although I belive one of the papers I read said exactly that).
> But then - why people are not using a simple polymonial for that?
Right after I posted, I thought "why not use a rational approximation", or
better a polynomial approximation: the derivatives of the latter are of
I did a quick Google and found
Which is basically about implementing the sigmoid in hardware, but Eq15
1 - 0.5 * (1 - (0.25 * x)^2)
is a viable approximation.
If ldexp is faster than a multiply then the multiplication by powers of 2
could be replaced with ldexp.
> Oh, I know why.... its integral is this:
> (this is a sigmoid function)
> I belive that backpropagation will work correctly when the
> derivatives/integrals are exact.
> Neural networks use both: the sigmoid and it's derivative (a bell
> curve), and they both need to be fast.
> Oh, and I found another reason right here in wikipedia :)
> A reason for sigmoid popularity in neural networks is because the
> sigmoid function satisfies this property:
> sigmoid_derivative(x) = sigmoid(x)*(1-sigmoid(x))
> So I was wrong saying that it's only the shape that is important :)
> So, are you going to add those?
> - sigmoid,
> - sigmoid_derivative,
> - double_sigmoid,
> - double_sigmoid_derivative
> The derivative of double sigmoid can be done as a connection of
> two sigmoid_derivatives :)
No, I believe it's outside the scope of the current library: providing
support for the statistical code.
Or to put it another way: there's too much else to do!
> Another question: how about Fast Fourier Transform? is it by
> definition already out of the scope of your library, or maybe not?
Yes, definitely outside the scope, and also patent protected I believe.
Boost list run by bdawes at acm.org, gregod at cs.rpi.edu, cpdaniel at pacbell.net, john at johnmaddock.co.uk