Re: [boost] Proposal for a Differential Evolution C++ library

Subject: Re: [boost] Proposal for a Differential Evolution C++ library
From: Adrian Michel (adrian_at_[hidden])
Date: 2012-01-10 16:33:51

On 1/9/2012 12:09 PM, Simonson, Lucanus J wrote:
>
>
> I think that numerical optimization is more general than its application to
> machine learning.  While you argue that you'd like to see the library go
> deep in machine learning I don't see anything wrong with a general purpose
> numerical optimization library.  However, DE is more general than numerical
> optimization, while the library seems to be an application of DE to
> numerical optimization.  I'm not sure that DE is even one of the best ways
> to do numerical optimization (I'm pretty sure genetic algorithm is a bad
> way, so I'm skeptical), and I know I'd like a more general interface to a DE
> engine than one designed for numerical optimization problems.  I'd like to
> see the library go wide for max applicability rather than deep.  Hopefully
> that will keep it out of all the different cans of worms that could be
> opened in each application domain.  Put numerical optimization and machine
> learning applications in the example code of the library and submit just the
> core engine as the library itself. Narrowing the scope of a library submission
> is usually the path to success.
>

I forwarded Ken Price (one of the DE co-authors) some of the comments I 
received for his view on things and here's his reply, which hopefully 
should clarify some of the issues that have been raised:

"The skeptic is right in believing that GAs have traditionally done 
poorly at numerical optimization, primarily because they rely on general 
data manipulation operations (bit flips, exchanges, rotations, 
insertions, replacements...)  that are not appropriate for 
real-parameter optimization. DE, however, was specifically designed for 
numerical optimization (although it has since been applied with limited 
success to combinatorial problems). Instead of relying on general data 
manipulation operations, DE's unique differential mutation operation is 
a real-valued vector operation that is appropriate for the numerical 
optimization problem domain. More than just appropriate, the 
differential mutation operation at the core of DE is effective because 
it exploits the tendency of a population to distribute itself around 
function contours. Randomly sampling the difference in the location of 
population members when they are distributed along a function's level 
surfaces turns out to be a very effective way to deal with some of the 
most challenging aspects of numerical optimization, e.g. multiple 
optima, parameter-dependence, slow-down at high resolution and disparate 
parameter sensitivities.

As evidence of DE's utility as a numerical optimizer, I would offer that 
it has been part of Mathematica's numerical optimization toolbox for 
over a decade. It helped the Mathematica team win the SIAM 100-digit 
challenge (ten problems to ten-digit accuracy) by solving what the 
contest participants agreed was the contest's most difficult (numerical 
optimization) problem. (DE also solved one of the other problems in the 
contest). In the many other contests that have been held to compare 
numerical optimizers, DE has consistently placed at or near the top.

DE is also part of the OPTIMUS package, which is highly respected and in 
wide use by several automobile manufactures. Alternatively, skeptics can 
do an Internet search and discover for themselves the enormous range of 
engineering problems the DE has successfully solved, nearly all of which 
are numerical optimization problems, often with multiple, non-linear 
constraints. DE has also proven effective in the domain of 
multi-objective numerical optimization.

Finally, I would suggest that since DE is so simple and easy to 
implement, skeptics ought to just try it. Most people are surprised by 
how effective something as simple as DE can be."