Tuning scikit-learn parameters using optimization: Work smart not hard

The problem
On previous posts I described how to perform non-linear curve fitting in Ptyhon and Julia. At their core non-linear and linear curve fitting (or regression) are optimization problems in which we find the parameters that minimize an objective function. The entire field of mathematical optimization is concerned with finding the most efficient and accurate methods to minimize such functions.
On the other hand, the current standard to find the optimal values for the parameters of the algorithms used in machine learning is to perform a `random search` or a `grid search` throughout the space of the possible values that such parameters can take. These approaches have several limitations:

1. They are not computationally efficient for large data sets
2. The parameters tested are not informed in any way by the results from the previous step.

The solution
However, the implementation of optimization-driven approaches for scikit-learn is not a trivial matter. Thankfully, James Bergstra and other brave souls have created hyperopt, a Python library for optimizing over awkward search spaces with real-valued, discrete, and conditional dimensions, which makes it ideal for tuning hyper parameters with scikit-learn.

What we need
In order to tune the parameters of a scikit-learn estimator, hyperopt needs the following:
1. Data
2. The objective function to be minimized
3. The search space from which to sample the parameters
4. The algorithm to be used for the minimization of the objective function, and the number of time the optimization should be run

Python implementation

Test function with Boston Housing data set