skopt module
ScikitOptimize, or skopt
, is a simple and efficient library to
minimize (very) expensive and noisy blackbox functions. It implements
several methods for sequential modelbased optimization. skopt
is reusable
in many contexts and accessible.
Install
pip install scikitoptimize
Getting started
Find the minimum of the noisy function f(x)
over the range 2 < x < 2
with skopt
:
import numpy as np from skopt import gp_minimize def f(x): return (np.sin(5 * x[0]) * (1  np.tanh(x[0] ** 2)) * np.random.randn() * 0.1) res = gp_minimize(f, [(2.0, 2.0)])
For more read our introduction to bayesian optimization and the other examples.
Development
The library is still experimental and under heavy development.
The development version can be installed through:
git clone https://github.com/scikitoptimize/scikitoptimize.git cd scikitoptimize pip install r requirements.txt python setup.py develop
Run the tests by executing pytest
in the top level directory.
""" ScikitOptimize, or `skopt`, is a simple and efficient library to minimize (very) expensive and noisy blackbox functions. It implements several methods for sequential modelbased optimization. `skopt` is reusable in many contexts and accessible. [![Build Status](https://travisci.org/scikitoptimize/scikitoptimize.svg?branch=master)](https://travisci.org/scikitoptimize/scikitoptimize) ## Install ``` pip install scikitoptimize ``` ## Getting started Find the minimum of the noisy function `f(x)` over the range `2 < x < 2` with `skopt`: ```python import numpy as np from skopt import gp_minimize def f(x): return (np.sin(5 * x[0]) * (1  np.tanh(x[0] ** 2)) * np.random.randn() * 0.1) res = gp_minimize(f, [(2.0, 2.0)]) ``` For more read our [introduction to bayesian optimization](https://scikitoptimize.github.io/notebooks/bayesianoptimization.html) and the other [examples](https://github.com/scikitoptimize/scikitoptimize/tree/master/examples). ## Development The library is still experimental and under heavy development. The development version can be installed through: git clone https://github.com/scikitoptimize/scikitoptimize.git cd scikitoptimize pip install r requirements.txt python setup.py develop Run the tests by executing `pytest` in the top level directory. """ from . import acquisition from . import benchmarks from . import callbacks from . import learning from . import optimizer from . import space from .optimizer import dummy_minimize from .optimizer import forest_minimize from .optimizer import gbrt_minimize from .optimizer import gp_minimize from .optimizer import Optimizer from .searchcv import BayesSearchCV from .space import Space from .utils import dump from .utils import expected_minimum from .utils import load __version__ = "0.5.2" __all__ = ( "acquisition", "benchmarks", "callbacks", "learning", "optimizer", "plots", "space", "gp_minimize", "dummy_minimize", "forest_minimize", "gbrt_minimize", "Optimizer", "dump", "load", "expected_minimum", "BayesSearchCV", "Space" )
Functions
def dummy_minimize(
func, dimensions, n_calls=100, x0=None, y0=None, random_state=None, verbose=False, callback=None)
Random search by uniform sampling within the given bounds.
Parameters

func
[callable]: Function to minimize. Should take a single list of parameters and return the objective value.If you have a searchspace where all dimensions have names, then you can use
skopt.utils.use_named_args
as a decorator on your objective function, in order to call it directly with the named arguments. Seeuse_named_args
for an example. 
dimensions
[list, shape=(n_dims,)]: List of search space dimensions. Each search dimension can be defined either as a
(lower_bound, upper_bound)
tuple (forReal
orInteger
dimensions),  a
(lower_bound, upper_bound, prior)
tuple (forReal
dimensions),  as a list of categories (for
Categorical
dimensions), or  an instance of a
Dimension
object (Real
,Integer
orCategorical
).
 a

n_calls
[int, default=100]: Number of calls tofunc
to find the minimum. 
x0
[list, list of lists orNone
]: Initial input points. If it is a list of lists, use it as a list of input points.
 If it is a list, use it as a single initial input point.
 If it is
None
, no initial input points are used.

y0
[list, scalar orNone
]: Evaluation of initial input points. If it is a list, then it corresponds to evaluations of the function
at each element of
x0
: the ith element ofy0
corresponds to the function evaluated at the ith element ofx0
.  If it is a scalar, then it corresponds to the evaluation of the
function at
x0
.  If it is None and
x0
is provided, then the function is evaluated at each element ofx0
.
 If it is a list, then it corresponds to evaluations of the function
at each element of

random_state
[int, RandomState instance, or None (default)]: Set random state to something other than None for reproducible results. 
verbose
[boolean, default=False]: Control the verbosity. It is advised to set the verbosity to True for long optimization runs. 
callback
[callable, list of callables, optional] If callable thencallback(res)
is called after each call tofunc
. If list of callables, then each callable in the list is called.
Returns

res
[OptimizeResult
, scipy object]: The optimization result returned as a OptimizeResult object. Important attributes are:x
[list]: location of the minimum.fun
[float]: function value at the minimum.x_iters
[list of lists]: location of function evaluation for each iteration.func_vals
[array]: function value for each iteration.space
[Space]: the optimisation space.specs
[dict]: the call specifications.rng
[RandomState instance]: State of the random state at the end of minimization.
For more details related to the OptimizeResult object, refer http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.OptimizeResult.html
def dummy_minimize(func, dimensions, n_calls=100, x0=None, y0=None, random_state=None, verbose=False, callback=None): """Random search by uniform sampling within the given bounds. Parameters  * `func` [callable]: Function to minimize. Should take a single list of parameters and return the objective value. If you have a searchspace where all dimensions have names, then you can use `skopt.utils.use_named_args` as a decorator on your objective function, in order to call it directly with the named arguments. See `use_named_args` for an example. * `dimensions` [list, shape=(n_dims,)]: List of search space dimensions. Each search dimension can be defined either as  a `(lower_bound, upper_bound)` tuple (for `Real` or `Integer` dimensions),  a `(lower_bound, upper_bound, prior)` tuple (for `Real` dimensions),  as a list of categories (for `Categorical` dimensions), or  an instance of a `Dimension` object (`Real`, `Integer` or `Categorical`). * `n_calls` [int, default=100]: Number of calls to `func` to find the minimum. * `x0` [list, list of lists or `None`]: Initial input points.  If it is a list of lists, use it as a list of input points.  If it is a list, use it as a single initial input point.  If it is `None`, no initial input points are used. * `y0` [list, scalar or `None`]: Evaluation of initial input points.  If it is a list, then it corresponds to evaluations of the function at each element of `x0` : the ith element of `y0` corresponds to the function evaluated at the ith element of `x0`.  If it is a scalar, then it corresponds to the evaluation of the function at `x0`.  If it is None and `x0` is provided, then the function is evaluated at each element of `x0`. * `random_state` [int, RandomState instance, or None (default)]: Set random state to something other than None for reproducible results. * `verbose` [boolean, default=False]: Control the verbosity. It is advised to set the verbosity to True for long optimization runs. * `callback` [callable, list of callables, optional] If callable then `callback(res)` is called after each call to `func`. If list of callables, then each callable in the list is called. Returns  * `res` [`OptimizeResult`, scipy object]: The optimization result returned as a OptimizeResult object. Important attributes are:  `x` [list]: location of the minimum.  `fun` [float]: function value at the minimum.  `x_iters` [list of lists]: location of function evaluation for each iteration.  `func_vals` [array]: function value for each iteration.  `space` [Space]: the optimisation space.  `specs` [dict]: the call specifications.  `rng` [RandomState instance]: State of the random state at the end of minimization. For more details related to the OptimizeResult object, refer http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.OptimizeResult.html """ # all our calls want random suggestions, except if we need to evaluate # some initial points if x0 is not None and y0 is None: n_random_calls = n_calls  len(x0) else: n_random_calls = n_calls return base_minimize(func, dimensions, base_estimator="dummy", # explicitly set optimizer to sampling as "dummy" # minimizer does not provide gradients. acq_optimizer="sampling", n_calls=n_calls, n_random_starts=n_random_calls, x0=x0, y0=y0, random_state=random_state, verbose=verbose, callback=callback)
def dump(
res, filename, store_objective=True, **kwargs)
Store an skopt optimization result into a file.
Parameters

res
[OptimizeResult
, scipy object]: Optimization result object to be stored. 
filename
[string orpathlib.Path
]: The path of the file in which it is to be stored. The compression method corresponding to one of the supported filename extensions ('.z', '.gz', '.bz2', '.xz' or '.lzma') will be used automatically. 
store_objective
[boolean, default=True]: Whether the objective function should be stored. Setstore_objective
toFalse
if your objective function (.specs['args']['func']
) is unserializable (i.e. if an exception is raised when trying to serialize the optimization result).Notice that if
store_objective
is set toFalse
, a deep copy of the optimization result is created, potentially leading to performance problems ifres
is very large. If the objective function is not critical, one can delete it before callingskopt.dump()
and thus avoid deep copying ofres
. 
**kwargs
[other keyword arguments]: All other keyword arguments will be passed tojoblib.dump
.
def dump(res, filename, store_objective=True, **kwargs): """ Store an skopt optimization result into a file. Parameters  * `res` [`OptimizeResult`, scipy object]: Optimization result object to be stored. * `filename` [string or `pathlib.Path`]: The path of the file in which it is to be stored. The compression method corresponding to one of the supported filename extensions ('.z', '.gz', '.bz2', '.xz' or '.lzma') will be used automatically. * `store_objective` [boolean, default=True]: Whether the objective function should be stored. Set `store_objective` to `False` if your objective function (`.specs['args']['func']`) is unserializable (i.e. if an exception is raised when trying to serialize the optimization result). Notice that if `store_objective` is set to `False`, a deep copy of the optimization result is created, potentially leading to performance problems if `res` is very large. If the objective function is not critical, one can delete it before calling `skopt.dump()` and thus avoid deep copying of `res`. * `**kwargs` [other keyword arguments]: All other keyword arguments will be passed to `joblib.dump`. """ if store_objective: dump_(res, filename, **kwargs) elif 'func' in res.specs['args']: # If the user does not want to store the objective and it is indeed # present in the provided object, then create a deep copy of it and # remove the objective function before dumping it with joblib.dump. res_without_func = deepcopy(res) del res_without_func.specs['args']['func'] dump_(res_without_func, filename, **kwargs) else: # If the user does not want to store the objective and it is already # missing in the provided object, dump it without copying. dump_(res, filename, **kwargs)
def expected_minimum(
res, n_random_starts=20, random_state=None)
Compute the minimum over the predictions of the last surrogate model.
Note that the returned minimum may not necessarily be an accurate prediction of the minimum of the true objective function.
Parameters

res
[OptimizeResult
, scipy object]: The optimization result returned by askopt
minimizer. 
n_random_starts
[int, default=20]: The number of random starts for the minimization of the surrogate model. 
random_state
[int, RandomState instance, or None (default)]: Set random state to something other than None for reproducible results.
Returns

x
[list]: location of the minimum. 
fun
[float]: the surrogate function value at the minimum.
def expected_minimum(res, n_random_starts=20, random_state=None): """ Compute the minimum over the predictions of the last surrogate model. Note that the returned minimum may not necessarily be an accurate prediction of the minimum of the true objective function. Parameters  * `res` [`OptimizeResult`, scipy object]: The optimization result returned by a `skopt` minimizer. * `n_random_starts` [int, default=20]: The number of random starts for the minimization of the surrogate model. * `random_state` [int, RandomState instance, or None (default)]: Set random state to something other than None for reproducible results. Returns  * `x` [list]: location of the minimum. * `fun` [float]: the surrogate function value at the minimum. """ def func(x): reg = res.models[1] x = res.space.transform(x.reshape(1, 1)) return reg.predict(x.reshape(1, 1))[0] xs = [res.x] if n_random_starts > 0: xs.extend(res.space.rvs(n_random_starts, random_state=random_state)) best_x = None best_fun = np.inf for x0 in xs: r = sp_minimize(func, x0=x0, bounds=res.space.bounds) if r.fun < best_fun: best_x = r.x best_fun = r.fun return [v for v in best_x], best_fun
def forest_minimize(
func, dimensions, base_estimator='ET', n_calls=100, n_random_starts=10, acq_func='EI', x0=None, y0=None, random_state=None, verbose=False, callback=None, n_points=10000, xi=0.01, kappa=1.96, n_jobs=1)
Sequential optimisation using decision trees.
A tree based regression model is used to model the expensive to evaluate
function func
. The model is improved by sequentially evaluating
the expensive function at the next best point. Thereby finding the
minimum of func
with as few evaluations as possible.
The total number of evaluations, n_calls
, are performed like the
following. If x0
is provided but not y0
, then the elements of x0
are first evaluated, followed by n_random_starts
evaluations.
Finally, n_calls  len(x0)  n_random_starts
evaluations are
made guided by the surrogate model. If x0
and y0
are both
provided then n_random_starts
evaluations are first made then
n_calls  n_random_starts
subsequent evaluations are made
guided by the surrogate model.
Parameters

func
[callable]: Function to minimize. Should take a single list of parameters and return the objective value.If you have a searchspace where all dimensions have names, then you can use
skopt.utils.use_named_args
as a decorator on your objective function, in order to call it directly with the named arguments. Seeuse_named_args
for an example. 
dimensions
[list, shape=(n_dims,)]: List of search space dimensions. Each search dimension can be defined either as a
(lower_bound, upper_bound)
tuple (forReal
orInteger
dimensions),  a
(lower_bound, upper_bound, prior)
tuple (forReal
dimensions),  as a list of categories (for
Categorical
dimensions), or  an instance of a
Dimension
object (Real
,Integer
orCategorical
).
NOTE: The upper and lower bounds are inclusive for
Integer
dimensions.  a

base_estimator
[string orRegressor
, default="ET"
]: The regressor to use as surrogate model. Can be either"RF"
for random forest regressor"ET"
for extra trees regressor instance of regressor with support for
return_std
in its predict method
The predefined models are initilized with good defaults. If you want to adjust the model parameters pass your own instance of a regressor which returns the mean and standard deviation when making predictions.

n_calls
[int, default=100]: Number of calls tofunc
. 
n_random_starts
[int, default=10]: Number of evaluations offunc
with random points before approximating it withbase_estimator
. 
acq_func
[string, default="LCB"
]: Function to minimize over the forest posterior. Can be either"LCB"
for lower confidence bound."EI"
for negative expected improvement."PI"
for negative probability of improvement. `"EIps" for negated expected improvement per second to take into account the function compute time. Then, the objective function is assumed to return two values, the first being the objective value and the second being the time taken in seconds.
"PIps"
for negated probability of improvement per second. The return type of the objective function is assumed to be similar to that of"EIps"

x0
[list, list of lists orNone
]: Initial input points. If it is a list of lists, use it as a list of input points.
 If it is a list, use it as a single initial input point.
 If it is
None
, no initial input points are used.

y0
[list, scalar orNone
]: Evaluation of initial input points. If it is a list, then it corresponds to evaluations of the function
at each element of
x0
: the ith element ofy0
corresponds to the function evaluated at the ith element ofx0
.  If it is a scalar, then it corresponds to the evaluation of the
function at
x0
.  If it is None and
x0
is provided, then the function is evaluated at each element ofx0
.
 If it is a list, then it corresponds to evaluations of the function
at each element of

random_state
[int, RandomState instance, or None (default)]: Set random state to something other than None for reproducible results. 
verbose
[boolean, default=False]: Control the verbosity. It is advised to set the verbosity to True for long optimization runs. 
callback
[callable, optional] If provided, thencallback(res)
is called after call to func. 
n_points
[int, default=10000]: Number of points to sample when minimizing the acquisition function. 
xi
[float, default=0.01]: Controls how much improvement one wants over the previous best values. Used when the acquisition is either"EI"
or"PI"
. 
kappa
[float, default=1.96]: Controls how much of the variance in the predicted values should be taken into account. If set to be very high, then we are favouring exploration over exploitation and vice versa. Used when the acquisition is"LCB"
. 
n_jobs
[int, default=1]: The number of jobs to run in parallel forfit
andpredict
. If 1, then the number of jobs is set to the number of cores.
Returns

res
[OptimizeResult
, scipy object]: The optimization result returned as a OptimizeResult object. Important attributes are:x
[list]: location of the minimum.fun
[float]: function value at the minimum.models
: surrogate models used for each iteration.x_iters
[list of lists]: location of function evaluation for each iteration.func_vals
[array]: function value for each iteration.space
[Space]: the optimization space.specs
[dict]`: the call specifications.
For more details related to the OptimizeResult object, refer http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.OptimizeResult.html
def forest_minimize(func, dimensions, base_estimator="ET", n_calls=100, n_random_starts=10, acq_func="EI", x0=None, y0=None, random_state=None, verbose=False, callback=None, n_points=10000, xi=0.01, kappa=1.96, n_jobs=1): """Sequential optimisation using decision trees. A tree based regression model is used to model the expensive to evaluate function `func`. The model is improved by sequentially evaluating the expensive function at the next best point. Thereby finding the minimum of `func` with as few evaluations as possible. The total number of evaluations, `n_calls`, are performed like the following. If `x0` is provided but not `y0`, then the elements of `x0` are first evaluated, followed by `n_random_starts` evaluations. Finally, `n_calls  len(x0)  n_random_starts` evaluations are made guided by the surrogate model. If `x0` and `y0` are both provided then `n_random_starts` evaluations are first made then `n_calls  n_random_starts` subsequent evaluations are made guided by the surrogate model. Parameters  * `func` [callable]: Function to minimize. Should take a single list of parameters and return the objective value. If you have a searchspace where all dimensions have names, then you can use `skopt.utils.use_named_args` as a decorator on your objective function, in order to call it directly with the named arguments. See `use_named_args` for an example. * `dimensions` [list, shape=(n_dims,)]: List of search space dimensions. Each search dimension can be defined either as  a `(lower_bound, upper_bound)` tuple (for `Real` or `Integer` dimensions),  a `(lower_bound, upper_bound, prior)` tuple (for `Real` dimensions),  as a list of categories (for `Categorical` dimensions), or  an instance of a `Dimension` object (`Real`, `Integer` or `Categorical`). NOTE: The upper and lower bounds are inclusive for `Integer` dimensions. * `base_estimator` [string or `Regressor`, default=`"ET"`]: The regressor to use as surrogate model. Can be either  `"RF"` for random forest regressor  `"ET"` for extra trees regressor  instance of regressor with support for `return_std` in its predict method The predefined models are initilized with good defaults. If you want to adjust the model parameters pass your own instance of a regressor which returns the mean and standard deviation when making predictions. * `n_calls` [int, default=100]: Number of calls to `func`. * `n_random_starts` [int, default=10]: Number of evaluations of `func` with random points before approximating it with `base_estimator`. * `acq_func` [string, default=`"LCB"`]: Function to minimize over the forest posterior. Can be either  `"LCB"` for lower confidence bound.  `"EI"` for negative expected improvement.  `"PI"` for negative probability of improvement.  `"EIps" for negated expected improvement per second to take into account the function compute time. Then, the objective function is assumed to return two values, the first being the objective value and the second being the time taken in seconds.  `"PIps"` for negated probability of improvement per second. The return type of the objective function is assumed to be similar to that of `"EIps"` * `x0` [list, list of lists or `None`]: Initial input points.  If it is a list of lists, use it as a list of input points.  If it is a list, use it as a single initial input point.  If it is `None`, no initial input points are used. * `y0` [list, scalar or `None`]: Evaluation of initial input points.  If it is a list, then it corresponds to evaluations of the function at each element of `x0` : the ith element of `y0` corresponds to the function evaluated at the ith element of `x0`.  If it is a scalar, then it corresponds to the evaluation of the function at `x0`.  If it is None and `x0` is provided, then the function is evaluated at each element of `x0`. * `random_state` [int, RandomState instance, or None (default)]: Set random state to something other than None for reproducible results. * `verbose` [boolean, default=False]: Control the verbosity. It is advised to set the verbosity to True for long optimization runs. * `callback` [callable, optional] If provided, then `callback(res)` is called after call to func. * `n_points` [int, default=10000]: Number of points to sample when minimizing the acquisition function. * `xi` [float, default=0.01]: Controls how much improvement one wants over the previous best values. Used when the acquisition is either `"EI"` or `"PI"`. * `kappa` [float, default=1.96]: Controls how much of the variance in the predicted values should be taken into account. If set to be very high, then we are favouring exploration over exploitation and vice versa. Used when the acquisition is `"LCB"`. * `n_jobs` [int, default=1]: The number of jobs to run in parallel for `fit` and `predict`. If 1, then the number of jobs is set to the number of cores. Returns  * `res` [`OptimizeResult`, scipy object]: The optimization result returned as a OptimizeResult object. Important attributes are:  `x` [list]: location of the minimum.  `fun` [float]: function value at the minimum.  `models`: surrogate models used for each iteration.  `x_iters` [list of lists]: location of function evaluation for each iteration.  `func_vals` [array]: function value for each iteration.  `space` [Space]: the optimization space.  `specs` [dict]`: the call specifications. For more details related to the OptimizeResult object, refer http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.OptimizeResult.html """ return base_minimize(func, dimensions, base_estimator, n_calls=n_calls, n_points=n_points, n_random_starts=n_random_starts, x0=x0, y0=y0, random_state=random_state, acq_func=acq_func, xi=xi, kappa=kappa, verbose=verbose, callback=callback, acq_optimizer="sampling")
def gbrt_minimize(
func, dimensions, base_estimator=None, n_calls=100, n_random_starts=10, acq_func='EI', acq_optimizer='auto', x0=None, y0=None, random_state=None, verbose=False, callback=None, n_points=10000, xi=0.01, kappa=1.96, n_jobs=1)
Sequential optimization using gradient boosted trees.
Gradient boosted regression trees are used to model the (very)
expensive to evaluate function func
. The model is improved
by sequentially evaluating the expensive function at the next
best point. Thereby finding the minimum of func
with as
few evaluations as possible.
The total number of evaluations, n_calls
, are performed like the
following. If x0
is provided but not y0
, then the elements of x0
are first evaluated, followed by n_random_starts
evaluations.
Finally, n_calls  len(x0)  n_random_starts
evaluations are
made guided by the surrogate model. If x0
and y0
are both
provided then n_random_starts
evaluations are first made then
n_calls  n_random_starts
subsequent evaluations are made
guided by the surrogate model.
Parameters

func
[callable]: Function to minimize. Should take a single list of parameters and return the objective value.If you have a searchspace where all dimensions have names, then you can use
skopt.utils.use_named_args
as a decorator on your objective function, in order to call it directly with the named arguments. Seeuse_named_args
for an example. 
dimensions
[list, shape=(n_dims,)]: List of search space dimensions. Each search dimension can be defined either as a
(lower_bound, upper_bound)
tuple (forReal
orInteger
dimensions),  a
(lower_bound, upper_bound, "prior")
tuple (forReal
dimensions),  as a list of categories (for
Categorical
dimensions), or  an instance of a
Dimension
object (Real
,Integer
orCategorical
).
 a

base_estimator
[GradientBoostingQuantileRegressor
]: The regressor to use as surrogate model 
n_calls
[int, default=100]: Number of calls tofunc
. 
n_random_starts
[int, default=10]: Number of evaluations offunc
with random points before approximating it withbase_estimator
. 
acq_func
[string, default="LCB"
]: Function to minimize over the forest posterior. Can be either"LCB"
for lower confidence bound."EI"
for negative expected improvement."PI"
for negative probability of improvement."EIps"
for negated expected improvement per second to take into account the function compute time. Then, the objective function is assumed to return two values, the first being the objective value and the second being the time taken."PIps"
for negated probability of improvement per second.

x0
[list, list of lists orNone
]: Initial input points. If it is a list of lists, use it as a list of input points.
 If it is a list, use it as a single initial input point.
 If it is
None
, no initial input points are used.

y0
[list, scalar orNone
]: Evaluation of initial input points. If it is a list, then it corresponds to evaluations of the function
at each element of
x0
: the ith element ofy0
corresponds to the function evaluated at the ith element ofx0
.  If it is a scalar, then it corresponds to the evaluation of the
function at
x0
.  If it is None and
x0
is provided, then the function is evaluated at each element ofx0
.
 If it is a list, then it corresponds to evaluations of the function
at each element of

random_state
[int, RandomState instance, or None (default)]: Set random state to something other than None for reproducible results. 
verbose
[boolean, default=False]: Control the verbosity. It is advised to set the verbosity to True for long optimization runs. 
callback
[callable, optional] If provided, thencallback(res)
is called after call to func. 
n_points
[int, default=10000]: Number of points to sample when minimizing the acquisition function. 
xi
[float, default=0.01]: Controls how much improvement one wants over the previous best values. Used when the acquisition is either"EI"
or"PI"
. 
kappa
[float, default=1.96]: Controls how much of the variance in the predicted values should be taken into account. If set to be very high, then we are favouring exploration over exploitation and vice versa. Used when the acquisition is"LCB"
. 
n_jobs
[int, default=1]: The number of jobs to run in parallel forfit
andpredict
. If 1, then the number of jobs is set to the number of cores.
Returns

res
[OptimizeResult
, scipy object]: The optimization result returned as a OptimizeResult object. Important attributes are:x
[list]: location of the minimum.fun
[float]: function value at the minimum.models
: surrogate models used for each iteration.x_iters
[list of lists]: location of function evaluation for each iteration.func_vals
[array]: function value for each iteration.space
[Space]: the optimization space.specs
[dict]`: the call specifications.rng
[RandomState instance]: State of the random state at the end of minimization.
For more details related to the OptimizeResult object, refer http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.OptimizeResult.html
def gbrt_minimize(func, dimensions, base_estimator=None, n_calls=100, n_random_starts=10, acq_func="EI", acq_optimizer="auto", x0=None, y0=None, random_state=None, verbose=False, callback=None, n_points=10000, xi=0.01, kappa=1.96, n_jobs=1): """Sequential optimization using gradient boosted trees. Gradient boosted regression trees are used to model the (very) expensive to evaluate function `func`. The model is improved by sequentially evaluating the expensive function at the next best point. Thereby finding the minimum of `func` with as few evaluations as possible. The total number of evaluations, `n_calls`, are performed like the following. If `x0` is provided but not `y0`, then the elements of `x0` are first evaluated, followed by `n_random_starts` evaluations. Finally, `n_calls  len(x0)  n_random_starts` evaluations are made guided by the surrogate model. If `x0` and `y0` are both provided then `n_random_starts` evaluations are first made then `n_calls  n_random_starts` subsequent evaluations are made guided by the surrogate model. Parameters  * `func` [callable]: Function to minimize. Should take a single list of parameters and return the objective value. If you have a searchspace where all dimensions have names, then you can use `skopt.utils.use_named_args` as a decorator on your objective function, in order to call it directly with the named arguments. See `use_named_args` for an example. * `dimensions` [list, shape=(n_dims,)]: List of search space dimensions. Each search dimension can be defined either as  a `(lower_bound, upper_bound)` tuple (for `Real` or `Integer` dimensions),  a `(lower_bound, upper_bound, "prior")` tuple (for `Real` dimensions),  as a list of categories (for `Categorical` dimensions), or  an instance of a `Dimension` object (`Real`, `Integer` or `Categorical`). * `base_estimator` [`GradientBoostingQuantileRegressor`]: The regressor to use as surrogate model * `n_calls` [int, default=100]: Number of calls to `func`. * `n_random_starts` [int, default=10]: Number of evaluations of `func` with random points before approximating it with `base_estimator`. * `acq_func` [string, default=`"LCB"`]: Function to minimize over the forest posterior. Can be either  `"LCB"` for lower confidence bound.  `"EI"` for negative expected improvement.  `"PI"` for negative probability of improvement.  ``"EIps"`` for negated expected improvement per second to take into account the function compute time. Then, the objective function is assumed to return two values, the first being the objective value and the second being the time taken.  `"PIps"` for negated probability of improvement per second. * `x0` [list, list of lists or `None`]: Initial input points.  If it is a list of lists, use it as a list of input points.  If it is a list, use it as a single initial input point.  If it is `None`, no initial input points are used. * `y0` [list, scalar or `None`]: Evaluation of initial input points.  If it is a list, then it corresponds to evaluations of the function at each element of `x0` : the ith element of `y0` corresponds to the function evaluated at the ith element of `x0`.  If it is a scalar, then it corresponds to the evaluation of the function at `x0`.  If it is None and `x0` is provided, then the function is evaluated at each element of `x0`. * `random_state` [int, RandomState instance, or None (default)]: Set random state to something other than None for reproducible results. * `verbose` [boolean, default=False]: Control the verbosity. It is advised to set the verbosity to True for long optimization runs. * `callback` [callable, optional] If provided, then `callback(res)` is called after call to func. * `n_points` [int, default=10000]: Number of points to sample when minimizing the acquisition function. * `xi` [float, default=0.01]: Controls how much improvement one wants over the previous best values. Used when the acquisition is either `"EI"` or `"PI"`. * `kappa` [float, default=1.96]: Controls how much of the variance in the predicted values should be taken into account. If set to be very high, then we are favouring exploration over exploitation and vice versa. Used when the acquisition is `"LCB"`. * `n_jobs` [int, default=1]: The number of jobs to run in parallel for `fit` and `predict`. If 1, then the number of jobs is set to the number of cores. Returns  * `res` [`OptimizeResult`, scipy object]: The optimization result returned as a OptimizeResult object. Important attributes are:  `x` [list]: location of the minimum.  `fun` [float]: function value at the minimum.  `models`: surrogate models used for each iteration.  `x_iters` [list of lists]: location of function evaluation for each iteration.  `func_vals` [array]: function value for each iteration.  `space` [Space]: the optimization space.  `specs` [dict]`: the call specifications.  `rng` [RandomState instance]: State of the random state at the end of minimization. For more details related to the OptimizeResult object, refer http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.OptimizeResult.html """ # Check params rng = check_random_state(random_state) if base_estimator is None: base_estimator = cook_estimator("GBRT", random_state=rng, n_jobs=n_jobs) return base_minimize(func, dimensions, base_estimator, n_calls=n_calls, n_points=n_points, n_random_starts=n_random_starts, x0=x0, y0=y0, random_state=random_state, xi=xi, kappa=kappa, acq_func=acq_func, verbose=verbose, callback=callback, acq_optimizer="sampling")
def gp_minimize(
func, dimensions, base_estimator=None, n_calls=100, n_random_starts=10, acq_func='gp_hedge', acq_optimizer='auto', x0=None, y0=None, random_state=None, verbose=False, callback=None, n_points=10000, n_restarts_optimizer=5, xi=0.01, kappa=1.96, noise='gaussian', n_jobs=1)
Bayesian optimization using Gaussian Processes.
If every function evaluation is expensive, for instance when the parameters are the hyperparameters of a neural network and the function evaluation is the mean crossvalidation score across ten folds, optimizing the hyperparameters by standard optimization routines would take for ever!
The idea is to approximate the function using a Gaussian process. In other words the function values are assumed to follow a multivariate gaussian. The covariance of the function values are given by a GP kernel between the parameters. Then a smart choice to choose the next parameter to evaluate can be made by the acquisition function over the Gaussian prior which is much quicker to evaluate.
The total number of evaluations, n_calls
, are performed like the
following. If x0
is provided but not y0
, then the elements of x0
are first evaluated, followed by n_random_starts
evaluations.
Finally, n_calls  len(x0)  n_random_starts
evaluations are
made guided by the surrogate model. If x0
and y0
are both
provided then n_random_starts
evaluations are first made then
n_calls  n_random_starts
subsequent evaluations are made
guided by the surrogate model.
Parameters

func
[callable]: Function to minimize. Should take a single list of parameters and return the objective value.If you have a searchspace where all dimensions have names, then you can use
skopt.utils.use_named_args
as a decorator on your objective function, in order to call it directly with the named arguments. Seeuse_named_args
for an example. 
dimensions
[list, shape=(n_dims,)]: List of search space dimensions. Each search dimension can be defined either as a
(lower_bound, upper_bound)
tuple (forReal
orInteger
dimensions),  a
(lower_bound, upper_bound, "prior")
tuple (forReal
dimensions),  as a list of categories (for
Categorical
dimensions), or  an instance of a
Dimension
object (Real
,Integer
orCategorical
).
NOTE: The upper and lower bounds are inclusive for
Integer
dimensions.  a

base_estimator
[a Gaussian process estimator]: The Gaussian process estimator to use for optimization. By default, a Matern kernel is used with the following hyperparameters tuned. All the length scales of the Matern kernel.
 The covariance amplitude that each element is multiplied with.
 Noise that is added to the matern kernel. The noise is assumed to be iid gaussian.

n_calls
[int, default=100]: Number of calls tofunc
. 
n_random_starts
[int, default=10]: Number of evaluations offunc
with random points before approximating it withbase_estimator
. 
acq_func
[string, default="gp_hedge"
]: Function to minimize over the gaussian prior. Can be either"LCB"
for lower confidence bound."EI"
for negative expected improvement."PI"
for negative probability of improvement."gp_hedge"
Probabilistically choose one of the above three acquisition functions at every iteration. The weightage given to these gains can be set by\eta
throughacq_func_kwargs
. The gains
g_i
are initialized to zero.  At every iteration,
 Each acquisition function is optimised independently to
propose an candidate point
X_i
.  Out of all these candidate points, the next point
X_best
is chosen bysoftmax(\eta g_i)
 After fitting the surrogate model with
(X_best, y_best)
, the gains are updated such thatg_i = \mu(X_i)
 Each acquisition function is optimised independently to
propose an candidate point
 The gains
"EIps"
for negated expected improvement per second to take into account the function compute time. Then, the objective function is assumed to return two values, the first being the objective value and the second being the time taken in seconds."PIps"
for negated probability of improvement per second. The return type of the objective function is assumed to be similar to that of `"EIps

acq_optimizer
[string,"sampling"
or"lbfgs"
, default="lbfgs"
]: Method to minimize the acquistion function. The fit model is updated with the optimal value obtained by optimizingacq_func
withacq_optimizer
.The
acq_func
is computed atn_points
sampled randomly. If set to
"auto"
, thenacq_optimizer
is configured on the basis of the space searched over. If the space is Categorical then this is set to be "sampling"`.  If set to
"sampling"
, then the point among thesen_points
where theacq_func
is minimum is the next candidate minimum.  If set to
"lbfgs"
, then The
n_restarts_optimizer
no. of points which the acquisition function is least are taken as start points. "lbfgs"
is run for 20 iterations with these points as initial points to find local minima. The optimal of these local minima is used to update the prior.
 The
 If set to

x0
[list, list of lists orNone
]: Initial input points. If it is a list of lists, use it as a list of input points.
 If it is a list, use it as a single initial input point.
 If it is
None
, no initial input points are used.

y0
[list, scalar orNone
] Evaluation of initial input points. If it is a list, then it corresponds to evaluations of the function
at each element of
x0
: the ith element ofy0
corresponds to the function evaluated at the ith element ofx0
.  If it is a scalar, then it corresponds to the evaluation of the
function at
x0
.  If it is None and
x0
is provided, then the function is evaluated at each element ofx0
.
 If it is a list, then it corresponds to evaluations of the function
at each element of

random_state
[int, RandomState instance, or None (default)]: Set random state to something other than None for reproducible results. 
verbose
[boolean, default=False]: Control the verbosity. It is advised to set the verbosity to True for long optimization runs. 
callback
[callable, list of callables, optional] If callable thencallback(res)
is called after each call tofunc
. If list of callables, then each callable in the list is called. 
n_points
[int, default=10000]: Number of points to sample to determine the next "best" point. Useless if acq_optimizer is set to"lbfgs"
. 
n_restarts_optimizer
[int, default=5]: The number of restarts of the optimizer whenacq_optimizer
is"lbfgs"
. 
kappa
[float, default=1.96]: Controls how much of the variance in the predicted values should be taken into account. If set to be very high, then we are favouring exploration over exploitation and vice versa. Used when the acquisition is"LCB"
. 
xi
[float, default=0.01]: Controls how much improvement one wants over the previous best values. Used when the acquisition is either"EI"
or"PI"
. 
noise
[float, default="gaussian"]: Use noise="gaussian" if the objective returns noisy observations. The noise of each observation is assumed to be iid with mean zero and a fixed variance.
 If the variance is known beforehand, this can be set directly to the variance of the noise.
 Set this to a value close to zero (1e10) if the function is noisefree. Setting to zero might cause stability issues.

n_jobs
[int, default=1] Number of cores to run in parallel while running the lbfgs optimizations over the acquisition function. Valid only whenacq_optimizer
is set to "lbfgs." Defaults to 1 core. Ifn_jobs=1
, then number of jobs is set to number of cores.
Returns

res
[OptimizeResult
, scipy object]: The optimization result returned as a OptimizeResult object. Important attributes are:x
[list]: location of the minimum.fun
[float]: function value at the minimum.models
: surrogate models used for each iteration.x_iters
[list of lists]: location of function evaluation for each iteration.func_vals
[array]: function value for each iteration.space
[Space]: the optimization space.specs
[dict]`: the call specifications.rng
[RandomState instance]: State of the random state at the end of minimization.
For more details related to the OptimizeResult object, refer http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.OptimizeResult.html
def gp_minimize(func, dimensions, base_estimator=None, n_calls=100, n_random_starts=10, acq_func="gp_hedge", acq_optimizer="auto", x0=None, y0=None, random_state=None, verbose=False, callback=None, n_points=10000, n_restarts_optimizer=5, xi=0.01, kappa=1.96, noise="gaussian", n_jobs=1): """Bayesian optimization using Gaussian Processes. If every function evaluation is expensive, for instance when the parameters are the hyperparameters of a neural network and the function evaluation is the mean crossvalidation score across ten folds, optimizing the hyperparameters by standard optimization routines would take for ever! The idea is to approximate the function using a Gaussian process. In other words the function values are assumed to follow a multivariate gaussian. The covariance of the function values are given by a GP kernel between the parameters. Then a smart choice to choose the next parameter to evaluate can be made by the acquisition function over the Gaussian prior which is much quicker to evaluate. The total number of evaluations, `n_calls`, are performed like the following. If `x0` is provided but not `y0`, then the elements of `x0` are first evaluated, followed by `n_random_starts` evaluations. Finally, `n_calls  len(x0)  n_random_starts` evaluations are made guided by the surrogate model. If `x0` and `y0` are both provided then `n_random_starts` evaluations are first made then `n_calls  n_random_starts` subsequent evaluations are made guided by the surrogate model. Parameters  * `func` [callable]: Function to minimize. Should take a single list of parameters and return the objective value. If you have a searchspace where all dimensions have names, then you can use `skopt.utils.use_named_args` as a decorator on your objective function, in order to call it directly with the named arguments. See `use_named_args` for an example. * `dimensions` [list, shape=(n_dims,)]: List of search space dimensions. Each search dimension can be defined either as  a `(lower_bound, upper_bound)` tuple (for `Real` or `Integer` dimensions),  a `(lower_bound, upper_bound, "prior")` tuple (for `Real` dimensions),  as a list of categories (for `Categorical` dimensions), or  an instance of a `Dimension` object (`Real`, `Integer` or `Categorical`). NOTE: The upper and lower bounds are inclusive for `Integer` dimensions. * `base_estimator` [a Gaussian process estimator]: The Gaussian process estimator to use for optimization. By default, a Matern kernel is used with the following hyperparameters tuned.  All the length scales of the Matern kernel.  The covariance amplitude that each element is multiplied with.  Noise that is added to the matern kernel. The noise is assumed to be iid gaussian. * `n_calls` [int, default=100]: Number of calls to `func`. * `n_random_starts` [int, default=10]: Number of evaluations of `func` with random points before approximating it with `base_estimator`. * `acq_func` [string, default=`"gp_hedge"`]: Function to minimize over the gaussian prior. Can be either  `"LCB"` for lower confidence bound.  `"EI"` for negative expected improvement.  `"PI"` for negative probability of improvement.  `"gp_hedge"` Probabilistically choose one of the above three acquisition functions at every iteration. The weightage given to these gains can be set by `\eta` through `acq_func_kwargs`.  The gains `g_i` are initialized to zero.  At every iteration,  Each acquisition function is optimised independently to propose an candidate point `X_i`.  Out of all these candidate points, the next point `X_best` is chosen by `softmax(\eta g_i)`  After fitting the surrogate model with `(X_best, y_best)`, the gains are updated such that `g_i = \mu(X_i)`  `"EIps"` for negated expected improvement per second to take into account the function compute time. Then, the objective function is assumed to return two values, the first being the objective value and the second being the time taken in seconds.  `"PIps"` for negated probability of improvement per second. The return type of the objective function is assumed to be similar to that of `"EIps * `acq_optimizer` [string, `"sampling"` or `"lbfgs"`, default=`"lbfgs"`]: Method to minimize the acquistion function. The fit model is updated with the optimal value obtained by optimizing `acq_func` with `acq_optimizer`. The `acq_func` is computed at `n_points` sampled randomly.  If set to `"auto"`, then `acq_optimizer` is configured on the basis of the space searched over. If the space is Categorical then this is set to be "sampling"`.  If set to `"sampling"`, then the point among these `n_points` where the `acq_func` is minimum is the next candidate minimum.  If set to `"lbfgs"`, then  The `n_restarts_optimizer` no. of points which the acquisition function is least are taken as start points.  `"lbfgs"` is run for 20 iterations with these points as initial points to find local minima.  The optimal of these local minima is used to update the prior. * `x0` [list, list of lists or `None`]: Initial input points.  If it is a list of lists, use it as a list of input points.  If it is a list, use it as a single initial input point.  If it is `None`, no initial input points are used. * `y0` [list, scalar or `None`] Evaluation of initial input points.  If it is a list, then it corresponds to evaluations of the function at each element of `x0` : the ith element of `y0` corresponds to the function evaluated at the ith element of `x0`.  If it is a scalar, then it corresponds to the evaluation of the function at `x0`.  If it is None and `x0` is provided, then the function is evaluated at each element of `x0`. * `random_state` [int, RandomState instance, or None (default)]: Set random state to something other than None for reproducible results. * `verbose` [boolean, default=False]: Control the verbosity. It is advised to set the verbosity to True for long optimization runs. * `callback` [callable, list of callables, optional] If callable then `callback(res)` is called after each call to `func`. If list of callables, then each callable in the list is called. * `n_points` [int, default=10000]: Number of points to sample to determine the next "best" point. Useless if acq_optimizer is set to `"lbfgs"`. * `n_restarts_optimizer` [int, default=5]: The number of restarts of the optimizer when `acq_optimizer` is `"lbfgs"`. * `kappa` [float, default=1.96]: Controls how much of the variance in the predicted values should be taken into account. If set to be very high, then we are favouring exploration over exploitation and vice versa. Used when the acquisition is `"LCB"`. * `xi` [float, default=0.01]: Controls how much improvement one wants over the previous best values. Used when the acquisition is either `"EI"` or `"PI"`. * `noise` [float, default="gaussian"]:  Use noise="gaussian" if the objective returns noisy observations. The noise of each observation is assumed to be iid with mean zero and a fixed variance.  If the variance is known beforehand, this can be set directly to the variance of the noise.  Set this to a value close to zero (1e10) if the function is noisefree. Setting to zero might cause stability issues. * `n_jobs` [int, default=1] Number of cores to run in parallel while running the lbfgs optimizations over the acquisition function. Valid only when `acq_optimizer` is set to "lbfgs." Defaults to 1 core. If `n_jobs=1`, then number of jobs is set to number of cores. Returns  * `res` [`OptimizeResult`, scipy object]: The optimization result returned as a OptimizeResult object. Important attributes are:  `x` [list]: location of the minimum.  `fun` [float]: function value at the minimum.  `models`: surrogate models used for each iteration.  `x_iters` [list of lists]: location of function evaluation for each iteration.  `func_vals` [array]: function value for each iteration.  `space` [Space]: the optimization space.  `specs` [dict]`: the call specifications.  `rng` [RandomState instance]: State of the random state at the end of minimization. For more details related to the OptimizeResult object, refer http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.OptimizeResult.html """ # Check params rng = check_random_state(random_state) space = normalize_dimensions(dimensions) if base_estimator is None: base_estimator = cook_estimator( "GP", space=space, random_state=rng.randint(0, np.iinfo(np.int32).max), noise=noise) return base_minimize( func, space, base_estimator=base_estimator, acq_func=acq_func, xi=xi, kappa=kappa, acq_optimizer=acq_optimizer, n_calls=n_calls, n_points=n_points, n_random_starts=n_random_starts, n_restarts_optimizer=n_restarts_optimizer, x0=x0, y0=y0, random_state=rng, verbose=verbose, callback=callback, n_jobs=n_jobs)
def load(
filename, **kwargs)
Reconstruct a skopt optimization result from a file persisted with skopt.dump.
Notice that the loaded optimization result can be missing
the objective function (.specs['args']['func']
) if dump
was called with store_objective=False
.
Parameters

filename
[string orpathlib.Path
]: The path of the file from which to load the optimization result. 
**kwargs
[other keyword arguments]: All other keyword arguments will be passed tojoblib.load
.
Returns
res
[OptimizeResult
, scipy object]: Reconstructed OptimizeResult instance.
def load(filename, **kwargs): """ Reconstruct a skopt optimization result from a file persisted with skopt.dump. Notice that the loaded optimization result can be missing the objective function (`.specs['args']['func']`) if `skopt.dump` was called with `store_objective=False`. Parameters  * `filename` [string or `pathlib.Path`]: The path of the file from which to load the optimization result. * `**kwargs` [other keyword arguments]: All other keyword arguments will be passed to `joblib.load`. Returns  * `res` [`OptimizeResult`, scipy object]: Reconstructed OptimizeResult instance. """ return load_(filename, **kwargs)
Classes
class BayesSearchCV
Bayesian optimization over hyper parameters.
BayesSearchCV implements a "fit" and a "score" method. It also implements "predict", "predict_proba", "decision_function", "transform" and "inverse_transform" if they are implemented in the estimator used.
The parameters of the estimator used to apply these methods are optimized by crossvalidated search over parameter settings.
In contrast to GridSearchCV, not all parameter values are tried out, but rather a fixed number of parameter settings is sampled from the specified distributions. The number of parameter settings that are tried is given by n_iter.
Parameters are presented as a list of skopt.space.Dimension objects.
Parameters
estimator : estimator object.
A object of that type is instantiated for each search point.
This object is assumed to implement the scikitlearn estimator api.
Either estimator needs to provide a score
function,
or scoring
must be passed.
search_spaces : dict, list of dict or list of tuple containing (dict, int). One of these cases: 1. dictionary, where keys are parameter names (strings) and values are skopt.space.Dimension instances (Real, Integer or Categorical) or any other valid value that defines skopt dimension (see skopt.Optimizer docs). Represents search space over parameters of the provided estimator. 2. list of dictionaries: a list of dictionaries, where every dictionary fits the description given in case 1 above. If a list of dictionary objects is given, then the search is performed sequentially for every parameter space with maximum number of evaluations set to self.n_iter. 3. list of (dict, int > 0): an extension of case 2 above, where first element of every tuple is a dictionary representing some search subspace, similarly as in case 2, and second element is a number of iterations that will be spent optimizing over this subspace.
n_iter : int, default=50
Number of parameter settings that are sampled. n_iter trades
off runtime vs quality of the solution. Consider increasing
n_points
if you want to try more parameter settings in
parallel.
optimizer_kwargs : dict, optional
Dict of arguments passed to :class:Optimizer
. For example,
{'base_estimator': 'RF'}
would use a Random Forest surrogate
instead of the default Gaussian Process.
scoring : string, callable or None, default=None
A string (see model evaluation documentation) or
a scorer callable object / function with signature
scorer(estimator, X, y)
.
If None
, the score
method of the estimator is used.
fit_params : dict, optional Parameters to pass to the fit method.
n_jobs : int, default=1
Number of jobs to run in parallel. At maximum there are
n_points
times cv
jobs available during each iteration.
n_points : int, default=1
Number of parameter settings to sample in parallel. If this does
not align with n_iter
, the last iteration will sample less
points. See also :func:~Optimizer.ask
pre_dispatch : int, or string, optional Controls the number of jobs that get dispatched during parallel execution. Reducing this number can be useful to avoid an explosion of memory consumption when more jobs get dispatched than CPUs can process. This parameter can be:
 None, in which case all the jobs are immediately created and spawned. Use this for lightweight and fastrunning jobs, to avoid delays due to ondemand spawning of the jobs  An int, giving the exact number of total jobs that are spawned  A string, giving an expression as a function of n_jobs, as in '2*n_jobs'
iid : boolean, default=True If True, the data is assumed to be identically distributed across the folds, and the loss minimized is the total loss per sample, and not the mean loss across the folds.
cv : int, crossvalidation generator or an iterable, optional
Determines the crossvalidation splitting strategy.
Possible inputs for cv are:
 None, to use the default 3fold cross validation,
 integer, to specify the number of folds in a (Stratified)KFold
,
 An object to be used as a crossvalidation generator.
 An iterable yielding train, test splits.
For integer/None inputs, if the estimator is a classifier and ``y`` is either binary or multiclass, :class:`StratifiedKFold` is used. In all other cases, :class:`KFold` is used. Refer :ref:`User Guide <cross_validation>` for the various crossvalidation strategies that can be used here.
refit : boolean, default=True Refit the best estimator with the entire dataset. If "False", it is impossible to make predictions using this RandomizedSearchCV instance after fitting.
verbose : integer Controls the verbosity: the higher, the more messages.
random_state : int or RandomState Pseudo random number generator state used for random uniform sampling from lists of possible values instead of scipy.stats distributions.
error_score : 'raise' (default) or numeric Value to assign to the score if an error occurs in estimator fitting. If set to 'raise', the error is raised. If a numeric value is given, FitFailedWarning is raised. This parameter does not affect the refit step, which will always raise the error.
return_train_score : boolean, default=False
If 'True'
, the cv_results_
attribute will include training
scores.
Example
from skopt import BayesSearchCV
parameter ranges are specified by one of below
from skopt.space import Real, Categorical, Integer
from sklearn.datasets import load_iris from sklearn.svm import SVC from sklearn.model_selection import train_test_split
X, y = load_iris(True) X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.75, random_state=0)
loguniform: understand as search over p = exp(x) by varying x
opt = BayesSearchCV( SVC(), { 'C': Real(1e6, 1e+6, prior='loguniform'), 'gamma': Real(1e6, 1e+1, prior='loguniform'), 'degree': Integer(1,8), 'kernel': Categorical(['linear', 'poly', 'rbf']), }, n_iter=32 )
executes bayesian optimization
opt.fit(X_train, y_train)
model can be saved, used for predictions or scoring
print(opt.score(X_test, y_test))
Attributes
cv_results_ : dict of numpy (masked) ndarrays
A dict with keys as column headers and values as columns, that can be
imported into a pandas DataFrame
.
For instance the below given table ++++++  param_kernel  param_gamma  split0_test_score ...rank_test_score +==============+=============+===================+===+===============+  'rbf'  0.1  0.8 ... 2  ++++++  'rbf'  0.2  0.9 ... 1  ++++++  'rbf'  0.3  0.7 ... 1  ++++++ will be represented by a ``cv_results_`` dict of:: { 'param_kernel' : masked_array(data = ['rbf', 'rbf', 'rbf'], mask = False), 'param_gamma' : masked_array(data = [0.1 0.2 0.3], mask = False), 'split0_test_score' : [0.8, 0.9, 0.7], 'split1_test_score' : [0.82, 0.5, 0.7], 'mean_test_score' : [0.81, 0.7, 0.7], 'std_test_score' : [0.02, 0.2, 0.], 'rank_test_score' : [3, 1, 1], 'split0_train_score' : [0.8, 0.9, 0.7], 'split1_train_score' : [0.82, 0.5, 0.7], 'mean_train_score' : [0.81, 0.7, 0.7], 'std_train_score' : [0.03, 0.03, 0.04], 'mean_fit_time' : [0.73, 0.63, 0.43, 0.49], 'std_fit_time' : [0.01, 0.02, 0.01, 0.01], 'mean_score_time' : [0.007, 0.06, 0.04, 0.04], 'std_score_time' : [0.001, 0.002, 0.003, 0.005], 'params' : [{'kernel' : 'rbf', 'gamma' : 0.1}, ...], } NOTE that the key ``'params'`` is used to store a list of parameter settings dict for all the parameter candidates. The ``mean_fit_time``, ``std_fit_time``, ``mean_score_time`` and ``std_score_time`` are all in seconds.
best_estimator_ : estimator Estimator that was chosen by the search, i.e. estimator which gave highest score (or smallest loss if specified) on the left out data. Not available if refit=False.
best_score_ : float Score of best_estimator on the left out data.
best_params_ : dict Parameter setting that gave the best results on the hold out data.
best_index_ : int
The index (of the cv_results_
arrays) which corresponds to the best
candidate parameter setting.
The dict at ``search.cv_results_['params'][search.best_index_]`` gives the parameter setting for the best model, that gives the highest mean score (``search.best_score_``).
scorer_ : function Scorer function used on the held out data to choose the best parameters for the model.
n_splits_ : int The number of crossvalidation splits (folds/iterations).
Notes
The parameters selected are those that maximize the score of the heldout data, according to the scoring parameter.
If n_jobs
was set to a value higher than one, the data is copied for each
parameter setting(and not n_jobs
times). This is done for efficiency
reasons if individual jobs take very little time, but may raise errors if
the dataset is large and not enough memory is available. A workaround in
this case is to set pre_dispatch
. Then, the memory is copied only
pre_dispatch
many times. A reasonable value for pre_dispatch
is 2 *
n_jobs
.
See Also
:class:GridSearchCV
:
Does exhaustive search over a grid of parameters.
class BayesSearchCV(BaseSearchCV): """Bayesian optimization over hyper parameters. BayesSearchCV implements a "fit" and a "score" method. It also implements "predict", "predict_proba", "decision_function", "transform" and "inverse_transform" if they are implemented in the estimator used. The parameters of the estimator used to apply these methods are optimized by crossvalidated search over parameter settings. In contrast to GridSearchCV, not all parameter values are tried out, but rather a fixed number of parameter settings is sampled from the specified distributions. The number of parameter settings that are tried is given by n_iter. Parameters are presented as a list of skopt.space.Dimension objects. Parameters  estimator : estimator object. A object of that type is instantiated for each search point. This object is assumed to implement the scikitlearn estimator api. Either estimator needs to provide a ``score`` function, or ``scoring`` must be passed. search_spaces : dict, list of dict or list of tuple containing (dict, int). One of these cases: 1. dictionary, where keys are parameter names (strings) and values are skopt.space.Dimension instances (Real, Integer or Categorical) or any other valid value that defines skopt dimension (see skopt.Optimizer docs). Represents search space over parameters of the provided estimator. 2. list of dictionaries: a list of dictionaries, where every dictionary fits the description given in case 1 above. If a list of dictionary objects is given, then the search is performed sequentially for every parameter space with maximum number of evaluations set to self.n_iter. 3. list of (dict, int > 0): an extension of case 2 above, where first element of every tuple is a dictionary representing some search subspace, similarly as in case 2, and second element is a number of iterations that will be spent optimizing over this subspace. n_iter : int, default=50 Number of parameter settings that are sampled. n_iter trades off runtime vs quality of the solution. Consider increasing ``n_points`` if you want to try more parameter settings in parallel. optimizer_kwargs : dict, optional Dict of arguments passed to :class:`Optimizer`. For example, ``{'base_estimator': 'RF'}`` would use a Random Forest surrogate instead of the default Gaussian Process. scoring : string, callable or None, default=None A string (see model evaluation documentation) or a scorer callable object / function with signature ``scorer(estimator, X, y)``. If ``None``, the ``score`` method of the estimator is used. fit_params : dict, optional Parameters to pass to the fit method. n_jobs : int, default=1 Number of jobs to run in parallel. At maximum there are ``n_points`` times ``cv`` jobs available during each iteration. n_points : int, default=1 Number of parameter settings to sample in parallel. If this does not align with ``n_iter``, the last iteration will sample less points. See also :func:`~Optimizer.ask` pre_dispatch : int, or string, optional Controls the number of jobs that get dispatched during parallel execution. Reducing this number can be useful to avoid an explosion of memory consumption when more jobs get dispatched than CPUs can process. This parameter can be:  None, in which case all the jobs are immediately created and spawned. Use this for lightweight and fastrunning jobs, to avoid delays due to ondemand spawning of the jobs  An int, giving the exact number of total jobs that are spawned  A string, giving an expression as a function of n_jobs, as in '2*n_jobs' iid : boolean, default=True If True, the data is assumed to be identically distributed across the folds, and the loss minimized is the total loss per sample, and not the mean loss across the folds. cv : int, crossvalidation generator or an iterable, optional Determines the crossvalidation splitting strategy. Possible inputs for cv are:  None, to use the default 3fold cross validation,  integer, to specify the number of folds in a `(Stratified)KFold`,  An object to be used as a crossvalidation generator.  An iterable yielding train, test splits. For integer/None inputs, if the estimator is a classifier and ``y`` is either binary or multiclass, :class:`StratifiedKFold` is used. In all other cases, :class:`KFold` is used. Refer :ref:`User Guide <cross_validation>` for the various crossvalidation strategies that can be used here. refit : boolean, default=True Refit the best estimator with the entire dataset. If "False", it is impossible to make predictions using this RandomizedSearchCV instance after fitting. verbose : integer Controls the verbosity: the higher, the more messages. random_state : int or RandomState Pseudo random number generator state used for random uniform sampling from lists of possible values instead of scipy.stats distributions. error_score : 'raise' (default) or numeric Value to assign to the score if an error occurs in estimator fitting. If set to 'raise', the error is raised. If a numeric value is given, FitFailedWarning is raised. This parameter does not affect the refit step, which will always raise the error. return_train_score : boolean, default=False If ``'True'``, the ``cv_results_`` attribute will include training scores. Example  from skopt import BayesSearchCV # parameter ranges are specified by one of below from skopt.space import Real, Categorical, Integer from sklearn.datasets import load_iris from sklearn.svm import SVC from sklearn.model_selection import train_test_split X, y = load_iris(True) X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.75, random_state=0) # loguniform: understand as search over p = exp(x) by varying x opt = BayesSearchCV( SVC(), { 'C': Real(1e6, 1e+6, prior='loguniform'), 'gamma': Real(1e6, 1e+1, prior='loguniform'), 'degree': Integer(1,8), 'kernel': Categorical(['linear', 'poly', 'rbf']), }, n_iter=32 ) # executes bayesian optimization opt.fit(X_train, y_train) # model can be saved, used for predictions or scoring print(opt.score(X_test, y_test)) Attributes  cv_results_ : dict of numpy (masked) ndarrays A dict with keys as column headers and values as columns, that can be imported into a pandas ``DataFrame``. For instance the below given table ++++++  param_kernel  param_gamma  split0_test_score ...rank_test_score +==============+=============+===================+===+===============+  'rbf'  0.1  0.8 ... 2  ++++++  'rbf'  0.2  0.9 ... 1  ++++++  'rbf'  0.3  0.7 ... 1  ++++++ will be represented by a ``cv_results_`` dict of:: { 'param_kernel' : masked_array(data = ['rbf', 'rbf', 'rbf'], mask = False), 'param_gamma' : masked_array(data = [0.1 0.2 0.3], mask = False), 'split0_test_score' : [0.8, 0.9, 0.7], 'split1_test_score' : [0.82, 0.5, 0.7], 'mean_test_score' : [0.81, 0.7, 0.7], 'std_test_score' : [0.02, 0.2, 0.], 'rank_test_score' : [3, 1, 1], 'split0_train_score' : [0.8, 0.9, 0.7], 'split1_train_score' : [0.82, 0.5, 0.7], 'mean_train_score' : [0.81, 0.7, 0.7], 'std_train_score' : [0.03, 0.03, 0.04], 'mean_fit_time' : [0.73, 0.63, 0.43, 0.49], 'std_fit_time' : [0.01, 0.02, 0.01, 0.01], 'mean_score_time' : [0.007, 0.06, 0.04, 0.04], 'std_score_time' : [0.001, 0.002, 0.003, 0.005], 'params' : [{'kernel' : 'rbf', 'gamma' : 0.1}, ...], } NOTE that the key ``'params'`` is used to store a list of parameter settings dict for all the parameter candidates. The ``mean_fit_time``, ``std_fit_time``, ``mean_score_time`` and ``std_score_time`` are all in seconds. best_estimator_ : estimator Estimator that was chosen by the search, i.e. estimator which gave highest score (or smallest loss if specified) on the left out data. Not available if refit=False. best_score_ : float Score of best_estimator on the left out data. best_params_ : dict Parameter setting that gave the best results on the hold out data. best_index_ : int The index (of the ``cv_results_`` arrays) which corresponds to the best candidate parameter setting. The dict at ``search.cv_results_['params'][search.best_index_]`` gives the parameter setting for the best model, that gives the highest mean score (``search.best_score_``). scorer_ : function Scorer function used on the held out data to choose the best parameters for the model. n_splits_ : int The number of crossvalidation splits (folds/iterations). Notes  The parameters selected are those that maximize the score of the heldout data, according to the scoring parameter. If `n_jobs` was set to a value higher than one, the data is copied for each parameter setting(and not `n_jobs` times). This is done for efficiency reasons if individual jobs take very little time, but may raise errors if the dataset is large and not enough memory is available. A workaround in this case is to set `pre_dispatch`. Then, the memory is copied only `pre_dispatch` many times. A reasonable value for `pre_dispatch` is `2 * n_jobs`. See Also  :class:`GridSearchCV`: Does exhaustive search over a grid of parameters. """ def __init__(self, estimator, search_spaces, optimizer_kwargs=None, n_iter=50, scoring=None, fit_params=None, n_jobs=1, n_points=1, iid=True, refit=True, cv=None, verbose=0, pre_dispatch='2*n_jobs', random_state=None, error_score='raise', return_train_score=False): self.search_spaces = search_spaces self.n_iter = n_iter self.n_points = n_points self.random_state = random_state self.optimizer_kwargs = optimizer_kwargs self._check_search_space(self.search_spaces) super(BayesSearchCV, self).__init__( estimator=estimator, scoring=scoring, fit_params=fit_params, n_jobs=n_jobs, iid=iid, refit=refit, cv=cv, verbose=verbose, pre_dispatch=pre_dispatch, error_score=error_score, return_train_score=return_train_score) def _check_search_space(self, search_space): """Checks whether the search space argument is correct""" if len(search_space) == 0: raise ValueError( "The search_spaces parameter should contain at least one" "nonempty search space, got %s" % search_space ) # check if space is a single dict, convert to list if so if isinstance(search_space, dict): search_space = [search_space] # check if the structure of the space is proper if isinstance(search_space, list): # convert to just a list of dicts dicts_only = [] # 1. check the case when a tuple of space, n_iter is provided for elem in search_space: if isinstance(elem, tuple): if len(elem) != 2: raise ValueError( "All tuples in list of search spaces should have" "length 2, and contain (dict, int), got %s" % elem ) subspace, n_iter = elem if (not isinstance(n_iter, int)) or n_iter < 0: raise ValueError( "Number of iterations in search space should be" "positive integer, got %s in tuple %s " % (n_iter, elem) ) # save subspaces here for further checking dicts_only.append(subspace) elif isinstance(elem, dict): dicts_only.append(elem) else: raise TypeError( "A search space should be provided as a dict or" "tuple (dict, int), got %s" % elem) # 2. check all the dicts for correctness of contents for subspace in dicts_only: for k, v in subspace.items(): check_dimension(v) else: raise TypeError( "Search space should be provided as a dict or list of dict," "got %s" % search_space) # copied for compatibility with 0.19 sklearn from 0.18 BaseSearchCV @property def best_score_(self): check_is_fitted(self, 'cv_results_') return self.cv_results_['mean_test_score'][self.best_index_] # copied for compatibility with 0.19 sklearn from 0.18 BaseSearchCV @property def best_params_(self): check_is_fitted(self, 'cv_results_') return self.cv_results_['params'][self.best_index_] # copied for compatibility with 0.19 sklearn from 0.18 BaseSearchCV def _fit(self, X, y, groups, parameter_iterable): """ Actual fitting, performing the search over parameters. Taken from https://github.com/scikitlearn/scikitlearn/blob/0.18.X .../sklearn/model_selection/_search.py """ estimator = self.estimator cv = sklearn.model_selection._validation.check_cv( self.cv, y, classifier=is_classifier(estimator)) self.scorer_ = check_scoring( self.estimator, scoring=self.scoring) X, y, groups = indexable(X, y, groups) n_splits = cv.get_n_splits(X, y, groups) if self.verbose > 0 and isinstance(parameter_iterable, Sized): n_candidates = len(parameter_iterable) print("Fitting {0} folds for each of {1} candidates, totalling" " {2} fits".format(n_splits, n_candidates, n_candidates * n_splits)) base_estimator = clone(self.estimator) pre_dispatch = self.pre_dispatch cv_iter = list(cv.split(X, y, groups)) out = Parallel( n_jobs=self.n_jobs, verbose=self.verbose, pre_dispatch=pre_dispatch )(delayed(sklearn.model_selection._validation._fit_and_score)( clone(base_estimator), X, y, self.scorer_, train, test, self.verbose, parameters, fit_params=self.fit_params, return_train_score=self.return_train_score, return_n_test_samples=True, return_times=True, return_parameters=True, error_score=self.error_score ) for parameters in parameter_iterable for train, test in cv_iter) # if one choose to see train score, "out" will contain train score info if self.return_train_score: (train_scores, test_scores, test_sample_counts, fit_time, score_time, parameters) = zip(*out) else: (test_scores, test_sample_counts, fit_time, score_time, parameters) = zip(*out) candidate_params = parameters[::n_splits] n_candidates = len(candidate_params) results = dict() def _store(key_name, array, weights=None, splits=False, rank=False): """A small helper to store the scores/times to the cv_results_""" array = np.array(array, dtype=np.float64).reshape(n_candidates, n_splits) if splits: for split_i in range(n_splits): results["split%d_%s" % (split_i, key_name)] = array[:, split_i] array_means = np.average(array, axis=1, weights=weights) results['mean_%s' % key_name] = array_means # Weighted std is not directly available in numpy array_stds = np.sqrt(np.average((array  array_means[:, np.newaxis]) ** 2, axis=1, weights=weights)) results['std_%s' % key_name] = array_stds if rank: results["rank_%s" % key_name] = np.asarray( rankdata(array_means, method='min'), dtype=np.int32) # Computed the (weighted) mean and std for test scores alone # NOTE test_sample counts (weights) remain the same for all candidates test_sample_counts = np.array(test_sample_counts[:n_splits], dtype=np.int) _store('test_score', test_scores, splits=True, rank=True, weights=test_sample_counts if self.iid else None) if self.return_train_score: _store('train_score', train_scores, splits=True) _store('fit_time', fit_time) _store('score_time', score_time) best_index = np.flatnonzero(results["rank_test_score"] == 1)[0] best_parameters = candidate_params[best_index] # Use one MaskedArray and mask all the places where the param is not # applicable for that candidate. Use defaultdict as each candidate may # not contain all the params param_results = defaultdict(partial( MaskedArray, np.empty(n_candidates,), mask=True, dtype=object)) for cand_i, params in enumerate(candidate_params): for name, value in params.items(): # An all masked empty array gets created for the key # `"param_%s" % name` at the first occurence of `name`. # Setting the value at an index also unmasks that index param_results["param_%s" % name][cand_i] = value results.update(param_results) # Store a list of param dicts at the key 'params' results['params'] = candidate_params self.cv_results_ = results self.best_index_ = best_index self.n_splits_ = n_splits if self.refit: # fit the best estimator using the entire dataset # clone first to work around broken estimators best_estimator = clone(base_estimator).set_params( **best_parameters) if y is not None: best_estimator.fit(X, y, **self.fit_params) else: best_estimator.fit(X, **self.fit_params) self.best_estimator_ = best_estimator return self def _fit_best_model(self, X, y): """Fit the estimator copy with best parameters found to the provided data. Parameters  X : arraylike, shape = [n_samples, n_features] Input data, where n_samples is the number of samples and n_features is the number of features. y : arraylike, shape = [n_samples] or [n_samples, n_output], Target relative to X for classification or regression. Returns  self """ self.best_estimator_ = clone(self.estimator) self.best_estimator_.set_params(**self.best_params_) self.best_estimator_.fit(X, y) return self def _make_optimizer(self, params_space): """Instantiate skopt Optimizer class. Parameters  params_space : dict Represents parameter search space. The keys are parameter names (strings) and values are skopt.space.Dimension instances, one of Real, Integer or Categorical. Returns  optimizer: Instance of the `Optimizer` class used for for search in some parameter space. """ kwargs = self.optimizer_kwargs_.copy() kwargs['dimensions'] = dimensions_aslist(params_space) optimizer = Optimizer(**kwargs) return optimizer def _step(self, X, y, search_space, optimizer, groups=None, n_points=1): """Generate n_jobs parameters and evaluate them in parallel. """ # get parameter values to evaluate params = optimizer.ask(n_points=n_points) params_dict = [point_asdict(search_space, p) for p in params] # HACK: self.cv_results_ is reset at every call to _fit, keep current all_cv_results = self.cv_results_ # HACK: this adds compatibility with different versions of sklearn refit = self.refit self.refit = False self._fit(X, y, groups, params_dict) self.refit = refit # merge existing and new cv_results_ for k in self.cv_results_: all_cv_results[k].extend(self.cv_results_[k]) self.cv_results_ = all_cv_results self.best_index_ = np.argmax(self.cv_results_['mean_test_score']) # feed the point and objective back into optimizer local_results = self.cv_results_['mean_test_score'][len(params):] # optimizer minimizes objective, hence provide negative score return optimizer.tell(params, [score for score in local_results]) @property def total_iterations(self): """ Count total iterations that will be taken to explore all subspaces with `fit` method. Returns  max_iter: int, total number of iterations to explore """ total_iter = 0 for elem in self.search_spaces: if isinstance(elem, tuple): space, n_iter = elem else: n_iter = self.n_iter total_iter += n_iter return total_iter def fit(self, X, y=None, groups=None, callback=None): """Run fit on the estimator with randomly drawn parameters. Parameters  X : arraylike or sparse matrix, shape = [n_samples, n_features] The training input samples. y : arraylike, shape = [n_samples] or [n_samples, n_output] Target relative to X for classification or regression (class labels should be integers or strings). groups : arraylike, with shape (n_samples,), optional Group labels for the samples used while splitting the dataset into train/test set. callback: [callable, list of callables, optional] If callable then `callback(res)` is called after each parameter combination tested. If list of callables, then each callable in the list is called. """ # check if space is a single dict, convert to list if so search_spaces = self.search_spaces if isinstance(search_spaces, dict): search_spaces = [search_spaces] callbacks = check_callback(callback) if self.optimizer_kwargs is None: self.optimizer_kwargs_ = {} else: self.optimizer_kwargs_ = dict(self.optimizer_kwargs) random_state = check_random_state(self.random_state) self.optimizer_kwargs_['random_state'] = random_state # Instantiate optimizers for all the search spaces. optimizers = [] for search_space in search_spaces: if isinstance(search_space, tuple): search_space = search_space[0] optimizers.append(self._make_optimizer(search_space)) self.optimizers_ = optimizers # will save the states of the optimizers self.cv_results_ = defaultdict(list) self.best_index_ = None self.multimetric_ = False n_points = self.n_points for search_space, optimizer in zip(search_spaces, optimizers): # if not provided with search subspace, n_iter is taken as # self.n_iter if isinstance(search_space, tuple): search_space, n_iter = search_space else: n_iter = self.n_iter # do the optimization for particular search space while n_iter > 0: # when n_iter < n_points points left for evaluation n_points_adjusted = min(n_iter, n_points) optim_result = self._step( X, y, search_space, optimizer, groups=groups, n_points=n_points_adjusted ) n_iter = n_points if eval_callbacks(callbacks, optim_result): break # Refit the best model on the the whole dataset if self.refit: self._fit_best_model(X, y) return self
Ancestors (in MRO)
 BayesSearchCV
 sklearn.model_selection._search.BaseSearchCV
 abc.NewBase
 sklearn.base.BaseEstimator
 sklearn.base.MetaEstimatorMixin
 builtins.object
Static methods
def __init__(
self, estimator, search_spaces, optimizer_kwargs=None, n_iter=50, scoring=None, fit_params=None, n_jobs=1, n_points=1, iid=True, refit=True, cv=None, verbose=0, pre_dispatch='2*n_jobs', random_state=None, error_score='raise', return_train_score=False)
Initialize self. See help(type(self)) for accurate signature.
def __init__(self, estimator, search_spaces, optimizer_kwargs=None, n_iter=50, scoring=None, fit_params=None, n_jobs=1, n_points=1, iid=True, refit=True, cv=None, verbose=0, pre_dispatch='2*n_jobs', random_state=None, error_score='raise', return_train_score=False): self.search_spaces = search_spaces self.n_iter = n_iter self.n_points = n_points self.random_state = random_state self.optimizer_kwargs = optimizer_kwargs self._check_search_space(self.search_spaces) super(BayesSearchCV, self).__init__( estimator=estimator, scoring=scoring, fit_params=fit_params, n_jobs=n_jobs, iid=iid, refit=refit, cv=cv, verbose=verbose, pre_dispatch=pre_dispatch, error_score=error_score, return_train_score=return_train_score)
def fit(
self, X, y=None, groups=None, callback=None)
Run fit on the estimator with randomly drawn parameters.
Parameters
X : arraylike or sparse matrix, shape = [n_samples, n_features] The training input samples.
y : arraylike, shape = [n_samples] or [n_samples, n_output] Target relative to X for classification or regression (class labels should be integers or strings).
groups : arraylike, with shape (n_samples,), optional Group labels for the samples used while splitting the dataset into train/test set.
callback: [callable, list of callables, optional]
If callable then callback(res)
is called after each parameter
combination tested. If list of callables, then each callable in
the list is called.
def fit(self, X, y=None, groups=None, callback=None): """Run fit on the estimator with randomly drawn parameters. Parameters  X : arraylike or sparse matrix, shape = [n_samples, n_features] The training input samples. y : arraylike, shape = [n_samples] or [n_samples, n_output] Target relative to X for classification or regression (class labels should be integers or strings). groups : arraylike, with shape (n_samples,), optional Group labels for the samples used while splitting the dataset into train/test set. callback: [callable, list of callables, optional] If callable then `callback(res)` is called after each parameter combination tested. If list of callables, then each callable in the list is called. """ # check if space is a single dict, convert to list if so search_spaces = self.search_spaces if isinstance(search_spaces, dict): search_spaces = [search_spaces] callbacks = check_callback(callback) if self.optimizer_kwargs is None: self.optimizer_kwargs_ = {} else: self.optimizer_kwargs_ = dict(self.optimizer_kwargs) random_state = check_random_state(self.random_state) self.optimizer_kwargs_['random_state'] = random_state # Instantiate optimizers for all the search spaces. optimizers = [] for search_space in search_spaces: if isinstance(search_space, tuple): search_space = search_space[0] optimizers.append(self._make_optimizer(search_space)) self.optimizers_ = optimizers # will save the states of the optimizers self.cv_results_ = defaultdict(list) self.best_index_ = None self.multimetric_ = False n_points = self.n_points for search_space, optimizer in zip(search_spaces, optimizers): # if not provided with search subspace, n_iter is taken as # self.n_iter if isinstance(search_space, tuple): search_space, n_iter = search_space else: n_iter = self.n_iter # do the optimization for particular search space while n_iter > 0: # when n_iter < n_points points left for evaluation n_points_adjusted = min(n_iter, n_points) optim_result = self._step( X, y, search_space, optimizer, groups=groups, n_points=n_points_adjusted ) n_iter = n_points if eval_callbacks(callbacks, optim_result): break # Refit the best model on the the whole dataset if self.refit: self._fit_best_model(X, y) return self
Instance variables
var best_params_
var best_score_
var classes_
var grid_scores_
var n_iter
var n_points
var optimizer_kwargs
var random_state
var search_spaces
var total_iterations
Count total iterations that will be taken to explore
all subspaces with fit
method.
Returns
max_iter: int, total number of iterations to explore
class Optimizer
Run bayesian optimisation loop.
An Optimizer
represents the steps of a bayesian optimisation loop. To
use it you need to provide your own loop mechanism. The various
optimisers provided by skopt
use this class under the hood.
Use this class directly if you want to control the iterations of your bayesian optimisation loop.
Parameters

dimensions
[list, shape=(n_dims,)]: List of search space dimensions. Each search dimension can be defined either as a
(lower_bound, upper_bound)
tuple (forReal
orInteger
dimensions),  a
(lower_bound, upper_bound, "prior")
tuple (forReal
dimensions),  as a list of categories (for
Categorical
dimensions), or  an instance of a
Dimension
object (Real
,Integer
orCategorical
).
 a

base_estimator
["GP", "RF", "ET", "GBRT" or sklearn regressor, default="GP"]: Should inherit fromsklearn.base.RegressorMixin
. In addition thepredict
method, should have an optionalreturn_std
argument, which returnsstd(Y  x)`` along with
E[Y  x]`. If base_estimator is one of ["GP", "RF", "ET", "GBRT"], a default surrogate model of the corresponding type is used corresponding to what is used in the minimize functions. 
n_random_starts
[int, default=10]: DEPRECATED, usen_initial_points
instead. 
n_initial_points
[int, default=10]: Number of evaluations offunc
with initialization points before approximating it withbase_estimator
. Points provided asx0
count as initialization points. If len(x0) < n_initial_points additional points are sampled at random. 
acq_func
[string, default="gp_hedge"
]: Function to minimize over the posterior distribution. Can be either"LCB"
for lower confidence bound."EI"
for negative expected improvement."PI"
for negative probability of improvement."gp_hedge"
Probabilistically choose one of the above three acquisition functions at every iteration. The gains
g_i
are initialized to zero.  At every iteration,
 Each acquisition function is optimised independently to
propose an candidate point
X_i
.  Out of all these candidate points, the next point
X_best
is chosen by $softmax(\eta g_i)$  After fitting the surrogate model with
(X_best, y_best)
, the gains are updated such that $g_i = \mu(X_i)$
 Each acquisition function is optimised independently to
propose an candidate point
 The gains
 `"EIps" for negated expected improvement per second to take into account the function compute time. Then, the objective function is assumed to return two values, the first being the objective value and the second being the time taken in seconds.
"PIps"
for negated probability of improvement per second. The return type of the objective function is assumed to be similar to that of `"EIps

acq_optimizer
[string,"sampling"
or"lbfgs"
, default="auto"
]: Method to minimize the acquistion function. The fit model is updated with the optimal value obtained by optimizingacq_func
withacq_optimizer
. If set to
"auto"
, thenacq_optimizer
is configured on the basis of the base_estimator and the space searched over. If the space is Categorical or if the estimator provided based on treemodels then this is set to be "sampling"`.  If set to
"sampling"
, thenacq_func
is optimized by computingacq_func
atn_points
randomly sampled points.  If set to
"lbfgs"
, thenacq_func
is optimized by Sampling
n_restarts_optimizer
points randomly. "lbfgs"
is run for 20 iterations with these points as initial points to find local minima. The optimal of these local minima is used to update the prior.
 Sampling
 If set to

random_state
[int, RandomState instance, or None (default)]: Set random state to something other than None for reproducible results. 
acq_func_kwargs
[dict]: Additional arguments to be passed to the acquistion function. 
acq_optimizer_kwargs
[dict]: Additional arguments to be passed to the acquistion optimizer.
Attributes
Xi
[list]: Points at which objective has been evaluated.yi
[scalar]: Values of objective at corresponding points inXi
.models
[list]: Regression models used to fit observations and compute acquisition function.space
An instance ofskopt.space.Space
. Stores parameter search space used to sample points, bounds, and type of parameters.
class Optimizer(object): """Run bayesian optimisation loop. An `Optimizer` represents the steps of a bayesian optimisation loop. To use it you need to provide your own loop mechanism. The various optimisers provided by `skopt` use this class under the hood. Use this class directly if you want to control the iterations of your bayesian optimisation loop. Parameters  * `dimensions` [list, shape=(n_dims,)]: List of search space dimensions. Each search dimension can be defined either as  a `(lower_bound, upper_bound)` tuple (for `Real` or `Integer` dimensions),  a `(lower_bound, upper_bound, "prior")` tuple (for `Real` dimensions),  as a list of categories (for `Categorical` dimensions), or  an instance of a `Dimension` object (`Real`, `Integer` or `Categorical`). * `base_estimator` ["GP", "RF", "ET", "GBRT" or sklearn regressor, default="GP"]: Should inherit from `sklearn.base.RegressorMixin`. In addition the `predict` method, should have an optional `return_std` argument, which returns `std(Y  x)`` along with `E[Y  x]`. If base_estimator is one of ["GP", "RF", "ET", "GBRT"], a default surrogate model of the corresponding type is used corresponding to what is used in the minimize functions. * `n_random_starts` [int, default=10]: DEPRECATED, use `n_initial_points` instead. * `n_initial_points` [int, default=10]: Number of evaluations of `func` with initialization points before approximating it with `base_estimator`. Points provided as `x0` count as initialization points. If len(x0) < n_initial_points additional points are sampled at random. * `acq_func` [string, default=`"gp_hedge"`]: Function to minimize over the posterior distribution. Can be either  `"LCB"` for lower confidence bound.  `"EI"` for negative expected improvement.  `"PI"` for negative probability of improvement.  `"gp_hedge"` Probabilistically choose one of the above three acquisition functions at every iteration.  The gains `g_i` are initialized to zero.  At every iteration,  Each acquisition function is optimised independently to propose an candidate point `X_i`.  Out of all these candidate points, the next point `X_best` is chosen by $softmax(\eta g_i)$  After fitting the surrogate model with `(X_best, y_best)`, the gains are updated such that $g_i = \mu(X_i)$  `"EIps" for negated expected improvement per second to take into account the function compute time. Then, the objective function is assumed to return two values, the first being the objective value and the second being the time taken in seconds.  `"PIps"` for negated probability of improvement per second. The return type of the objective function is assumed to be similar to that of `"EIps * `acq_optimizer` [string, `"sampling"` or `"lbfgs"`, default=`"auto"`]: Method to minimize the acquistion function. The fit model is updated with the optimal value obtained by optimizing `acq_func` with `acq_optimizer`.  If set to `"auto"`, then `acq_optimizer` is configured on the basis of the base_estimator and the space searched over. If the space is Categorical or if the estimator provided based on treemodels then this is set to be "sampling"`.  If set to `"sampling"`, then `acq_func` is optimized by computing `acq_func` at `n_points` randomly sampled points.  If set to `"lbfgs"`, then `acq_func` is optimized by  Sampling `n_restarts_optimizer` points randomly.  `"lbfgs"` is run for 20 iterations with these points as initial points to find local minima.  The optimal of these local minima is used to update the prior. * `random_state` [int, RandomState instance, or None (default)]: Set random state to something other than None for reproducible results. * `acq_func_kwargs` [dict]: Additional arguments to be passed to the acquistion function. * `acq_optimizer_kwargs` [dict]: Additional arguments to be passed to the acquistion optimizer. Attributes  * `Xi` [list]: Points at which objective has been evaluated. * `yi` [scalar]: Values of objective at corresponding points in `Xi`. * `models` [list]: Regression models used to fit observations and compute acquisition function. * `space` An instance of `skopt.space.Space`. Stores parameter search space used to sample points, bounds, and type of parameters. """ def __init__(self, dimensions, base_estimator="gp", n_random_starts=None, n_initial_points=10, acq_func="gp_hedge", acq_optimizer="auto", random_state=None, acq_func_kwargs=None, acq_optimizer_kwargs=None): self.rng = check_random_state(random_state) # Configure acquisition function # Store and creat acquisition function set self.acq_func = acq_func self.acq_func_kwargs = acq_func_kwargs allowed_acq_funcs = ["gp_hedge", "EI", "LCB", "PI", "EIps", "PIps"] if self.acq_func not in allowed_acq_funcs: raise ValueError("expected acq_func to be in %s, got %s" % (",".join(allowed_acq_funcs), self.acq_func)) # treat hedging method separately if self.acq_func == "gp_hedge": self.cand_acq_funcs_ = ["EI", "LCB", "PI"] self.gains_ = np.zeros(3) else: self.cand_acq_funcs_ = [self.acq_func] if acq_func_kwargs is None: acq_func_kwargs = dict() self.eta = acq_func_kwargs.get("eta", 1.0) # Configure counters of points # Check `n_random_starts` deprecation first if n_random_starts is not None: warnings.warn(("n_random_starts will be removed in favour of " "n_initial_points."), DeprecationWarning) n_initial_points = n_random_starts if n_initial_points < 0: raise ValueError( "Expected `n_initial_points` >= 0, got %d" % n_initial_points) self._n_initial_points = n_initial_points self.n_initial_points_ = n_initial_points # Configure estimator # build base_estimator if doesn't exist if isinstance(base_estimator, str): base_estimator = cook_estimator( base_estimator, space=dimensions, random_state=self.rng.randint(0, np.iinfo(np.int32).max)) # check if regressor if not is_regressor(base_estimator) and base_estimator is not None: raise ValueError( "%s has to be a regressor." % base_estimator) # treat per second acqusition function specially is_multi_regressor = isinstance(base_estimator, MultiOutputRegressor) if "ps" in self.acq_func and not is_multi_regressor: self.base_estimator_ = MultiOutputRegressor(base_estimator) else: self.base_estimator_ = base_estimator # Configure optimizer # decide optimizer based on gradient information if acq_optimizer == "auto": if has_gradients(self.base_estimator_): acq_optimizer = "lbfgs" else: acq_optimizer = "sampling" if acq_optimizer not in ["lbfgs", "sampling"]: raise ValueError("Expected acq_optimizer to be 'lbfgs' or " "'sampling', got {0}".format(acq_optimizer)) if (not has_gradients(self.base_estimator_) and acq_optimizer != "sampling"): raise ValueError("The regressor {0} should run with " "acq_optimizer" "='sampling'.".format(type(base_estimator))) self.acq_optimizer = acq_optimizer # record other arguments if acq_optimizer_kwargs is None: acq_optimizer_kwargs = dict() self.n_points = acq_optimizer_kwargs.get("n_points", 10000) self.n_restarts_optimizer = acq_optimizer_kwargs.get( "n_restarts_optimizer", 5) n_jobs = acq_optimizer_kwargs.get("n_jobs", 1) self.n_jobs = n_jobs self.acq_optimizer_kwargs = acq_optimizer_kwargs # Configure search space # normalize space if GP regressor if isinstance(self.base_estimator_, GaussianProcessRegressor): dimensions = normalize_dimensions(dimensions) self.space = Space(dimensions) # record categorical and noncategorical indices self._cat_inds = [] self._non_cat_inds = [] for ind, dim in enumerate(self.space.dimensions): if isinstance(dim, Categorical): self._cat_inds.append(ind) else: self._non_cat_inds.append(ind) # Initialize storage for optimization self.models = [] self.Xi = [] self.yi = [] # Initialize cache for `ask` method responses # This ensures that multiple calls to `ask` with n_points set # return same sets of points. Reset to {} at every call to `tell`. self.cache_ = {} def copy(self, random_state=None): """Create a shallow copy of an instance of the optimizer. Parameters  * `random_state` [int, RandomState instance, or None (default)]: Set the random state of the copy. """ optimizer = Optimizer( dimensions=self.space.dimensions, base_estimator=self.base_estimator_, n_initial_points=self.n_initial_points_, acq_func=self.acq_func, acq_optimizer=self.acq_optimizer, acq_func_kwargs=self.acq_func_kwargs, acq_optimizer_kwargs=self.acq_optimizer_kwargs, random_state=random_state, ) if hasattr(self, "gains_"): optimizer.gains_ = np.copy(self.gains_) if self.Xi: optimizer._tell(self.Xi, self.yi) return optimizer def ask(self, n_points=None, strategy="cl_min"): """Query point or multiple points at which objective should be evaluated. * `n_points` [int or None, default=None]: Number of points returned by the ask method. If the value is None, a single point to evaluate is returned. Otherwise a list of points to evaluate is returned of size n_points. This is useful if you can evaluate your objective in parallel, and thus obtain more objective function evaluations per unit of time. * `strategy` [string, default=`"cl_min"`]: Method to use to sample multiple points (see also `n_points` description). This parameter is ignored if n_points = None. Supported options are `"cl_min"`, `"cl_mean"` or `"cl_max"`.  If set to `"cl_min"`, then constant liar strtategy is used with lie objective value being minimum of observed objective values. `"cl_mean"` and `"cl_max"` means mean and max of values respectively. For details on this strategy see: https://hal.archivesouvertes.fr/hal00732512/document With this strategy a copy of optimizer is created, which is then asked for a point, and the point is told to the copy of optimizer with some fake objective (lie), the next point is asked from copy, it is also told to the copy with fake objective and so on. The type of lie defines different flavours of `cl_x` strategies. """ if n_points is None: return self._ask() supported_strategies = ["cl_min", "cl_mean", "cl_max"] if not (isinstance(n_points, int) and n_points > 0): raise ValueError( "n_points should be int > 0, got " + str(n_points) ) if strategy not in supported_strategies: raise ValueError( "Expected parallel_strategy to be one of " + str(supported_strategies) + ", " + "got %s" % strategy ) # Caching the result with n_points not None. If some new parameters # are provided to the ask, the cache_ is not used. if (n_points, strategy) in self.cache_: return self.cache_[(n_points, strategy)] # Copy of the optimizer is made in order to manage the # deletion of points with "lie" objective (the copy of # oiptimizer is simply discarded) opt = self.copy(random_state=self.rng.randint(0, np.iinfo(np.int32).max)) X = [] for i in range(n_points): x = opt.ask() X.append(x) ti_available = "ps" in self.acq_func and len(opt.yi) > 0 ti = [t for (_, t) in opt.yi] if ti_available else None if strategy == "cl_min": y_lie = np.min(opt.yi) if opt.yi else 0.0 # CLmin lie t_lie = np.min(ti) if ti is not None else log(sys.float_info.max) elif strategy == "cl_mean": y_lie = np.mean(opt.yi) if opt.yi else 0.0 # CLmean lie t_lie = np.mean(ti) if ti is not None else log(sys.float_info.max) else: y_lie = np.max(opt.yi) if opt.yi else 0.0 # CLmax lie t_lie = np.max(ti) if ti is not None else log(sys.float_info.max) # Lie to the optimizer. if "ps" in self.acq_func: # Use `_tell()` instead of `tell()` to prevent repeated # log transformations of the computation times. opt._tell(x, (y_lie, t_lie)) else: opt._tell(x, y_lie) self.cache_ = {(n_points, strategy): X} # cache_ the result return X def _ask(self): """Suggest next point at which to evaluate the objective. Return a random point while not at least `n_initial_points` observations have been `tell`ed, after that `base_estimator` is used to determine the next point. """ if self._n_initial_points > 0 or self.base_estimator_ is None: # this will not make a copy of `self.rng` and hence keep advancing # our random state. return self.space.rvs(random_state=self.rng)[0] else: if not self.models: raise RuntimeError("Random evaluations exhausted and no " "model has been fit.") next_x = self._next_x min_delta_x = min([self.space.distance(next_x, xi) for xi in self.Xi]) if abs(min_delta_x) <= 1e8: warnings.warn("The objective has been evaluated " "at this point before.") # return point computed from last call to tell() return next_x def tell(self, x, y, fit=True): """Record an observation (or several) of the objective function. Provide values of the objective function at points suggested by `ask()` or other points. By default a new model will be fit to all observations. The new model is used to suggest the next point at which to evaluate the objective. This point can be retrieved by calling `ask()`. To add observations without fitting a new model set `fit` to False. To add multiple observations in a batch pass a listoflists for `x` and a list of scalars for `y`. Parameters  * `x` [list or listoflists]: Point at which objective was evaluated. * `y` [scalar or list]: Value of objective at `x`. * `fit` [bool, default=True] Fit a model to observed evaluations of the objective. A model will only be fitted after `n_initial_points` points have been told to the optimizer irrespective of the value of `fit`. """ check_x_in_space(x, self.space) self._check_y_is_valid(x, y) # take the logarithm of the computation times if "ps" in self.acq_func: if is_2Dlistlike(x): y = [[val, log(t)] for (val, t) in y] elif is_listlike(x): y = list(y) y[1] = log(y[1]) return self._tell(x, y, fit=fit) def _tell(self, x, y, fit=True): """Perform the actual work of incorporating one or more new points. See `tell()` for the full description. This method exists to give access to the internals of adding points by side stepping all input validation and transformation.""" if "ps" in self.acq_func: if is_2Dlistlike(x): self.Xi.extend(x) self.yi.extend(y) self._n_initial_points = len(y) elif is_listlike(x): self.Xi.append(x) self.yi.append(y) self._n_initial_points = 1 # if y isn't a scalar it means we have been handed a batch of points elif is_listlike(y) and is_2Dlistlike(x): self.Xi.extend(x) self.yi.extend(y) self._n_initial_points = len(y) elif is_listlike(x): self.Xi.append(x) self.yi.append(y) self._n_initial_points = 1 else: raise ValueError("Type of arguments `x` (%s) and `y` (%s) " "not compatible." % (type(x), type(y))) # optimizer learned something new  discard cache self.cache_ = {} # after being "told" n_initial_points we switch from sampling # random points to using a surrogate model if (fit and self._n_initial_points <= 0 and self.base_estimator_ is not None): transformed_bounds = np.array(self.space.transformed_bounds) est = clone(self.base_estimator_) with warnings.catch_warnings(): warnings.simplefilter("ignore") est.fit(self.space.transform(self.Xi), self.yi) if hasattr(self, "next_xs_") and self.acq_func == "gp_hedge": self.gains_ = est.predict(np.vstack(self.next_xs_)) self.models.append(est) # even with BFGS as optimizer we want to sample a large number # of points and then pick the best ones as starting points X = self.space.transform(self.space.rvs( n_samples=self.n_points, random_state=self.rng)) self.next_xs_ = [] for cand_acq_func in self.cand_acq_funcs_: values = _gaussian_acquisition( X=X, model=est, y_opt=np.min(self.yi), acq_func=cand_acq_func, acq_func_kwargs=self.acq_func_kwargs) # Find the minimum of the acquisition function by randomly # sampling points from the space if self.acq_optimizer == "sampling": next_x = X[np.argmin(values)] # Use BFGS to find the mimimum of the acquisition function, the # minimization starts from `n_restarts_optimizer` different # points and the best minimum is used elif self.acq_optimizer == "lbfgs": x0 = X[np.argsort(values)[:self.n_restarts_optimizer]] with warnings.catch_warnings(): warnings.simplefilter("ignore") results = Parallel(n_jobs=self.n_jobs)( delayed(fmin_l_bfgs_b)( gaussian_acquisition_1D, x, args=(est, np.min(self.yi), cand_acq_func, self.acq_func_kwargs), bounds=self.space.transformed_bounds, approx_grad=False, maxiter=20) for x in x0) cand_xs = np.array([r[0] for r in results]) cand_acqs = np.array([r[1] for r in results]) next_x = cand_xs[np.argmin(cand_acqs)] # lbfgs should handle this but just in case there are # precision errors. if not self.space.is_categorical: next_x = np.clip( next_x, transformed_bounds[:, 0], transformed_bounds[:, 1]) self.next_xs_.append(next_x) if self.acq_func == "gp_hedge": logits = np.array(self.gains_) logits = np.max(logits) exp_logits = np.exp(self.eta * logits) probs = exp_logits / np.sum(exp_logits) next_x = self.next_xs_[np.argmax(self.rng.multinomial(1, probs))] else: next_x = self.next_xs_[0] # note the need for [0] at the end self._next_x = self.space.inverse_transform( next_x.reshape((1, 1)))[0] # Pack results return create_result(self.Xi, self.yi, self.space, self.rng, models=self.models) def _check_y_is_valid(self, x, y): """Check if the shape and types of x and y are consistent.""" if "ps" in self.acq_func: if is_2Dlistlike(x): if not (np.ndim(y) == 2 and np.shape(y)[1] == 2): raise TypeError("expected y to be a list of (func_val, t)") elif is_listlike(x): if not (np.ndim(y) == 1 and len(y) == 2): raise TypeError("expected y to be (func_val, t)") # if y isn't a scalar it means we have been handed a batch of points elif is_listlike(y) and is_2Dlistlike(x): for y_value in y: if not isinstance(y_value, Number): raise ValueError("expected y to be a list of scalars") elif is_listlike(x): if not isinstance(y, Number): raise ValueError("`func` should return a scalar") else: raise ValueError("Type of arguments `x` (%s) and `y` (%s) " "not compatible." % (type(x), type(y))) def run(self, func, n_iter=1): """Execute ask() + tell() `n_iter` times""" for _ in range(n_iter): x = self.ask() self.tell(x, func(x)) return create_result(self.Xi, self.yi, self.space, self.rng, models=self.models)
Ancestors (in MRO)
 Optimizer
 builtins.object
Static methods
def __init__(
self, dimensions, base_estimator='gp', n_random_starts=None, n_initial_points=10, acq_func='gp_hedge', acq_optimizer='auto', random_state=None, acq_func_kwargs=None, acq_optimizer_kwargs=None)
Initialize self. See help(type(self)) for accurate signature.
def __init__(self, dimensions, base_estimator="gp", n_random_starts=None, n_initial_points=10, acq_func="gp_hedge", acq_optimizer="auto", random_state=None, acq_func_kwargs=None, acq_optimizer_kwargs=None): self.rng = check_random_state(random_state) # Configure acquisition function # Store and creat acquisition function set self.acq_func = acq_func self.acq_func_kwargs = acq_func_kwargs allowed_acq_funcs = ["gp_hedge", "EI", "LCB", "PI", "EIps", "PIps"] if self.acq_func not in allowed_acq_funcs: raise ValueError("expected acq_func to be in %s, got %s" % (",".join(allowed_acq_funcs), self.acq_func)) # treat hedging method separately if self.acq_func == "gp_hedge": self.cand_acq_funcs_ = ["EI", "LCB", "PI"] self.gains_ = np.zeros(3) else: self.cand_acq_funcs_ = [self.acq_func] if acq_func_kwargs is None: acq_func_kwargs = dict() self.eta = acq_func_kwargs.get("eta", 1.0) # Configure counters of points # Check `n_random_starts` deprecation first if n_random_starts is not None: warnings.warn(("n_random_starts will be removed in favour of " "n_initial_points."), DeprecationWarning) n_initial_points = n_random_starts if n_initial_points < 0: raise ValueError( "Expected `n_initial_points` >= 0, got %d" % n_initial_points) self._n_initial_points = n_initial_points self.n_initial_points_ = n_initial_points # Configure estimator # build base_estimator if doesn't exist if isinstance(base_estimator, str): base_estimator = cook_estimator( base_estimator, space=dimensions, random_state=self.rng.randint(0, np.iinfo(np.int32).max)) # check if regressor if not is_regressor(base_estimator) and base_estimator is not None: raise ValueError( "%s has to be a regressor." % base_estimator) # treat per second acqusition function specially is_multi_regressor = isinstance(base_estimator, MultiOutputRegressor) if "ps" in self.acq_func and not is_multi_regressor: self.base_estimator_ = MultiOutputRegressor(base_estimator) else: self.base_estimator_ = base_estimator # Configure optimizer # decide optimizer based on gradient information if acq_optimizer == "auto": if has_gradients(self.base_estimator_): acq_optimizer = "lbfgs" else: acq_optimizer = "sampling" if acq_optimizer not in ["lbfgs", "sampling"]: raise ValueError("Expected acq_optimizer to be 'lbfgs' or " "'sampling', got {0}".format(acq_optimizer)) if (not has_gradients(self.base_estimator_) and acq_optimizer != "sampling"): raise ValueError("The regressor {0} should run with " "acq_optimizer" "='sampling'.".format(type(base_estimator))) self.acq_optimizer = acq_optimizer # record other arguments if acq_optimizer_kwargs is None: acq_optimizer_kwargs = dict() self.n_points = acq_optimizer_kwargs.get("n_points", 10000) self.n_restarts_optimizer = acq_optimizer_kwargs.get( "n_restarts_optimizer", 5) n_jobs = acq_optimizer_kwargs.get("n_jobs", 1) self.n_jobs = n_jobs self.acq_optimizer_kwargs = acq_optimizer_kwargs # Configure search space # normalize space if GP regressor if isinstance(self.base_estimator_, GaussianProcessRegressor): dimensions = normalize_dimensions(dimensions) self.space = Space(dimensions) # record categorical and noncategorical indices self._cat_inds = [] self._non_cat_inds = [] for ind, dim in enumerate(self.space.dimensions): if isinstance(dim, Categorical): self._cat_inds.append(ind) else: self._non_cat_inds.append(ind) # Initialize storage for optimization self.models = [] self.Xi = [] self.yi = [] # Initialize cache for `ask` method responses # This ensures that multiple calls to `ask` with n_points set # return same sets of points. Reset to {} at every call to `tell`. self.cache_ = {}
def ask(
self, n_points=None, strategy='cl_min')
Query point or multiple points at which objective should be evaluated.

n_points
[int or None, default=None]: Number of points returned by the ask method. If the value is None, a single point to evaluate is returned. Otherwise a list of points to evaluate is returned of size n_points. This is useful if you can evaluate your objective in parallel, and thus obtain more objective function evaluations per unit of time. 
strategy
[string, default="cl_min"
]: Method to use to sample multiple points (see alson_points
description). This parameter is ignored if n_points = None. Supported options are"cl_min"
,"cl_mean"
or"cl_max"
. If set to
"cl_min"
, then constant liar strtategy is used with lie objective value being minimum of observed objective values."cl_mean"
and"cl_max"
means mean and max of values respectively. For details on this strategy see:
https://hal.archivesouvertes.fr/hal00732512/document
With this strategy a copy of optimizer is created, which is then asked for a point, and the point is told to the copy of optimizer with some fake objective (lie), the next point is asked from copy, it is also told to the copy with fake objective and so on. The type of lie defines different flavours of
cl_x
strategies.  If set to
def ask(self, n_points=None, strategy="cl_min"): """Query point or multiple points at which objective should be evaluated. * `n_points` [int or None, default=None]: Number of points returned by the ask method. If the value is None, a single point to evaluate is returned. Otherwise a list of points to evaluate is returned of size n_points. This is useful if you can evaluate your objective in parallel, and thus obtain more objective function evaluations per unit of time. * `strategy` [string, default=`"cl_min"`]: Method to use to sample multiple points (see also `n_points` description). This parameter is ignored if n_points = None. Supported options are `"cl_min"`, `"cl_mean"` or `"cl_max"`.  If set to `"cl_min"`, then constant liar strtategy is used with lie objective value being minimum of observed objective values. `"cl_mean"` and `"cl_max"` means mean and max of values respectively. For details on this strategy see: https://hal.archivesouvertes.fr/hal00732512/document With this strategy a copy of optimizer is created, which is then asked for a point, and the point is told to the copy of optimizer with some fake objective (lie), the next point is asked from copy, it is also told to the copy with fake objective and so on. The type of lie defines different flavours of `cl_x` strategies. """ if n_points is None: return self._ask() supported_strategies = ["cl_min", "cl_mean", "cl_max"] if not (isinstance(n_points, int) and n_points > 0): raise ValueError( "n_points should be int > 0, got " + str(n_points) ) if strategy not in supported_strategies: raise ValueError( "Expected parallel_strategy to be one of " + str(supported_strategies) + ", " + "got %s" % strategy ) # Caching the result with n_points not None. If some new parameters # are provided to the ask, the cache_ is not used. if (n_points, strategy) in self.cache_: return self.cache_[(n_points, strategy)] # Copy of the optimizer is made in order to manage the # deletion of points with "lie" objective (the copy of # oiptimizer is simply discarded) opt = self.copy(random_state=self.rng.randint(0, np.iinfo(np.int32).max)) X = [] for i in range(n_points): x = opt.ask() X.append(x) ti_available = "ps" in self.acq_func and len(opt.yi) > 0 ti = [t for (_, t) in opt.yi] if ti_available else None if strategy == "cl_min": y_lie = np.min(opt.yi) if opt.yi else 0.0 # CLmin lie t_lie = np.min(ti) if ti is not None else log(sys.float_info.max) elif strategy == "cl_mean": y_lie = np.mean(opt.yi) if opt.yi else 0.0 # CLmean lie t_lie = np.mean(ti) if ti is not None else log(sys.float_info.max) else: y_lie = np.max(opt.yi) if opt.yi else 0.0 # CLmax lie t_lie = np.max(ti) if ti is not None else log(sys.float_info.max) # Lie to the optimizer. if "ps" in self.acq_func: # Use `_tell()` instead of `tell()` to prevent repeated # log transformations of the computation times. opt._tell(x, (y_lie, t_lie)) else: opt._tell(x, y_lie) self.cache_ = {(n_points, strategy): X} # cache_ the result return X
def copy(
self, random_state=None)
Create a shallow copy of an instance of the optimizer.
Parameters
random_state
[int, RandomState instance, or None (default)]: Set the random state of the copy.
def copy(self, random_state=None): """Create a shallow copy of an instance of the optimizer. Parameters  * `random_state` [int, RandomState instance, or None (default)]: Set the random state of the copy. """ optimizer = Optimizer( dimensions=self.space.dimensions, base_estimator=self.base_estimator_, n_initial_points=self.n_initial_points_, acq_func=self.acq_func, acq_optimizer=self.acq_optimizer, acq_func_kwargs=self.acq_func_kwargs, acq_optimizer_kwargs=self.acq_optimizer_kwargs, random_state=random_state, ) if hasattr(self, "gains_"): optimizer.gains_ = np.copy(self.gains_) if self.Xi: optimizer._tell(self.Xi, self.yi) return optimizer
def run(
self, func, n_iter=1)
Execute ask() + tell() n_iter
times
def run(self, func, n_iter=1): """Execute ask() + tell() `n_iter` times""" for _ in range(n_iter): x = self.ask() self.tell(x, func(x)) return create_result(self.Xi, self.yi, self.space, self.rng, models=self.models)
def tell(
self, x, y, fit=True)
Record an observation (or several) of the objective function.
Provide values of the objective function at points suggested by ask()
or other points. By default a new model will be fit to all
observations. The new model is used to suggest the next point at
which to evaluate the objective. This point can be retrieved by calling
ask()
.
To add observations without fitting a new model set fit
to False.
To add multiple observations in a batch pass a listoflists for x
and a list of scalars for y
.
Parameters

x
[list or listoflists]: Point at which objective was evaluated. 
y
[scalar or list]: Value of objective atx
. 
fit
[bool, default=True] Fit a model to observed evaluations of the objective. A model will only be fitted aftern_initial_points
points have been told to the optimizer irrespective of the value offit
.
def tell(self, x, y, fit=True): """Record an observation (or several) of the objective function. Provide values of the objective function at points suggested by `ask()` or other points. By default a new model will be fit to all observations. The new model is used to suggest the next point at which to evaluate the objective. This point can be retrieved by calling `ask()`. To add observations without fitting a new model set `fit` to False. To add multiple observations in a batch pass a listoflists for `x` and a list of scalars for `y`. Parameters  * `x` [list or listoflists]: Point at which objective was evaluated. * `y` [scalar or list]: Value of objective at `x`. * `fit` [bool, default=True] Fit a model to observed evaluations of the objective. A model will only be fitted after `n_initial_points` points have been told to the optimizer irrespective of the value of `fit`. """ check_x_in_space(x, self.space) self._check_y_is_valid(x, y) # take the logarithm of the computation times if "ps" in self.acq_func: if is_2Dlistlike(x): y = [[val, log(t)] for (val, t) in y] elif is_listlike(x): y = list(y) y[1] = log(y[1]) return self._tell(x, y, fit=fit)
Instance variables
var Xi
var acq_func
var acq_func_kwargs
var acq_optimizer
var acq_optimizer_kwargs
var cache_
var eta
var models
var n_initial_points_
var n_jobs
var n_points
var n_restarts_optimizer
var rng
var space
var yi
class Space
Search space.
class Space(object): """Search space.""" def __init__(self, dimensions): """Initialize a search space from given specifications. Parameters  * `dimensions` [list, shape=(n_dims,)]: List of search space dimensions. Each search dimension can be defined either as  a `(lower_bound, upper_bound)` tuple (for `Real` or `Integer` dimensions),  a `(lower_bound, upper_bound, "prior")` tuple (for `Real` dimensions),  as a list of categories (for `Categorical` dimensions), or  an instance of a `Dimension` object (`Real`, `Integer` or `Categorical`). NOTE: The upper and lower bounds are inclusive for `Integer` dimensions. """ self.dimensions = [check_dimension(dim) for dim in dimensions] def __eq__(self, other): return all([a == b for a, b in zip(self.dimensions, other.dimensions)]) def __repr__(self): if len(self.dimensions) > 31: dims = self.dimensions[:15] + [_Ellipsis()] + self.dimensions[15:] else: dims = self.dimensions return "Space([{}])".format(',\n '.join(map(str, dims))) def __iter__(self): return iter(self.dimensions) @property def is_real(self): """ Returns true if all dimensions are Real """ return all([isinstance(dim, Real) for dim in self.dimensions]) @classmethod def from_yaml(cls, yml_path, namespace=None): """Create Space from yaml configuration file Parameters  * `yml_path` [str]: Full path to yaml configuration file, example YaML below: Space:  Integer: low: 5 high: 5  Categorical: categories:  a  b  Real: low: 1.0 high: 5.0 prior: loguniform * `namespace` [str, default=None]: Namespace within configuration file to use, will use first namespace if not provided Returns  * `space` [Space]: Instantiated Space object """ with open(yml_path, 'rb') as f: config = yaml.load(f) dimension_classes = {'real': Real, 'integer': Integer, 'categorical': Categorical} # Extract space options for configuration file if isinstance(config, dict): if namespace is None: options = next(iter(config.values())) else: options = config[namespace] elif isinstance(config, list): options = config else: raise TypeError('YaML does not specify a list or dictionary') # Populate list with Dimension objects dimensions = [] for option in options: key = next(iter(option.keys())) # Make configuration case insensitive dimension_class = key.lower() values = {k.lower(): v for k, v in option[key].items()} if dimension_class in dimension_classes: # Instantiate Dimension subclass and add it to the list dimension = dimension_classes[dimension_class](**values) dimensions.append(dimension) space = cls(dimensions=dimensions) return space def rvs(self, n_samples=1, random_state=None): """Draw random samples. The samples are in the original space. They need to be transformed before being passed to a model or minimizer by `space.transform()`. Parameters  * `n_samples` [int, default=1]: Number of samples to be drawn from the space. * `random_state` [int, RandomState instance, or None (default)]: Set random state to something other than None for reproducible results. Returns  * `points`: [list of lists, shape=(n_points, n_dims)] Points sampled from the space. """ rng = check_random_state(random_state) # Draw columns = [] for dim in self.dimensions: if sp_version < (0, 16): columns.append(dim.rvs(n_samples=n_samples)) else: columns.append(dim.rvs(n_samples=n_samples, random_state=rng)) # Transpose rows = [] for i in range(n_samples): r = [] for j in range(self.n_dims): r.append(columns[j][i]) rows.append(r) return rows def transform(self, X): """Transform samples from the original space into a warped space. Note: this transformation is expected to be used to project samples into a suitable space for numerical optimization. Parameters  * `X` [list of lists, shape=(n_samples, n_dims)]: The samples to transform. Returns  * `Xt` [array of floats, shape=(n_samples, transformed_n_dims)] The transformed samples. """ # Pack by dimension columns = [] for dim in self.dimensions: columns.append([]) for i in range(len(X)): for j in range(self.n_dims): columns[j].append(X[i][j]) # Transform for j in range(self.n_dims): columns[j] = self.dimensions[j].transform(columns[j]) # Repack as an array Xt = np.hstack([np.asarray(c).reshape((len(X), 1)) for c in columns]) return Xt def inverse_transform(self, Xt): """Inverse transform samples from the warped space back to the original space. Parameters  * `Xt` [array of floats, shape=(n_samples, transformed_n_dims)]: The samples to inverse transform. Returns  * `X` [list of lists, shape=(n_samples, n_dims)] The original samples. """ # Inverse transform columns = [] start = 0 for j in range(self.n_dims): dim = self.dimensions[j] offset = dim.transformed_size if offset == 1: columns.append(dim.inverse_transform(Xt[:, start])) else: columns.append( dim.inverse_transform(Xt[:, start:start+offset])) start += offset # Transpose rows = [] for i in range(len(Xt)): r = [] for j in range(self.n_dims): r.append(columns[j][i]) rows.append(r) return rows @property def n_dims(self): """The dimensionality of the original space.""" return len(self.dimensions) @property def transformed_n_dims(self): """The dimensionality of the warped space.""" return sum([dim.transformed_size for dim in self.dimensions]) @property def bounds(self): """The dimension bounds, in the original space.""" b = [] for dim in self.dimensions: if dim.size == 1: b.append(dim.bounds) else: b.extend(dim.bounds) return b def __contains__(self, point): """Check that `point` is within the bounds of the space.""" for component, dim in zip(point, self.dimensions): if component not in dim: return False return True @property def transformed_bounds(self): """The dimension bounds, in the warped space.""" b = [] for dim in self.dimensions: if dim.transformed_size == 1: b.append(dim.transformed_bounds) else: b.extend(dim.transformed_bounds) return b @property def is_categorical(self): """Space contains exclusively categorical dimensions""" return all([isinstance(dim, Categorical) for dim in self.dimensions]) def distance(self, point_a, point_b): """Compute distance between two points in this space. Parameters  * `a` [array] First point. * `b` [array] Second point. """ distance = 0. for a, b, dim in zip(point_a, point_b, self.dimensions): distance += dim.distance(a, b) return distance
Ancestors (in MRO)
 Space
 builtins.object
Static methods
def __init__(
self, dimensions)
Initialize a search space from given specifications.
Parameters

dimensions
[list, shape=(n_dims,)]: List of search space dimensions. Each search dimension can be defined either as a
(lower_bound, upper_bound)
tuple (forReal
orInteger
dimensions),  a
(lower_bound, upper_bound, "prior")
tuple (forReal
dimensions),  as a list of categories (for
Categorical
dimensions), or  an instance of a
Dimension
object (Real
,Integer
orCategorical
).
NOTE: The upper and lower bounds are inclusive for
Integer
dimensions.  a
def __init__(self, dimensions): """Initialize a search space from given specifications. Parameters  * `dimensions` [list, shape=(n_dims,)]: List of search space dimensions. Each search dimension can be defined either as  a `(lower_bound, upper_bound)` tuple (for `Real` or `Integer` dimensions),  a `(lower_bound, upper_bound, "prior")` tuple (for `Real` dimensions),  as a list of categories (for `Categorical` dimensions), or  an instance of a `Dimension` object (`Real`, `Integer` or `Categorical`). NOTE: The upper and lower bounds are inclusive for `Integer` dimensions. """ self.dimensions = [check_dimension(dim) for dim in dimensions]
def distance(
self, point_a, point_b)
Compute distance between two points in this space.
Parameters

a
[array] First point. 
b
[array] Second point.
def distance(self, point_a, point_b): """Compute distance between two points in this space. Parameters  * `a` [array] First point. * `b` [array] Second point. """ distance = 0. for a, b, dim in zip(point_a, point_b, self.dimensions): distance += dim.distance(a, b) return distance
def inverse_transform(
self, Xt)
Inverse transform samples from the warped space back to the original space.
Parameters
Xt
[array of floats, shape=(n_samples, transformed_n_dims)]: The samples to inverse transform.
Returns
X
[list of lists, shape=(n_samples, n_dims)] The original samples.
def inverse_transform(self, Xt): """Inverse transform samples from the warped space back to the original space. Parameters  * `Xt` [array of floats, shape=(n_samples, transformed_n_dims)]: The samples to inverse transform. Returns  * `X` [list of lists, shape=(n_samples, n_dims)] The original samples. """ # Inverse transform columns = [] start = 0 for j in range(self.n_dims): dim = self.dimensions[j] offset = dim.transformed_size if offset == 1: columns.append(dim.inverse_transform(Xt[:, start])) else: columns.append( dim.inverse_transform(Xt[:, start:start+offset])) start += offset # Transpose rows = [] for i in range(len(Xt)): r = [] for j in range(self.n_dims): r.append(columns[j][i]) rows.append(r) return rows
def rvs(
self, n_samples=1, random_state=None)
Draw random samples.
The samples are in the original space. They need to be transformed
before being passed to a model or minimizer by space.transform()
.
Parameters

n_samples
[int, default=1]: Number of samples to be drawn from the space. 
random_state
[int, RandomState instance, or None (default)]: Set random state to something other than None for reproducible results.
Returns
points
: [list of lists, shape=(n_points, n_dims)] Points sampled from the space.
def rvs(self, n_samples=1, random_state=None): """Draw random samples. The samples are in the original space. They need to be transformed before being passed to a model or minimizer by `space.transform()`. Parameters  * `n_samples` [int, default=1]: Number of samples to be drawn from the space. * `random_state` [int, RandomState instance, or None (default)]: Set random state to something other than None for reproducible results. Returns  * `points`: [list of lists, shape=(n_points, n_dims)] Points sampled from the space. """ rng = check_random_state(random_state) # Draw columns = [] for dim in self.dimensions: if sp_version < (0, 16): columns.append(dim.rvs(n_samples=n_samples)) else: columns.append(dim.rvs(n_samples=n_samples, random_state=rng)) # Transpose rows = [] for i in range(n_samples): r = [] for j in range(self.n_dims): r.append(columns[j][i]) rows.append(r) return rows
def transform(
self, X)
Transform samples from the original space into a warped space.
Note: this transformation is expected to be used to project samples into a suitable space for numerical optimization.
Parameters
X
[list of lists, shape=(n_samples, n_dims)]: The samples to transform.
Returns
Xt
[array of floats, shape=(n_samples, transformed_n_dims)] The transformed samples.
def transform(self, X): """Transform samples from the original space into a warped space. Note: this transformation is expected to be used to project samples into a suitable space for numerical optimization. Parameters  * `X` [list of lists, shape=(n_samples, n_dims)]: The samples to transform. Returns  * `Xt` [array of floats, shape=(n_samples, transformed_n_dims)] The transformed samples. """ # Pack by dimension columns = [] for dim in self.dimensions: columns.append([]) for i in range(len(X)): for j in range(self.n_dims): columns[j].append(X[i][j]) # Transform for j in range(self.n_dims): columns[j] = self.dimensions[j].transform(columns[j]) # Repack as an array Xt = np.hstack([np.asarray(c).reshape((len(X), 1)) for c in columns]) return Xt
Instance variables
var bounds
The dimension bounds, in the original space.
var dimensions
var is_categorical
Space contains exclusively categorical dimensions
var is_real
Returns true if all dimensions are Real
var n_dims
The dimensionality of the original space.
var transformed_bounds
The dimension bounds, in the warped space.
var transformed_n_dims
The dimensionality of the warped space.
Methods
def from_yaml(
cls, yml_path, namespace=None)
Create Space from yaml configuration file
Parameters
yml_path
[str]: Full path to yaml configuration file, example YaML below: Space: Integer: low: 5 high: 5
 Categorical:
categories:
 a
 b
 Real: low: 1.0 high: 5.0 prior: loguniform
namespace
[str, default=None]: Namespace within configuration file to use, will use first namespace if not provided
Returns
space
[Space]: Instantiated Space object
@classmethod def from_yaml(cls, yml_path, namespace=None): """Create Space from yaml configuration file Parameters  * `yml_path` [str]: Full path to yaml configuration file, example YaML below: Space:  Integer: low: 5 high: 5  Categorical: categories:  a  b  Real: low: 1.0 high: 5.0 prior: loguniform * `namespace` [str, default=None]: Namespace within configuration file to use, will use first namespace if not provided Returns  * `space` [Space]: Instantiated Space object """ with open(yml_path, 'rb') as f: config = yaml.load(f) dimension_classes = {'real': Real, 'integer': Integer, 'categorical': Categorical} # Extract space options for configuration file if isinstance(config, dict): if namespace is None: options = next(iter(config.values())) else: options = config[namespace] elif isinstance(config, list): options = config else: raise TypeError('YaML does not specify a list or dictionary') # Populate list with Dimension objects dimensions = [] for option in options: key = next(iter(option.keys())) # Make configuration case insensitive dimension_class = key.lower() values = {k.lower(): v for k, v in option[key].items()} if dimension_class in dimension_classes: # Instantiate Dimension subclass and add it to the list dimension = dimension_classes[dimension_class](**values) dimensions.append(dimension) space = cls(dimensions=dimensions) return space