Fork me on GitHub Top

skopt module

Scikit-Optimize, or skopt, is a simple and efficient library to minimize (very) expensive and noisy black-box functions. It implements several methods for sequential model-based optimization. skopt is reusable in many contexts and accessible.

Build Status

Install

pip install numpy
pip install scikit-optimize

Getting started

Find the minimum of the noisy function f(x) over the range -2 < x < 2 with skopt:

import numpy as np
from skopt import gp_minimize

def f(x):
    return (np.sin(5 * x[0]) * (1 - np.tanh(x[0] ** 2)) *
            np.random.randn() * 0.1)

res = gp_minimize(f, [(-2.0, 2.0)])

For more read our introduction to bayesian optimization and the other examples.

Development

The library is still experimental and under heavy development.

The development version can be installed through:

git clone https://github.com/scikit-optimize/scikit-optimize.git
cd scikit-optimize
pip install -r requirements.txt
python setup.py develop

Run the tests by executing pytest in the top level directory.

"""
Scikit-Optimize, or `skopt`, is a simple and efficient library to
minimize (very) expensive and noisy black-box functions. It implements
several methods for sequential model-based optimization. `skopt` is reusable
in many contexts and accessible.

[![Build Status](https://travis-ci.org/scikit-optimize/scikit-optimize.svg?branch=master)](https://travis-ci.org/scikit-optimize/scikit-optimize)

## Install

```
pip install numpy
pip install scikit-optimize
```

## Getting started

Find the minimum of the noisy function `f(x)` over the range `-2 < x < 2`
with `skopt`:

```python
import numpy as np
from skopt import gp_minimize

def f(x):
    return (np.sin(5 * x[0]) * (1 - np.tanh(x[0] ** 2)) *
            np.random.randn() * 0.1)

res = gp_minimize(f, [(-2.0, 2.0)])
```

For more read our [introduction to bayesian optimization](https://scikit-optimize.github.io/notebooks/bayesian-optimization.html)
and the other [examples](https://github.com/scikit-optimize/scikit-optimize/tree/master/examples).


## Development

The library is still experimental and under heavy development.

The development version can be installed through:

    git clone https://github.com/scikit-optimize/scikit-optimize.git
    cd scikit-optimize
    pip install -r requirements.txt
    python setup.py develop

Run the tests by executing `pytest` in the top level directory.
"""

from . import acquisition
from . import benchmarks
from . import callbacks
from . import learning
from . import optimizer
from . import plots
from . import space
from .optimizer import dummy_minimize
from .optimizer import forest_minimize
from .optimizer import gbrt_minimize
from .optimizer import gp_minimize
from .optimizer import Optimizer
from .searchcv import BayesSearchCV
from .space import Space
from .utils import dump
from .utils import expected_minimum
from .utils import load

__version__ = "0.4rc1"


__all__ = (
    "acquisition",
    "benchmarks",
    "callbacks",
    "learning",
    "optimizer",
    "plots",
    "space",
    "gp_minimize",
    "dummy_minimize",
    "forest_minimize",
    "gbrt_minimize",
    "Optimizer",
    "dump",
    "load",
    "expected_minimum",
    "BayesSearchCV",
    "Space"
)

Functions

def dummy_minimize(

func, dimensions, n_calls=100, x0=None, y0=None, random_state=None, verbose=False, callback=None)

Random search by uniform sampling within the given bounds.

Parameters

  • func [callable]: Function to minimize. Should take a array of parameters and return the function values.

  • dimensions [list, shape=(n_dims,)]: List of search space dimensions. Each search dimension can be defined either as

    • a (upper_bound, lower_bound) tuple (for Real or Integer dimensions),
    • a (upper_bound, lower_bound, prior) tuple (for Real dimensions),
    • as a list of categories (for Categorical dimensions), or
    • an instance of a Dimension object (Real, Integer or Categorical).
  • n_calls [int, default=100]: Number of calls to func to find the minimum.

  • x0 [list, list of lists or None]: Initial input points.

    • If it is a list of lists, use it as a list of input points.
    • If it is a list, use it as a single initial input point.
    • If it is None, no initial input points are used.
  • y0 [list, scalar or None]: Evaluation of initial input points.

    • If it is a list, then it corresponds to evaluations of the function at each element of x0 : the i-th element of y0 corresponds to the function evaluated at the i-th element of x0.
    • If it is a scalar, then it corresponds to the evaluation of the function at x0.
    • If it is None and x0 is provided, then the function is evaluated at each element of x0.
  • random_state [int, RandomState instance, or None (default)]: Set random state to something other than None for reproducible results.

  • verbose [boolean, default=False]: Control the verbosity. It is advised to set the verbosity to True for long optimization runs.

  • callback [callable, list of callables, optional] If callable then callback(res) is called after each call to func. If list of callables, then each callable in the list is called.

Returns

  • res [OptimizeResult, scipy object]: The optimization result returned as a OptimizeResult object. Important attributes are:

    • x [list]: location of the minimum.
    • fun [float]: function value at the minimum.
    • x_iters [list of lists]: location of function evaluation for each iteration.
    • func_vals [array]: function value for each iteration.
    • space [Space]: the optimisation space.
    • specs [dict]: the call specifications.
    • rng [RandomState instance]: State of the random state at the end of minimization.

    For more details related to the OptimizeResult object, refer http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.OptimizeResult.html

def dummy_minimize(func, dimensions, n_calls=100, x0=None, y0=None,
                   random_state=None, verbose=False, callback=None):
    """Random search by uniform sampling within the given bounds.

    Parameters
    ----------
    * `func` [callable]:
        Function to minimize. Should take a array of parameters and
        return the function values.

    * `dimensions` [list, shape=(n_dims,)]:
        List of search space dimensions.
        Each search dimension can be defined either as

        - a `(upper_bound, lower_bound)` tuple (for `Real` or `Integer`
          dimensions),
        - a `(upper_bound, lower_bound, prior)` tuple (for `Real`
          dimensions),
        - as a list of categories (for `Categorical` dimensions), or
        - an instance of a `Dimension` object (`Real`, `Integer` or
          `Categorical`).

    * `n_calls` [int, default=100]:
        Number of calls to `func` to find the minimum.

    * `x0` [list, list of lists or `None`]:
        Initial input points.

        - If it is a list of lists, use it as a list of input points.
        - If it is a list, use it as a single initial input point.
        - If it is `None`, no initial input points are used.

    * `y0` [list, scalar or `None`]:
        Evaluation of initial input points.

        - If it is a list, then it corresponds to evaluations of the function
          at each element of `x0` : the i-th element of `y0` corresponds
          to the function evaluated at the i-th element of `x0`.
        - If it is a scalar, then it corresponds to the evaluation of the
          function at `x0`.
        - If it is None and `x0` is provided, then the function is evaluated
          at each element of `x0`.

    * `random_state` [int, RandomState instance, or None (default)]:
        Set random state to something other than None for reproducible
        results.

    * `verbose` [boolean, default=False]:
        Control the verbosity. It is advised to set the verbosity to True
        for long optimization runs.

    * `callback` [callable, list of callables, optional]
        If callable then `callback(res)` is called after each call to `func`.
        If list of callables, then each callable in the list is called.

    Returns
    -------
    * `res` [`OptimizeResult`, scipy object]:
        The optimization result returned as a OptimizeResult object.
        Important attributes are:

        - `x` [list]: location of the minimum.
        - `fun` [float]: function value at the minimum.
        - `x_iters` [list of lists]: location of function evaluation for each
           iteration.
        - `func_vals` [array]: function value for each iteration.
        - `space` [Space]: the optimisation space.
        - `specs` [dict]: the call specifications.
        - `rng` [RandomState instance]: State of the random state
           at the end of minimization.

        For more details related to the OptimizeResult object, refer
        http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.OptimizeResult.html
    """
    # all our calls want random suggestions, except if we need to evaluate
    # some initial points
    if x0 is not None and y0 is None:
        n_random_calls = n_calls - len(x0)
    else:
        n_random_calls = n_calls

    return base_minimize(func, dimensions, base_estimator="dummy",
                         # explicitly set optimizer to sampling as "dummy"
                         # minimizer does not provide gradients.
                         acq_optimizer="sampling",
                         n_calls=n_calls, n_random_starts=n_random_calls,
                         x0=x0, y0=y0, random_state=random_state,
                         verbose=verbose,
                         callback=callback)

def dump(

res, filename, store_objective=True, **kwargs)

Store an skopt optimization result into a file.

Parameters

  • res [OptimizeResult, scipy object]: Optimization result object to be stored.

  • filename [string or pathlib.Path]: The path of the file in which it is to be stored. The compression method corresponding to one of the supported filename extensions ('.z', '.gz', '.bz2', '.xz' or '.lzma') will be used automatically.

  • store_objective [boolean, default=True]: Whether the objective function should be stored. Set store_objective to False if your objective function (.specs['args']['func']) is unserializable (i.e. if an exception is raised when trying to serialize the optimization result).

    Notice that if store_objective is set to False, a deep copy of the optimization result is created, potentially leading to performance problems if res is very large. If the objective function is not critical, one can delete it before calling skopt.dump() and thus avoid deep copying of res.

  • **kwargs [other keyword arguments]: All other keyword arguments will be passed to joblib.dump.

def dump(res, filename, store_objective=True, **kwargs):
    """
    Store an skopt optimization result into a file.

    Parameters
    ----------
    * `res` [`OptimizeResult`, scipy object]:
        Optimization result object to be stored.

    * `filename` [string or `pathlib.Path`]:
        The path of the file in which it is to be stored. The compression
        method corresponding to one of the supported filename extensions ('.z',
        '.gz', '.bz2', '.xz' or '.lzma') will be used automatically.

    * `store_objective` [boolean, default=True]:
        Whether the objective function should be stored. Set `store_objective`
        to `False` if your objective function (`.specs['args']['func']`) is
        unserializable (i.e. if an exception is raised when trying to serialize
        the optimization result).

        Notice that if `store_objective` is set to `False`, a deep copy of the
        optimization result is created, potentially leading to performance
        problems if `res` is very large. If the objective function is not
        critical, one can delete it before calling `skopt.dump()` and thus
        avoid deep copying of `res`.

    * `**kwargs` [other keyword arguments]:
        All other keyword arguments will be passed to `joblib.dump`.
    """
    if store_objective:
        dump_(res, filename, **kwargs)

    elif 'func' in res.specs['args']:
        # If the user does not want to store the objective and it is indeed
        # present in the provided object, then create a deep copy of it and
        # remove the objective function before dumping it with joblib.dump.
        res_without_func = deepcopy(res)
        del res_without_func.specs['args']['func']
        dump_(res_without_func, filename, **kwargs)

    else:
        # If the user does not want to store the objective and it is already
        # missing in the provided object, dump it without copying.
        dump_(res, filename, **kwargs)

def expected_minimum(

res, n_random_starts=20, random_state=None)

Compute the minimum over the predictions of the last surrogate model.

Note that the returned minimum may not necessarily be an accurate prediction of the minimum of the true objective function.

Parameters

  • res [OptimizeResult, scipy object]: The optimization result returned by a skopt minimizer.

  • n_random_starts [int, default=20]: The number of random starts for the minimization of the surrogate model.

  • random_state [int, RandomState instance, or None (default)]: Set random state to something other than None for reproducible results.

Returns

  • x [list]: location of the minimum.

  • fun [float]: the surrogate function value at the minimum.

def expected_minimum(res, n_random_starts=20, random_state=None):
    """
    Compute the minimum over the predictions of the last surrogate model.

    Note that the returned minimum may not necessarily be an accurate
    prediction of the minimum of the true objective function.

    Parameters
    ----------
    * `res`  [`OptimizeResult`, scipy object]:
        The optimization result returned by a `skopt` minimizer.

    * `n_random_starts` [int, default=20]:
        The number of random starts for the minimization of the surrogate
        model.

    * `random_state` [int, RandomState instance, or None (default)]:
        Set random state to something other than None for reproducible
        results.

    Returns
    -------
    * `x` [list]: location of the minimum.

    * `fun` [float]: the surrogate function value at the minimum.
    """
    def func(x):
        reg = res.models[-1]
        return reg.predict(x.reshape(1, -1))[0]

    xs = [res.x]
    if n_random_starts > 0:
        xs.extend(res.space.rvs(n_random_starts, random_state=random_state))

    best_x = None
    best_fun = np.inf

    for x0 in xs:
        r = sp_minimize(func, x0=x0, bounds=res.space.bounds)

        if r.fun < best_fun:
            best_x = r.x
            best_fun = r.fun

    return [v for v in best_x], best_fun

def forest_minimize(

func, dimensions, base_estimator='ET', n_calls=100, n_random_starts=10, acq_func='EI', x0=None, y0=None, random_state=None, verbose=False, callback=None, n_points=10000, xi=0.01, kappa=1.96, n_jobs=1)

Sequential optimisation using decision trees.

A tree based regression model is used to model the expensive to evaluate function func. The model is improved by sequentially evaluating the expensive function at the next best point. Thereby finding the minimum of func with as few evaluations as possible.

The total number of evaluations, n_calls, are performed like the following. If x0 is provided but not y0, then the elements of x0 are first evaluated, followed by n_random_starts evaluations. Finally, n_calls - len(x0) - n_random_starts evaluations are made guided by the surrogate model. If x0 and y0 are both provided then n_random_starts evaluations are first made then n_calls - n_random_starts subsequent evaluations are made guided by the surrogate model.

Parameters

  • func [callable]: Function to minimize. Should take a array of parameters and return the function values.

  • dimensions [list, shape=(n_dims,)]: List of search space dimensions. Each search dimension can be defined either as

    • a (upper_bound, lower_bound) tuple (for Real or Integer dimensions),
    • a (upper_bound, lower_bound, prior) tuple (for Real dimensions),
    • as a list of categories (for Categorical dimensions), or
    • an instance of a Dimension object (Real, Integer or Categorical).

    NOTE: The upper and lower bounds are inclusive for Integer dimensions.

  • base_estimator [string or Regressor, default="ET"]: The regressor to use as surrogate model. Can be either

    • "RF" for random forest regressor
    • "ET" for extra trees regressor
    • instance of regressor with support for return_std in its predict method

    The predefined models are initilized with good defaults. If you want to adjust the model parameters pass your own instance of a regressor which returns the mean and standard deviation when making predictions.

  • n_calls [int, default=100]: Number of calls to func.

  • n_random_starts [int, default=10]: Number of evaluations of func with random points before approximating it with base_estimator.

  • acq_func [string, default="LCB"]: Function to minimize over the forest posterior. Can be either

    • "LCB" for lower confidence bound.
    • "EI" for negative expected improvement.
    • "PI" for negative probability of improvement.
    • `"EIps" for negated expected improvement per second to take into account the function compute time. Then, the objective function is assumed to return two values, the first being the objective value and the second being the time taken in seconds.
    • "PIps" for negated probability of improvement per second. The return type of the objective function is assumed to be similar to that of "EIps"
  • x0 [list, list of lists or None]: Initial input points.

    • If it is a list of lists, use it as a list of input points.
    • If it is a list, use it as a single initial input point.
    • If it is None, no initial input points are used.
  • y0 [list, scalar or None]: Evaluation of initial input points.

    • If it is a list, then it corresponds to evaluations of the function at each element of x0 : the i-th element of y0 corresponds to the function evaluated at the i-th element of x0.
    • If it is a scalar, then it corresponds to the evaluation of the function at x0.
    • If it is None and x0 is provided, then the function is evaluated at each element of x0.
  • random_state [int, RandomState instance, or None (default)]: Set random state to something other than None for reproducible results.

  • verbose [boolean, default=False]: Control the verbosity. It is advised to set the verbosity to True for long optimization runs.

  • callback [callable, optional] If provided, then callback(res) is called after call to func.

  • n_points [int, default=10000]: Number of points to sample when minimizing the acquisition function.

  • xi [float, default=0.01]: Controls how much improvement one wants over the previous best values. Used when the acquisition is either "EI" or "PI".

  • kappa [float, default=1.96]: Controls how much of the variance in the predicted values should be taken into account. If set to be very high, then we are favouring exploration over exploitation and vice versa. Used when the acquisition is "LCB".

  • n_jobs [int, default=1]: The number of jobs to run in parallel for fit and predict. If -1, then the number of jobs is set to the number of cores.

Returns

  • res [OptimizeResult, scipy object]: The optimization result returned as a OptimizeResult object. Important attributes are:

    • x [list]: location of the minimum.
    • fun [float]: function value at the minimum.
    • models: surrogate models used for each iteration.
    • x_iters [list of lists]: location of function evaluation for each iteration.
    • func_vals [array]: function value for each iteration.
    • space [Space]: the optimization space.
    • specs [dict]`: the call specifications.

    For more details related to the OptimizeResult object, refer http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.OptimizeResult.html

def forest_minimize(func, dimensions, base_estimator="ET", n_calls=100,
                    n_random_starts=10, acq_func="EI",
                    x0=None, y0=None, random_state=None, verbose=False,
                    callback=None, n_points=10000, xi=0.01, kappa=1.96,
                    n_jobs=1):
    """Sequential optimisation using decision trees.

    A tree based regression model is used to model the expensive to evaluate
    function `func`. The model is improved by sequentially evaluating
    the expensive function at the next best point. Thereby finding the
    minimum of `func` with as few evaluations as possible.

    The total number of evaluations, `n_calls`, are performed like the
    following. If `x0` is provided but not `y0`, then the elements of `x0`
    are first evaluated, followed by `n_random_starts` evaluations.
    Finally, `n_calls - len(x0) - n_random_starts` evaluations are
    made guided by the surrogate model. If `x0` and `y0` are both
    provided then `n_random_starts` evaluations are first made then
    `n_calls - n_random_starts` subsequent evaluations are made
    guided by the surrogate model.

    Parameters
    ----------
    * `func` [callable]:
        Function to minimize. Should take a array of parameters and
        return the function values.

    * `dimensions` [list, shape=(n_dims,)]:
        List of search space dimensions.
        Each search dimension can be defined either as

        - a `(upper_bound, lower_bound)` tuple (for `Real` or `Integer`
          dimensions),
        - a `(upper_bound, lower_bound, prior)` tuple (for `Real`
          dimensions),
        - as a list of categories (for `Categorical` dimensions), or
        - an instance of a `Dimension` object (`Real`, `Integer` or
          `Categorical`).

         NOTE: The upper and lower bounds are inclusive for `Integer`
         dimensions.

    * `base_estimator` [string or `Regressor`, default=`"ET"`]:
        The regressor to use as surrogate model. Can be either

        - `"RF"` for random forest regressor
        - `"ET"` for extra trees regressor
        - instance of regressor with support for `return_std` in its predict
          method

        The predefined models are initilized with good defaults. If you
        want to adjust the model parameters pass your own instance of
        a regressor which returns the mean and standard deviation when
        making predictions.

    * `n_calls` [int, default=100]:
        Number of calls to `func`.

    * `n_random_starts` [int, default=10]:
        Number of evaluations of `func` with random points before
        approximating it with `base_estimator`.

    * `acq_func` [string, default=`"LCB"`]:
        Function to minimize over the forest posterior. Can be either

        - `"LCB"` for lower confidence bound.
        - `"EI"` for negative expected improvement.
        - `"PI"` for negative probability of improvement.
        - `"EIps" for negated expected improvement per second to take into
          account the function compute time. Then, the objective function is
          assumed to return two values, the first being the objective value and
          the second being the time taken in seconds.
        - `"PIps"` for negated probability of improvement per second. The
          return type of the objective function is assumed to be similar to
          that of `"EIps"`

    * `x0` [list, list of lists or `None`]:
        Initial input points.

        - If it is a list of lists, use it as a list of input points.
        - If it is a list, use it as a single initial input point.
        - If it is `None`, no initial input points are used.

    * `y0` [list, scalar or `None`]:
        Evaluation of initial input points.

        - If it is a list, then it corresponds to evaluations of the function
          at each element of `x0` : the i-th element of `y0` corresponds
          to the function evaluated at the i-th element of `x0`.
        - If it is a scalar, then it corresponds to the evaluation of the
          function at `x0`.
        - If it is None and `x0` is provided, then the function is evaluated
          at each element of `x0`.

    * `random_state` [int, RandomState instance, or None (default)]:
        Set random state to something other than None for reproducible
        results.

    * `verbose` [boolean, default=False]:
        Control the verbosity. It is advised to set the verbosity to True
        for long optimization runs.

    * `callback` [callable, optional]
        If provided, then `callback(res)` is called after call to func.

    * `n_points` [int, default=10000]:
        Number of points to sample when minimizing the acquisition function.

    * `xi` [float, default=0.01]:
        Controls how much improvement one wants over the previous best
        values. Used when the acquisition is either `"EI"` or `"PI"`.

    * `kappa` [float, default=1.96]:
        Controls how much of the variance in the predicted values should be
        taken into account. If set to be very high, then we are favouring
        exploration over exploitation and vice versa.
        Used when the acquisition is `"LCB"`.

    * `n_jobs` [int, default=1]:
        The number of jobs to run in parallel for `fit` and `predict`.
        If -1, then the number of jobs is set to the number of cores.

    Returns
    -------
    * `res` [`OptimizeResult`, scipy object]:
        The optimization result returned as a OptimizeResult object.
        Important attributes are:

        - `x` [list]: location of the minimum.
        - `fun` [float]: function value at the minimum.
        - `models`: surrogate models used for each iteration.
        - `x_iters` [list of lists]: location of function evaluation for each
           iteration.
        - `func_vals` [array]: function value for each iteration.
        - `space` [Space]: the optimization space.
        - `specs` [dict]`: the call specifications.

        For more details related to the OptimizeResult object, refer
        http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.OptimizeResult.html
    """
    rng = check_random_state(random_state)

    if isinstance(base_estimator, str):
        base_estimator = cook_estimator(base_estimator, n_jobs=n_jobs,
                                        random_state=rng)

    return base_minimize(func, dimensions, base_estimator,
                         n_calls=n_calls, n_points=n_points,
                         n_random_starts=n_random_starts,
                         x0=x0, y0=y0, random_state=random_state,
                         acq_func=acq_func,
                         xi=xi, kappa=kappa, verbose=verbose,
                         callback=callback, acq_optimizer="sampling")

def gbrt_minimize(

func, dimensions, base_estimator=None, n_calls=100, n_random_starts=10, acq_func='EI', acq_optimizer='auto', x0=None, y0=None, random_state=None, verbose=False, callback=None, n_points=10000, xi=0.01, kappa=1.96, n_jobs=1)

Sequential optimization using gradient boosted trees.

Gradient boosted regression trees are used to model the (very) expensive to evaluate function func. The model is improved by sequentially evaluating the expensive function at the next best point. Thereby finding the minimum of func with as few evaluations as possible.

The total number of evaluations, n_calls, are performed like the following. If x0 is provided but not y0, then the elements of x0 are first evaluated, followed by n_random_starts evaluations. Finally, n_calls - len(x0) - n_random_starts evaluations are made guided by the surrogate model. If x0 and y0 are both provided then n_random_starts evaluations are first made then n_calls - n_random_starts subsequent evaluations are made guided by the surrogate model.

Parameters

  • func [callable]: Function to minimize. Should take a array of parameters and return the function values.

  • dimensions [list, shape=(n_dims,)]: List of search space dimensions. Each search dimension can be defined either as

    • a (upper_bound, lower_bound) tuple (for Real or Integer dimensions),
    • a (upper_bound, lower_bound, "prior") tuple (for Real dimensions),
    • as a list of categories (for Categorical dimensions), or
    • an instance of a Dimension object (Real, Integer or Categorical).
  • base_estimator [GradientBoostingQuantileRegressor]: The regressor to use as surrogate model

  • n_calls [int, default=100]: Number of calls to func.

  • n_random_starts [int, default=10]: Number of evaluations of func with random points before approximating it with base_estimator.

  • acq_func [string, default="LCB"]: Function to minimize over the forest posterior. Can be either

    • "LCB" for lower confidence bound.
    • "EI" for negative expected improvement.
    • "PI" for negative probability of improvement.
    • "EIps" for negated expected improvement per second to take into account the function compute time. Then, the objective function is assumed to return two values, the first being the objective value and the second being the time taken.
    • "PIps" for negated probability of improvement per second.
  • x0 [list, list of lists or None]: Initial input points.

    • If it is a list of lists, use it as a list of input points.
    • If it is a list, use it as a single initial input point.
    • If it is None, no initial input points are used.
  • y0 [list, scalar or None]: Evaluation of initial input points.

    • If it is a list, then it corresponds to evaluations of the function at each element of x0 : the i-th element of y0 corresponds to the function evaluated at the i-th element of x0.
    • If it is a scalar, then it corresponds to the evaluation of the function at x0.
    • If it is None and x0 is provided, then the function is evaluated at each element of x0.
  • random_state [int, RandomState instance, or None (default)]: Set random state to something other than None for reproducible results.

  • verbose [boolean, default=False]: Control the verbosity. It is advised to set the verbosity to True for long optimization runs.

  • callback [callable, optional] If provided, then callback(res) is called after call to func.

  • n_points [int, default=10000]: Number of points to sample when minimizing the acquisition function.

  • xi [float, default=0.01]: Controls how much improvement one wants over the previous best values. Used when the acquisition is either "EI" or "PI".

  • kappa [float, default=1.96]: Controls how much of the variance in the predicted values should be taken into account. If set to be very high, then we are favouring exploration over exploitation and vice versa. Used when the acquisition is "LCB".

  • n_jobs [int, default=1]: The number of jobs to run in parallel for fit and predict. If -1, then the number of jobs is set to the number of cores.

Returns

  • res [OptimizeResult, scipy object]: The optimization result returned as a OptimizeResult object. Important attributes are:

    • x [list]: location of the minimum.
    • fun [float]: function value at the minimum.
    • models: surrogate models used for each iteration.
    • x_iters [list of lists]: location of function evaluation for each iteration.
    • func_vals [array]: function value for each iteration.
    • space [Space]: the optimization space.
    • specs [dict]`: the call specifications.
    • rng [RandomState instance]: State of the random state at the end of minimization.

    For more details related to the OptimizeResult object, refer http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.OptimizeResult.html

def gbrt_minimize(func, dimensions, base_estimator=None,
                  n_calls=100, n_random_starts=10,
                  acq_func="EI", acq_optimizer="auto",
                  x0=None, y0=None, random_state=None, verbose=False,
                  callback=None, n_points=10000, xi=0.01, kappa=1.96,
                  n_jobs=1):
    """Sequential optimization using gradient boosted trees.

    Gradient boosted regression trees are used to model the (very)
    expensive to evaluate function `func`. The model is improved
    by sequentially evaluating the expensive function at the next
    best point. Thereby finding the minimum of `func` with as
    few evaluations as possible.

    The total number of evaluations, `n_calls`, are performed like the
    following. If `x0` is provided but not `y0`, then the elements of `x0`
    are first evaluated, followed by `n_random_starts` evaluations.
    Finally, `n_calls - len(x0) - n_random_starts` evaluations are
    made guided by the surrogate model. If `x0` and `y0` are both
    provided then `n_random_starts` evaluations are first made then
    `n_calls - n_random_starts` subsequent evaluations are made
    guided by the surrogate model.

    Parameters
    ----------
    * `func` [callable]:
        Function to minimize. Should take a array of parameters and
        return the function values.

    * `dimensions` [list, shape=(n_dims,)]:
        List of search space dimensions.
        Each search dimension can be defined either as

        - a `(upper_bound, lower_bound)` tuple (for `Real` or `Integer`
          dimensions),
        - a `(upper_bound, lower_bound, "prior")` tuple (for `Real`
          dimensions),
        - as a list of categories (for `Categorical` dimensions), or
        - an instance of a `Dimension` object (`Real`, `Integer` or
          `Categorical`).

    * `base_estimator` [`GradientBoostingQuantileRegressor`]:
        The regressor to use as surrogate model

    * `n_calls` [int, default=100]:
        Number of calls to `func`.

    * `n_random_starts` [int, default=10]:
        Number of evaluations of `func` with random points before
        approximating it with `base_estimator`.

    * `acq_func` [string, default=`"LCB"`]:
        Function to minimize over the forest posterior. Can be either

        - `"LCB"` for lower confidence bound.
        - `"EI"` for negative expected improvement.
        - `"PI"` for negative probability of improvement.
        - ``"EIps"`` for negated expected improvement per second to take into
          account the function compute time. Then, the objective function is
          assumed to return two values, the first being the objective value and
          the second being the time taken.
        - `"PIps"` for negated probability of improvement per second.

    * `x0` [list, list of lists or `None`]:
        Initial input points.

        - If it is a list of lists, use it as a list of input points.
        - If it is a list, use it as a single initial input point.
        - If it is `None`, no initial input points are used.

    * `y0` [list, scalar or `None`]:
        Evaluation of initial input points.

        - If it is a list, then it corresponds to evaluations of the function
          at each element of `x0` : the i-th element of `y0` corresponds
          to the function evaluated at the i-th element of `x0`.
        - If it is a scalar, then it corresponds to the evaluation of the
          function at `x0`.
        - If it is None and `x0` is provided, then the function is evaluated
          at each element of `x0`.

    * `random_state` [int, RandomState instance, or None (default)]:
        Set random state to something other than None for reproducible
        results.

    * `verbose` [boolean, default=False]:
        Control the verbosity. It is advised to set the verbosity to True
        for long optimization runs.

    * `callback` [callable, optional]
        If provided, then `callback(res)` is called after call to func.

    * `n_points` [int, default=10000]:
        Number of points to sample when minimizing the acquisition function.

    * `xi` [float, default=0.01]:
        Controls how much improvement one wants over the previous best
        values. Used when the acquisition is either `"EI"` or `"PI"`.

    * `kappa` [float, default=1.96]:
        Controls how much of the variance in the predicted values should be
        taken into account. If set to be very high, then we are favouring
        exploration over exploitation and vice versa.
        Used when the acquisition is `"LCB"`.

    * `n_jobs` [int, default=1]:
        The number of jobs to run in parallel for `fit` and `predict`.
        If -1, then the number of jobs is set to the number of cores.

    Returns
    -------
    * `res` [`OptimizeResult`, scipy object]:
        The optimization result returned as a OptimizeResult object.
        Important attributes are:

        - `x` [list]: location of the minimum.
        - `fun` [float]: function value at the minimum.
        - `models`: surrogate models used for each iteration.
        - `x_iters` [list of lists]: location of function evaluation for each
           iteration.
        - `func_vals` [array]: function value for each iteration.
        - `space` [Space]: the optimization space.
        - `specs` [dict]`: the call specifications.
        - `rng` [RandomState instance]: State of the random state
           at the end of minimization.

        For more details related to the OptimizeResult object, refer
        http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.OptimizeResult.html
    """
    # Check params
    rng = check_random_state(random_state)

    if base_estimator is None:
        base_estimator = cook_estimator("GBRT", random_state=rng,
                                        n_jobs=n_jobs)
    return base_minimize(func, dimensions, base_estimator,
                         n_calls=n_calls, n_points=n_points,
                         n_random_starts=n_random_starts,
                         x0=x0, y0=y0, random_state=random_state, xi=xi,
                         kappa=kappa, acq_func=acq_func, verbose=verbose,
                         callback=callback, acq_optimizer="sampling")

def gp_minimize(

func, dimensions, base_estimator=None, n_calls=100, n_random_starts=10, acq_func='gp_hedge', acq_optimizer='auto', x0=None, y0=None, random_state=None, verbose=False, callback=None, n_points=10000, n_restarts_optimizer=5, xi=0.01, kappa=1.96, noise='gaussian', n_jobs=1)

Bayesian optimization using Gaussian Processes.

If every function evaluation is expensive, for instance when the parameters are the hyperparameters of a neural network and the function evaluation is the mean cross-validation score across ten folds, optimizing the hyperparameters by standard optimization routines would take for ever!

The idea is to approximate the function using a Gaussian process. In other words the function values are assumed to follow a multivariate gaussian. The covariance of the function values are given by a GP kernel between the parameters. Then a smart choice to choose the next parameter to evaluate can be made by the acquisition function over the Gaussian prior which is much quicker to evaluate.

The total number of evaluations, n_calls, are performed like the following. If x0 is provided but not y0, then the elements of x0 are first evaluated, followed by n_random_starts evaluations. Finally, n_calls - len(x0) - n_random_starts evaluations are made guided by the surrogate model. If x0 and y0 are both provided then n_random_starts evaluations are first made then n_calls - n_random_starts subsequent evaluations are made guided by the surrogate model.

Parameters

  • func [callable]: Function to minimize. Should take a array of parameters and return the function values.

  • dimensions [list, shape=(n_dims,)]: List of search space dimensions. Each search dimension can be defined either as

    • a (upper_bound, lower_bound) tuple (for Real or Integer dimensions),
    • a (upper_bound, lower_bound, "prior") tuple (for Real dimensions),
    • as a list of categories (for Categorical dimensions), or
    • an instance of a Dimension object (Real, Integer or Categorical).

    NOTE: The upper and lower bounds are inclusive for Integer dimensions.

  • base_estimator [a Gaussian process estimator]: The Gaussian process estimator to use for optimization. By default, a Matern kernel is used with the following hyperparameters tuned.

    • All the length scales of the Matern kernel.
    • The covariance amplitude that each element is multiplied with.
    • Noise that is added to the matern kernel. The noise is assumed to be iid gaussian.
  • n_calls [int, default=100]: Number of calls to func.

  • n_random_starts [int, default=10]: Number of evaluations of func with random points before approximating it with base_estimator.

  • acq_func [string, default="EI"]: Function to minimize over the gaussian prior. Can be either

    • "LCB" for lower confidence bound.
    • "EI" for negative expected improvement.
    • "PI" for negative probability of improvement.
    • "gp_hedge" Probabilistically choose one of the above three acquisition functions at every iteration. The weightage given to these gains can be set by \eta through acq_func_kwargs.
      • The gains g_i are initialized to zero.
      • At every iteration,
        • Each acquisition function is optimised independently to propose an candidate point X_i.
        • Out of all these candidate points, the next point X_best is chosen by softmax(\eta g_i)
        • After fitting the surrogate model with (X_best, y_best), the gains are updated such that g_i -= \mu(X_i)
    • "EIps" for negated expected improvement per second to take into account the function compute time. Then, the objective function is assumed to return two values, the first being the objective value and the second being the time taken in seconds.
    • "PIps" for negated probability of improvement per second. The return type of the objective function is assumed to be similar to that of `"EIps
  • acq_optimizer [string, "sampling" or "lbfgs", default="lbfgs"]: Method to minimize the acquistion function. The fit model is updated with the optimal value obtained by optimizing acq_func with acq_optimizer.

    The acq_func is computed at n_points sampled randomly.

    • If set to "auto", then acq_optimizer is configured on the basis of the space searched over. If the space is Categorical then this is set to be "sampling"`.
    • If set to "sampling", then the point among these n_points where the acq_func is minimum is the next candidate minimum.
    • If set to "lbfgs", then
      • The n_restarts_optimizer no. of points which the acquisition function is least are taken as start points.
      • "lbfgs" is run for 20 iterations with these points as initial points to find local minima.
      • The optimal of these local minima is used to update the prior.
  • x0 [list, list of lists or None]: Initial input points.

    • If it is a list of lists, use it as a list of input points.
    • If it is a list, use it as a single initial input point.
    • If it is None, no initial input points are used.
  • y0 [list, scalar or None] Evaluation of initial input points.

    • If it is a list, then it corresponds to evaluations of the function at each element of x0 : the i-th element of y0 corresponds to the function evaluated at the i-th element of x0.
    • If it is a scalar, then it corresponds to the evaluation of the function at x0.
    • If it is None and x0 is provided, then the function is evaluated at each element of x0.
  • random_state [int, RandomState instance, or None (default)]: Set random state to something other than None for reproducible results.

  • verbose [boolean, default=False]: Control the verbosity. It is advised to set the verbosity to True for long optimization runs.

  • callback [callable, list of callables, optional] If callable then callback(res) is called after each call to func. If list of callables, then each callable in the list is called.

  • n_points [int, default=10000]: Number of points to sample to determine the next "best" point. Useless if acq_optimizer is set to "lbfgs".

  • n_restarts_optimizer [int, default=5]: The number of restarts of the optimizer when acq_optimizer is "lbfgs".

  • kappa [float, default=1.96]: Controls how much of the variance in the predicted values should be taken into account. If set to be very high, then we are favouring exploration over exploitation and vice versa. Used when the acquisition is "LCB".

  • xi [float, default=0.01]: Controls how much improvement one wants over the previous best values. Used when the acquisition is either "EI" or "PI".

  • noise [float, default="gaussian"]:

    • Use noise="gaussian" if the objective returns noisy observations. The noise of each observation is assumed to be iid with mean zero and a fixed variance.
    • If the variance is known before-hand, this can be set directly to the variance of the noise.
    • Set this to a value close to zero (1e-10) if the function is noise-free. Setting to zero might cause stability issues.
  • n_jobs [int, default=1] Number of cores to run in parallel while running the lbfgs optimizations over the acquisition function. Valid only when acq_optimizer is set to "lbfgs." Defaults to 1 core. If n_jobs=-1, then number of jobs is set to number of cores.

Returns

  • res [OptimizeResult, scipy object]: The optimization result returned as a OptimizeResult object. Important attributes are:

    • x [list]: location of the minimum.
    • fun [float]: function value at the minimum.
    • models: surrogate models used for each iteration.
    • x_iters [list of lists]: location of function evaluation for each iteration.
    • func_vals [array]: function value for each iteration.
    • space [Space]: the optimization space.
    • specs [dict]`: the call specifications.
    • rng [RandomState instance]: State of the random state at the end of minimization.

    For more details related to the OptimizeResult object, refer http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.OptimizeResult.html

def gp_minimize(func, dimensions, base_estimator=None,
                n_calls=100, n_random_starts=10,
                acq_func="gp_hedge", acq_optimizer="auto", x0=None, y0=None,
                random_state=None, verbose=False, callback=None,
                n_points=10000, n_restarts_optimizer=5, xi=0.01, kappa=1.96,
                noise="gaussian", n_jobs=1):
    """Bayesian optimization using Gaussian Processes.

    If every function evaluation is expensive, for instance
    when the parameters are the hyperparameters of a neural network
    and the function evaluation is the mean cross-validation score across
    ten folds, optimizing the hyperparameters by standard optimization
    routines would take for ever!

    The idea is to approximate the function using a Gaussian process.
    In other words the function values are assumed to follow a multivariate
    gaussian. The covariance of the function values are given by a
    GP kernel between the parameters. Then a smart choice to choose the
    next parameter to evaluate can be made by the acquisition function
    over the Gaussian prior which is much quicker to evaluate.

    The total number of evaluations, `n_calls`, are performed like the
    following. If `x0` is provided but not `y0`, then the elements of `x0`
    are first evaluated, followed by `n_random_starts` evaluations.
    Finally, `n_calls - len(x0) - n_random_starts` evaluations are
    made guided by the surrogate model. If `x0` and `y0` are both
    provided then `n_random_starts` evaluations are first made then
    `n_calls - n_random_starts` subsequent evaluations are made
    guided by the surrogate model.

    Parameters
    ----------
    * `func` [callable]:
        Function to minimize. Should take a array of parameters and
        return the function values.

    * `dimensions` [list, shape=(n_dims,)]:
        List of search space dimensions.
        Each search dimension can be defined either as

        - a `(upper_bound, lower_bound)` tuple (for `Real` or `Integer`
          dimensions),
        - a `(upper_bound, lower_bound, "prior")` tuple (for `Real`
          dimensions),
        - as a list of categories (for `Categorical` dimensions), or
        - an instance of a `Dimension` object (`Real`, `Integer` or
          `Categorical`).

         NOTE: The upper and lower bounds are inclusive for `Integer`
         dimensions.

    * `base_estimator` [a Gaussian process estimator]:
        The Gaussian process estimator to use for optimization.
        By default, a Matern kernel is used with the following
        hyperparameters tuned.
        - All the length scales of the Matern kernel.
        - The covariance amplitude that each element is multiplied with.
        - Noise that is added to the matern kernel. The noise is assumed
          to be iid gaussian.

    * `n_calls` [int, default=100]:
        Number of calls to `func`.

    * `n_random_starts` [int, default=10]:
        Number of evaluations of `func` with random points before
        approximating it with `base_estimator`.

    * `acq_func` [string, default=`"EI"`]:
        Function to minimize over the gaussian prior. Can be either

        - `"LCB"` for lower confidence bound.
        - `"EI"` for negative expected improvement.
        - `"PI"` for negative probability of improvement.
        - `"gp_hedge"` Probabilistically choose one of the above three
          acquisition functions at every iteration. The weightage
          given to these gains can be set by `\eta` through `acq_func_kwargs`.
            - The gains `g_i` are initialized to zero.
            - At every iteration,
                - Each acquisition function is optimised independently to
                  propose an candidate point `X_i`.
                - Out of all these candidate points, the next point `X_best` is
                  chosen by `softmax(\eta g_i)`
                - After fitting the surrogate model with `(X_best, y_best)`,
                  the gains are updated such that `g_i -= \mu(X_i)`
        - `"EIps"` for negated expected improvement per second to take into
          account the function compute time. Then, the objective function is
          assumed to return two values, the first being the objective value and
          the second being the time taken in seconds.
        - `"PIps"` for negated probability of improvement per second. The
          return type of the objective function is assumed to be similar to
          that of `"EIps

    * `acq_optimizer` [string, `"sampling"` or `"lbfgs"`, default=`"lbfgs"`]:
        Method to minimize the acquistion function. The fit model
        is updated with the optimal value obtained by optimizing `acq_func`
        with `acq_optimizer`.

        The `acq_func` is computed at `n_points` sampled randomly.

        - If set to `"auto"`, then `acq_optimizer` is configured on the
          basis of the space searched over.
          If the space is Categorical then this is set to be "sampling"`.
        - If set to `"sampling"`, then the point among these `n_points`
          where the `acq_func` is minimum is the next candidate minimum.
        - If set to `"lbfgs"`, then
              - The `n_restarts_optimizer` no. of points which the acquisition
                function is least are taken as start points.
              - `"lbfgs"` is run for 20 iterations with these points as initial
                points to find local minima.
              - The optimal of these local minima is used to update the prior.

    * `x0` [list, list of lists or `None`]:
        Initial input points.

        - If it is a list of lists, use it as a list of input points.
        - If it is a list, use it as a single initial input point.
        - If it is `None`, no initial input points are used.

    * `y0` [list, scalar or `None`]
        Evaluation of initial input points.

        - If it is a list, then it corresponds to evaluations of the function
          at each element of `x0` : the i-th element of `y0` corresponds
          to the function evaluated at the i-th element of `x0`.
        - If it is a scalar, then it corresponds to the evaluation of the
          function at `x0`.
        - If it is None and `x0` is provided, then the function is evaluated
          at each element of `x0`.

    * `random_state` [int, RandomState instance, or None (default)]:
        Set random state to something other than None for reproducible
        results.

    * `verbose` [boolean, default=False]:
        Control the verbosity. It is advised to set the verbosity to True
        for long optimization runs.

    * `callback` [callable, list of callables, optional]
        If callable then `callback(res)` is called after each call to `func`.
        If list of callables, then each callable in the list is called.

    * `n_points` [int, default=10000]:
        Number of points to sample to determine the next "best" point.
        Useless if acq_optimizer is set to `"lbfgs"`.

    * `n_restarts_optimizer` [int, default=5]:
        The number of restarts of the optimizer when `acq_optimizer`
        is `"lbfgs"`.

    * `kappa` [float, default=1.96]:
        Controls how much of the variance in the predicted values should be
        taken into account. If set to be very high, then we are favouring
        exploration over exploitation and vice versa.
        Used when the acquisition is `"LCB"`.

    * `xi` [float, default=0.01]:
        Controls how much improvement one wants over the previous best
        values. Used when the acquisition is either `"EI"` or `"PI"`.

    * `noise` [float, default="gaussian"]:
        - Use noise="gaussian" if the objective returns noisy observations.
          The noise of each observation is assumed to be iid with
          mean zero and a fixed variance.
        - If the variance is known before-hand, this can be set directly
          to the variance of the noise.
        - Set this to a value close to zero (1e-10) if the function is
          noise-free. Setting to zero might cause stability issues.

    * `n_jobs` [int, default=1]
        Number of cores to run in parallel while running the lbfgs
        optimizations over the acquisition function. Valid only
        when `acq_optimizer` is set to "lbfgs."
        Defaults to 1 core. If `n_jobs=-1`, then number of jobs is set
        to number of cores.

    Returns
    -------
    * `res` [`OptimizeResult`, scipy object]:
        The optimization result returned as a OptimizeResult object.
        Important attributes are:

        - `x` [list]: location of the minimum.
        - `fun` [float]: function value at the minimum.
        - `models`: surrogate models used for each iteration.
        - `x_iters` [list of lists]: location of function evaluation for each
           iteration.
        - `func_vals` [array]: function value for each iteration.
        - `space` [Space]: the optimization space.
        - `specs` [dict]`: the call specifications.
        - `rng` [RandomState instance]: State of the random state
           at the end of minimization.

        For more details related to the OptimizeResult object, refer
        http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.OptimizeResult.html
    """
    # Check params
    rng = check_random_state(random_state)
    space = normalize_dimensions(dimensions)
    base_estimator = cook_estimator("GP", space=space, random_state=rng,
                                    noise=noise)

    return base_minimize(
        func, space, base_estimator=base_estimator,
        acq_func=acq_func,
        xi=xi, kappa=kappa, acq_optimizer=acq_optimizer, n_calls=n_calls,
        n_points=n_points, n_random_starts=n_random_starts,
        n_restarts_optimizer=n_restarts_optimizer,
        x0=x0, y0=y0, random_state=random_state, verbose=verbose,
        callback=callback, n_jobs=n_jobs)

def load(

filename, **kwargs)

Reconstruct a skopt optimization result from a file persisted with skopt.dump.

Notice that the loaded optimization result can be missing the objective function (.specs['args']['func']) if dump was called with store_objective=False.

Parameters

  • filename [string or pathlib.Path]: The path of the file from which to load the optimization result.

  • **kwargs [other keyword arguments]: All other keyword arguments will be passed to joblib.load.

Returns

  • res [OptimizeResult, scipy object]: Reconstructed OptimizeResult instance.
def load(filename, **kwargs):
    """
    Reconstruct a skopt optimization result from a file
    persisted with skopt.dump.

    Notice that the loaded optimization result can be missing
    the objective function (`.specs['args']['func']`) if `skopt.dump`
    was called with `store_objective=False`.

    Parameters
    ----------
    * `filename` [string or `pathlib.Path`]:
        The path of the file from which to load the optimization result.

    * `**kwargs` [other keyword arguments]:
        All other keyword arguments will be passed to `joblib.load`.

    Returns
    -------
    * `res` [`OptimizeResult`, scipy object]:
        Reconstructed OptimizeResult instance.
    """
    return load_(filename, **kwargs)

Classes

class BayesSearchCV

Bayesian optimization over hyper parameters.

BayesSearchCV implements a "fit" and a "score" method. It also implements "predict", "predict_proba", "decision_function", "transform" and "inverse_transform" if they are implemented in the estimator used.

The parameters of the estimator used to apply these methods are optimized by cross-validated search over parameter settings.

In contrast to GridSearchCV, not all parameter values are tried out, but rather a fixed number of parameter settings is sampled from the specified distributions. The number of parameter settings that are tried is given by n_iter.

Parameters are presented as a list of skopt.space.Dimension objects.

Parameters

estimator : estimator object. A object of that type is instantiated for each search point. This object is assumed to implement the scikit-learn estimator api. Either estimator needs to provide a score function, or scoring must be passed.

search_spaces : dict, list of dict or list of tuple containing (dict, int), or None. One of 4 following cases: 1. dictionary, where keys are parameter names (strings) and values are skopt.space.Dimension instances (Real, Integer or Categorical) or any other valid value that defines skopt dimension (see skopt.Optimizer docs). Represents search space over parameters of the provided estimator. 2. list of dictionaries: a list of dictionaries, where every dictionary fits the description given in case 1 above. If a list of dictionary objects is given, then the search is performed sequentially for every parameter space with maximum number of evaluations set to self.n_iter. 3. list of (dict, int > 0): an extension of case 2 above, where first element of every tuple is a dictionary representing some search subspace, similarly as in case 2, and second element is a number of iterations that will be spent optimizing over this subspace. 4. None, in which case it is assumed that a user will provide search space via the add_spaces function.

n_iter : int, default=128 Number of parameter settings that are sampled. n_iter trades off runtime vs quality of the solution.

surrogate : string or skopt surrogate, default='auto' Surrogate to use for optimization of score of estimator. By default skopt.learning.GaussianProcessRegressor() is used.

scoring : string, callable or None, default=None A string (see model evaluation documentation) or a scorer callable object / function with signature scorer(estimator, X, y). If None, the score method of the estimator is used.

fit_params : dict, optional Parameters to pass to the fit method.

n_jobs : int, default=1 Number of jobs to run in parallel.

pre_dispatch : int, or string, optional Controls the number of jobs that get dispatched during parallel execution. Reducing this number can be useful to avoid an explosion of memory consumption when more jobs get dispatched than CPUs can process. This parameter can be:

    - None, in which case all the jobs are immediately
      created and spawned. Use this for lightweight and
      fast-running jobs, to avoid delays due to on-demand
      spawning of the jobs

    - An int, giving the exact number of total jobs that are
      spawned

    - A string, giving an expression as a function of n_jobs,
      as in '2*n_jobs'

iid : boolean, default=True If True, the data is assumed to be identically distributed across the folds, and the loss minimized is the total loss per sample, and not the mean loss across the folds.

cv : int, cross-validation generator or an iterable, optional Determines the cross-validation splitting strategy. Possible inputs for cv are: - None, to use the default 3-fold cross validation, - integer, to specify the number of folds in a (Stratified)KFold, - An object to be used as a cross-validation generator. - An iterable yielding train, test splits.

For integer/None inputs, if the estimator is a classifier and ``y`` is
either binary or multiclass, :class:`StratifiedKFold` is used. In all
other cases, :class:`KFold` is used.

Refer :ref:`User Guide <cross_validation>` for the various
cross-validation strategies that can be used here.

refit : boolean, default=True Refit the best estimator with the entire dataset. If "False", it is impossible to make predictions using this RandomizedSearchCV instance after fitting.

verbose : integer Controls the verbosity: the higher, the more messages.

random_state : int or RandomState Pseudo random number generator state used for random uniform sampling from lists of possible values instead of scipy.stats distributions.

error_score : 'raise' (default) or numeric Value to assign to the score if an error occurs in estimator fitting. If set to 'raise', the error is raised. If a numeric value is given, FitFailedWarning is raised. This parameter does not affect the refit step, which will always raise the error.

return_train_score : boolean, default=True If 'False', the cv_results_ attribute will not include training scores.

Example

from skopt import BayesSearchCV

parameter ranges are specified by one of below

from skopt.space import Real, Categorical, Integer

from sklearn.datasets import load_iris from sklearn.svm import SVC from sklearn.model_selection import train_test_split

X, y = load_iris(True) X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.75, random_state=0)

log-uniform: understand as search over p = exp(x) by varying x

opt = BayesSearchCV( SVC(), { 'C': Real(1e-6, 1e+6, prior='log-uniform'), 'gamma': Real(1e-6, 1e+1, prior='log-uniform'), 'degree': Integer(1,8), 'kernel': Categorical(['linear', 'poly', 'rbf']), }, n_iter=32 )

executes bayesian optimization

opt.fit(X_train, y_train)

model can be saved, used for predictions or scoring

print(opt.score(X_test, y_test))

Attributes

cv_results_ : dict of numpy (masked) ndarrays A dict with keys as column headers and values as columns, that can be imported into a pandas DataFrame.

For instance the below given table

+--------------+-------------+-------------------+---+---------------+
| param_kernel | param_gamma | split0_test_score |...|rank_test_score|
+==============+=============+===================+===+===============+
|    'rbf'     |     0.1     |        0.8        |...|       2       |
+--------------+-------------+-------------------+---+---------------+
|    'rbf'     |     0.2     |        0.9        |...|       1       |
+--------------+-------------+-------------------+---+---------------+
|    'rbf'     |     0.3     |        0.7        |...|       1       |
+--------------+-------------+-------------------+---+---------------+

will be represented by a ``cv_results_`` dict of::

    {
    'param_kernel' : masked_array(data = ['rbf', 'rbf', 'rbf'],
                                  mask = False),
    'param_gamma'  : masked_array(data = [0.1 0.2 0.3], mask = False),
    'split0_test_score'  : [0.8, 0.9, 0.7],
    'split1_test_score'  : [0.82, 0.5, 0.7],
    'mean_test_score'    : [0.81, 0.7, 0.7],
    'std_test_score'     : [0.02, 0.2, 0.],
    'rank_test_score'    : [3, 1, 1],
    'split0_train_score' : [0.8, 0.9, 0.7],
    'split1_train_score' : [0.82, 0.5, 0.7],
    'mean_train_score'   : [0.81, 0.7, 0.7],
    'std_train_score'    : [0.03, 0.03, 0.04],
    'mean_fit_time'      : [0.73, 0.63, 0.43, 0.49],
    'std_fit_time'       : [0.01, 0.02, 0.01, 0.01],
    'mean_score_time'    : [0.007, 0.06, 0.04, 0.04],
    'std_score_time'     : [0.001, 0.002, 0.003, 0.005],
    'params' : [{'kernel' : 'rbf', 'gamma' : 0.1}, ...],
    }

NOTE that the key ``'params'`` is used to store a list of parameter
settings dict for all the parameter candidates.

The ``mean_fit_time``, ``std_fit_time``, ``mean_score_time`` and
``std_score_time`` are all in seconds.

best_estimator_ : estimator Estimator that was chosen by the search, i.e. estimator which gave highest score (or smallest loss if specified) on the left out data. Not available if refit=False.

best_score_ : float Score of best_estimator on the left out data.

best_params_ : dict Parameter setting that gave the best results on the hold out data.

best_index_ : int The index (of the cv_results_ arrays) which corresponds to the best candidate parameter setting.

The dict at ``search.cv_results_['params'][search.best_index_]`` gives
the parameter setting for the best model, that gives the highest
mean score (``search.best_score_``).

scorer_ : function Scorer function used on the held out data to choose the best parameters for the model.

n_splits_ : int The number of cross-validation splits (folds/iterations).

Notes

The parameters selected are those that maximize the score of the held-out data, according to the scoring parameter.

If n_jobs was set to a value higher than one, the data is copied for each parameter setting(and not n_jobs times). This is done for efficiency reasons if individual jobs take very little time, but may raise errors if the dataset is large and not enough memory is available. A workaround in this case is to set pre_dispatch. Then, the memory is copied only pre_dispatch many times. A reasonable value for pre_dispatch is 2 * n_jobs.

See Also

:class:GridSearchCV: Does exhaustive search over a grid of parameters.

class BayesSearchCV(sklearn.model_selection._search.BaseSearchCV):
    """Bayesian optimization over hyper parameters.

    BayesSearchCV implements a "fit" and a "score" method.
    It also implements "predict", "predict_proba", "decision_function",
    "transform" and "inverse_transform" if they are implemented in the
    estimator used.

    The parameters of the estimator used to apply these methods are optimized
    by cross-validated search over parameter settings.

    In contrast to GridSearchCV, not all parameter values are tried out, but
    rather a fixed number of parameter settings is sampled from the specified
    distributions. The number of parameter settings that are tried is
    given by n_iter.

    Parameters are presented as a list of skopt.space.Dimension objects.

    Parameters
    ----------
    estimator : estimator object.
        A object of that type is instantiated for each search point.
        This object is assumed to implement the scikit-learn estimator api.
        Either estimator needs to provide a ``score`` function,
        or ``scoring`` must be passed.

    search_spaces : dict, list of dict or list of tuple containing
        (dict, int), or None.
        One of 4 following cases:
        1. dictionary, where keys are parameter names (strings)
        and values are skopt.space.Dimension instances (Real, Integer
        or Categorical) or any other valid value that defines skopt
        dimension (see skopt.Optimizer docs). Represents search space
        over parameters of the provided estimator.
        2. list of dictionaries: a list of dictionaries, where every
        dictionary fits the description given in case 1 above.
        If a list of dictionary objects is given, then the search is
        performed sequentially for every parameter space with maximum
        number of evaluations set to self.n_iter.
        3. list of (dict, int > 0): an extension of case 2 above,
        where first element of every tuple is a dictionary representing
        some search subspace, similarly as in case 2, and second element
        is a number of iterations that will be spent optimizing over
        this subspace.
        4. None, in which case it is assumed that a user will provide
        search space via the `add_spaces` function.

    n_iter : int, default=128
        Number of parameter settings that are sampled. n_iter trades
        off runtime vs quality of the solution.

    surrogate : string or skopt surrogate, default='auto'
        Surrogate to use for optimization of score of estimator.
        By default skopt.learning.GaussianProcessRegressor() is used.

    scoring : string, callable or None, default=None
        A string (see model evaluation documentation) or
        a scorer callable object / function with signature
        ``scorer(estimator, X, y)``.
        If ``None``, the ``score`` method of the estimator is used.

    fit_params : dict, optional
        Parameters to pass to the fit method.

    n_jobs : int, default=1
        Number of jobs to run in parallel.

    pre_dispatch : int, or string, optional
        Controls the number of jobs that get dispatched during parallel
        execution. Reducing this number can be useful to avoid an
        explosion of memory consumption when more jobs get dispatched
        than CPUs can process. This parameter can be:

            - None, in which case all the jobs are immediately
              created and spawned. Use this for lightweight and
              fast-running jobs, to avoid delays due to on-demand
              spawning of the jobs

            - An int, giving the exact number of total jobs that are
              spawned

            - A string, giving an expression as a function of n_jobs,
              as in '2*n_jobs'

    iid : boolean, default=True
        If True, the data is assumed to be identically distributed across
        the folds, and the loss minimized is the total loss per sample,
        and not the mean loss across the folds.

    cv : int, cross-validation generator or an iterable, optional
        Determines the cross-validation splitting strategy.
        Possible inputs for cv are:
          - None, to use the default 3-fold cross validation,
          - integer, to specify the number of folds in a `(Stratified)KFold`,
          - An object to be used as a cross-validation generator.
          - An iterable yielding train, test splits.

        For integer/None inputs, if the estimator is a classifier and ``y`` is
        either binary or multiclass, :class:`StratifiedKFold` is used. In all
        other cases, :class:`KFold` is used.

        Refer :ref:`User Guide <cross_validation>` for the various
        cross-validation strategies that can be used here.

    refit : boolean, default=True
        Refit the best estimator with the entire dataset.
        If "False", it is impossible to make predictions using
        this RandomizedSearchCV instance after fitting.

    verbose : integer
        Controls the verbosity: the higher, the more messages.

    random_state : int or RandomState
        Pseudo random number generator state used for random uniform sampling
        from lists of possible values instead of scipy.stats distributions.

    error_score : 'raise' (default) or numeric
        Value to assign to the score if an error occurs in estimator fitting.
        If set to 'raise', the error is raised. If a numeric value is given,
        FitFailedWarning is raised. This parameter does not affect the refit
        step, which will always raise the error.

    return_train_score : boolean, default=True
        If ``'False'``, the ``cv_results_`` attribute will not include training
        scores.

    Example
    -------

    from skopt import BayesSearchCV
    # parameter ranges are specified by one of below
    from skopt.space import Real, Categorical, Integer

    from sklearn.datasets import load_iris
    from sklearn.svm import SVC
    from sklearn.model_selection import train_test_split

    X, y = load_iris(True)
    X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.75, random_state=0)

    # log-uniform: understand as search over p = exp(x) by varying x
    opt = BayesSearchCV(
        SVC(),
        {
            'C': Real(1e-6, 1e+6, prior='log-uniform'),
            'gamma': Real(1e-6, 1e+1, prior='log-uniform'),
            'degree': Integer(1,8),
            'kernel': Categorical(['linear', 'poly', 'rbf']),
        },
        n_iter=32
    )

    # executes bayesian optimization
    opt.fit(X_train, y_train)

    # model can be saved, used for predictions or scoring
    print(opt.score(X_test, y_test))

    Attributes
    ----------
    cv_results_ : dict of numpy (masked) ndarrays
        A dict with keys as column headers and values as columns, that can be
        imported into a pandas ``DataFrame``.

        For instance the below given table

        +--------------+-------------+-------------------+---+---------------+
        | param_kernel | param_gamma | split0_test_score |...|rank_test_score|
        +==============+=============+===================+===+===============+
        |    'rbf'     |     0.1     |        0.8        |...|       2       |
        +--------------+-------------+-------------------+---+---------------+
        |    'rbf'     |     0.2     |        0.9        |...|       1       |
        +--------------+-------------+-------------------+---+---------------+
        |    'rbf'     |     0.3     |        0.7        |...|       1       |
        +--------------+-------------+-------------------+---+---------------+

        will be represented by a ``cv_results_`` dict of::

            {
            'param_kernel' : masked_array(data = ['rbf', 'rbf', 'rbf'],
                                          mask = False),
            'param_gamma'  : masked_array(data = [0.1 0.2 0.3], mask = False),
            'split0_test_score'  : [0.8, 0.9, 0.7],
            'split1_test_score'  : [0.82, 0.5, 0.7],
            'mean_test_score'    : [0.81, 0.7, 0.7],
            'std_test_score'     : [0.02, 0.2, 0.],
            'rank_test_score'    : [3, 1, 1],
            'split0_train_score' : [0.8, 0.9, 0.7],
            'split1_train_score' : [0.82, 0.5, 0.7],
            'mean_train_score'   : [0.81, 0.7, 0.7],
            'std_train_score'    : [0.03, 0.03, 0.04],
            'mean_fit_time'      : [0.73, 0.63, 0.43, 0.49],
            'std_fit_time'       : [0.01, 0.02, 0.01, 0.01],
            'mean_score_time'    : [0.007, 0.06, 0.04, 0.04],
            'std_score_time'     : [0.001, 0.002, 0.003, 0.005],
            'params' : [{'kernel' : 'rbf', 'gamma' : 0.1}, ...],
            }

        NOTE that the key ``'params'`` is used to store a list of parameter
        settings dict for all the parameter candidates.

        The ``mean_fit_time``, ``std_fit_time``, ``mean_score_time`` and
        ``std_score_time`` are all in seconds.

    best_estimator_ : estimator
        Estimator that was chosen by the search, i.e. estimator
        which gave highest score (or smallest loss if specified)
        on the left out data. Not available if refit=False.

    best_score_ : float
        Score of best_estimator on the left out data.

    best_params_ : dict
        Parameter setting that gave the best results on the hold out data.

    best_index_ : int
        The index (of the ``cv_results_`` arrays) which corresponds to the best
        candidate parameter setting.

        The dict at ``search.cv_results_['params'][search.best_index_]`` gives
        the parameter setting for the best model, that gives the highest
        mean score (``search.best_score_``).

    scorer_ : function
        Scorer function used on the held out data to choose the best
        parameters for the model.

    n_splits_ : int
        The number of cross-validation splits (folds/iterations).

    Notes
    -----
    The parameters selected are those that maximize the score of the held-out
    data, according to the scoring parameter.

    If `n_jobs` was set to a value higher than one, the data is copied for each
    parameter setting(and not `n_jobs` times). This is done for efficiency
    reasons if individual jobs take very little time, but may raise errors if
    the dataset is large and not enough memory is available.  A workaround in
    this case is to set `pre_dispatch`. Then, the memory is copied only
    `pre_dispatch` many times. A reasonable value for `pre_dispatch` is `2 *
    n_jobs`.

    See Also
    --------
    :class:`GridSearchCV`:
        Does exhaustive search over a grid of parameters.

    """

    def __init__(self, estimator, search_spaces=None, optimizer_kwargs=None,
                 n_iter=50, scoring=None, fit_params=None, n_jobs=1,
                 iid=True, refit=True, cv=None, verbose=0,
                 pre_dispatch='2*n_jobs', random_state=None,
                 error_score='raise', return_train_score=True):

        # set of space name: space dict. Stored as a dict, in order
        # to make it easy to add or remove search spaces, in case
        # user wants to use `step` function directly.
        self.search_spaces_ = {}

        # can be None if user intends to provide spaces later manually
        if search_spaces is not None:
            # account for the case when search space is a dict
            if isinstance(search_spaces, dict):
                search_spaces = [search_spaces]
            self.add_spaces(list(range(len(search_spaces))), search_spaces)

        self.n_iter = n_iter
        self.random_state = random_state

        if optimizer_kwargs is None:
            self.optimizer_kwargs = {}
        else:
            self.optimizer_kwargs = optimizer_kwargs

        # this dict is used in order to keep track of skopt Optimizer
        # instances for different search spaces. `str(space)` is used as key
        # as `space` is a dict, which is unhashable. however, str(space)
        # is fixed and unique if space is not changed.
        self.optimizer_ = {}
        self.cv_results_ = defaultdict(list)

        self.best_index_ = None
        self.multimetric_ = False

        super(BayesSearchCV, self).__init__(
             estimator=estimator, scoring=scoring, fit_params=fit_params,
             n_jobs=n_jobs, iid=iid, refit=refit, cv=cv, verbose=verbose,
             pre_dispatch=pre_dispatch, error_score=error_score,
             return_train_score=return_train_score)

    def _check_search_space(self, search_space):
        """Checks whether the search space argument is correct"""

        # check if space is a single dict, convert to list if so
        if isinstance(search_space, dict):
            search_space = [search_space]

        # check if the structure of the space is proper
        if isinstance(search_space, list):
            # convert to just a list of dicts
            dicts_only = []

            # 1. check the case when a tuple of space, n_iter is provided
            for elem in search_space:
                if isinstance(elem, tuple):
                    if len(elem) != 2:
                        raise ValueError(
                            "All tuples in list of search spaces should have"
                            "length 2, and contain (dict, int), got %s" % elem
                        )
                    subspace, n_iter = elem

                    if (not isinstance(n_iter, int)) or n_iter < 0:
                        raise ValueError(
                            "Number of iterations in search space should be"
                            "positive integer, got %s in tuple %s " %
                            (n_iter, elem)
                        )

                    # save subspaces here for further checking
                    dicts_only.append(subspace)
                elif isinstance(elem, dict):
                    dicts_only.append(elem)
                else:
                    raise TypeError(
                        "A search space should be provided as a dict or"
                        "tuple (dict, int), got %s" % elem)

            # 2. check all the dicts for correctness of contents
            for subspace in dicts_only:
                for k, v in subspace.items():
                    check_dimension(v)
        else:
            raise TypeError(
                "Search space should be provided as a dict or list of dict,"
                "got %s" % search_space)

    def add_spaces(self, names, search_spaces):
        """
        Add a parameter search space over which to search for parameter
        values. Naming of search subspaces is necessary in order to specify
        for the `step` function over which subspace to perform search step.

        names: str or list of str
            Define names for the parameter search subspaces.
            if search_spaces is a single dict, then names should be str
            representing name of the single search subspace.
            if search_spaces is a list of dicts, defines names for every
            search subspace in the list.

        search_spaces : dict, list of dict or list of tuple (dict, int)
            Define search subspaces over which to search for parameters
            of the model.
            One of 3 following cases:
            1. dictionary, where keys are parameter names (strings)
            and values are skopt.space.Dimension instances (Real, Integer
            or Categorical) or any other valid value that defines skopt
            dimension (see skopt.Optimizer docs). Represents search space
            over parameters of the provided estimator.
            2. list of dictionaries: a list of dictionaries, where every
            dictionary fits the description given in case 1 above.
            If a list of dictionary objects is given, then the search is
            performed sequentially for every parameter space with maximum
            number of evaluations set to self.n_iter.
            3. list of (dict, int > 0): an extension of case 2 above,
            where first element of every tuple is a dictionary representing
            some search subspace, similarly as in case 2, and second element
            is a number of iterations that will be spent optimizing over
            this subspace.
        """

        self._check_search_space(search_spaces)

        if not isinstance(search_spaces, list):
            search_spaces = [search_spaces]
        if not isinstance(names, list):
            names = [names]

        # first check whether space already exits ...
        for space, name in zip(search_spaces, names):
            if name in self.search_spaces_:
                raise ValueError("Search space %s already exists!" % name)

        for space, name in zip(search_spaces, names):
            self.search_spaces_[name] = space

    # copied for compatibility with 0.19 sklearn from 0.18 BaseSearchCV
    @property
    def best_score_(self):
        check_is_fitted(self, 'cv_results_')
        return self.cv_results_['mean_test_score'][self.best_index_]

    # copied for compatibility with 0.19 sklearn from 0.18 BaseSearchCV
    @property
    def best_params_(self):
        check_is_fitted(self, 'cv_results_')
        return self.cv_results_['params'][self.best_index_]

    # copied for compatibility with 0.19 sklearn from 0.18 BaseSearchCV
    def _fit(self, X, y, groups, parameter_iterable):
        """
        Actual fitting,  performing the search over parameters.
        Taken from https://github.com/scikit-learn/scikit-learn/blob/0.18.X
                    .../sklearn/model_selection/_search.py
        """

        estimator = self.estimator
        cv = sklearn.model_selection._validation.check_cv(self.cv, y, classifier=
            is_classifier(estimator))
        self.scorer_ = check_scoring(
            self.estimator, scoring=self.scoring)

        X, y, groups = indexable(X, y, groups)
        n_splits = cv.get_n_splits(X, y, groups)
        if self.verbose > 0 and isinstance(parameter_iterable, Sized):
            n_candidates = len(parameter_iterable)
            print("Fitting {0} folds for each of {1} candidates, totalling"
                  " {2} fits".format(n_splits, n_candidates,
                                     n_candidates * n_splits))

        base_estimator = clone(self.estimator)
        pre_dispatch = self.pre_dispatch

        cv_iter = list(cv.split(X, y, groups))
        out = Parallel(
            n_jobs=self.n_jobs, verbose=self.verbose,
            pre_dispatch=pre_dispatch
        )(delayed(sklearn.model_selection._validation._fit_and_score)(
                clone(base_estimator),
                X, y, self.scorer_,
                train, test, self.verbose, parameters,
                fit_params=self.fit_params,
                return_train_score=self.return_train_score,
                return_n_test_samples=True,
                return_times=True, return_parameters=True,
                error_score=self.error_score
            )
            for parameters in parameter_iterable
            for train, test in cv_iter
        )

        # if one choose to see train score, "out" will contain train score info
        if self.return_train_score:
            (train_scores, test_scores, test_sample_counts,
             fit_time, score_time, parameters) = zip(*out)
        else:
            (test_scores, test_sample_counts,
             fit_time, score_time, parameters) = zip(*out)

        candidate_params = parameters[::n_splits]
        n_candidates = len(candidate_params)

        results = dict()

        def _store(key_name, array, weights=None, splits=False, rank=False):
            """A small helper to store the scores/times to the cv_results_"""
            array = np.array(array, dtype=np.float64).reshape(n_candidates,
                                                              n_splits)
            if splits:
                for split_i in range(n_splits):
                    results["split%d_%s"
                            % (split_i, key_name)] = array[:, split_i]

            array_means = np.average(array, axis=1, weights=weights)
            results['mean_%s' % key_name] = array_means
            # Weighted std is not directly available in numpy
            array_stds = np.sqrt(np.average((array -
                                             array_means[:, np.newaxis]) ** 2,
                                            axis=1, weights=weights))
            results['std_%s' % key_name] = array_stds

            if rank:
                results["rank_%s" % key_name] = np.asarray(
                    rankdata(-array_means, method='min'), dtype=np.int32)

        # Computed the (weighted) mean and std for test scores alone
        # NOTE test_sample counts (weights) remain the same for all candidates
        test_sample_counts = np.array(test_sample_counts[:n_splits],
                                      dtype=np.int)

        _store('test_score', test_scores, splits=True, rank=True,
               weights=test_sample_counts if self.iid else None)
        if self.return_train_score:
            _store('train_score', train_scores, splits=True)
        _store('fit_time', fit_time)
        _store('score_time', score_time)

        best_index = np.flatnonzero(results["rank_test_score"] == 1)[0]
        best_parameters = candidate_params[best_index]

        # Use one MaskedArray and mask all the places where the param is not
        # applicable for that candidate. Use defaultdict as each candidate may
        # not contain all the params
        param_results = defaultdict(partial(
                                            MaskedArray,
                                            np.empty(n_candidates,),
                                            mask=True,
                                            dtype=object))
        for cand_i, params in enumerate(candidate_params):
            for name, value in params.items():
                # An all masked empty array gets created for the key
                # `"param_%s" % name` at the first occurence of `name`.
                # Setting the value at an index also unmasks that index
                param_results["param_%s" % name][cand_i] = value

        results.update(param_results)

        # Store a list of param dicts at the key 'params'
        results['params'] = candidate_params

        self.cv_results_ = results
        self.best_index_ = best_index
        self.n_splits_ = n_splits

        if self.refit:
            # fit the best estimator using the entire dataset
            # clone first to work around broken estimators
            best_estimator = clone(base_estimator).set_params(
                **best_parameters)
            if y is not None:
                best_estimator.fit(X, y, **self.fit_params)
            else:
                best_estimator.fit(X, **self.fit_params)
            self.best_estimator_ = best_estimator
        return self

    def _fit_best_model(self, X, y):
        """Fit the estimator copy with best parameters found to the
        provided data.

        Parameters
        ----------
        X : array-like, shape = [n_samples, n_features]
            Input data, where n_samples is the number of samples and
            n_features is the number of features.

        y : array-like, shape = [n_samples] or [n_samples, n_output],
            Target relative to X for classification or regression.

        Returns
        -------
        self
        """
        self.best_estimator_ = clone(self.estimator)
        self.best_estimator_.set_params(**self.best_params_)
        self.best_estimator_.fit(X, y)
        return self

    def _make_optimizer(self, params_space):
        """Instantiate skopt Optimizer class.

        Parameters
        ----------
        params_space : dict
            Represents parameter search space. The keys are parameter
            names (strings) and values are skopt.space.Dimension instances,
            one of Real, Integer or Categorical.

        Returns
        -------
        optimizer: Instance of the `Optimizer` class used for for search
            in some parameter space.

        """

        kwargs = self.optimizer_kwargs.copy()
        kwargs['dimensions'] = dimensions_aslist(params_space)
        optimizer = Optimizer(**kwargs)

        return optimizer

    def step(self, X, y, space_id, groups=None, n_jobs=1):
        """Generate n_jobs parameters and evaluate them in parallel.

        Having a separate function for a single step for search allows to
        save easily checkpoints for the parameter search and restore from
        possible failures.

        Parameters
        ----------
        X : array-like or sparse matrix, shape = [n_samples, n_features]
            The training input samples. Internally, it will be converted to
            ``dtype=np.float32`` and if a sparse matrix is provided
            to a sparse ``csc_matrix``.

        y : array-like, shape = [n_samples] or [n_samples, n_outputs]
            The target values (class labels) as integers or strings.

        space_id : hashable
            Identifier of parameter search space. Add search spaces with

        groups : array-like, with shape (n_samples,), optional
            Group labels for the samples used while splitting the dataset into
            train/test set.

        n_jobs : int, default=1
            Number of parameters to evaluate in parallel.

        Returns
        -------
        params_dict: dictionary with parameter values.
        """

        # convert n_jobst to int > 0 if necessary
        if n_jobs < 0:
            n_jobs = max(1, cpu_count() + n_jobs + 1)

        # use the cached optimizer for particular parameter space
        if space_id not in self.search_spaces_:
            raise ValueError("Unknown space %s" % space_id)

        # get the search space for a step
        search_space = self.search_spaces_[space_id]
        if isinstance(search_space, tuple):
            search_space, _ = search_space

        # create optimizer if not created already
        if space_id not in self.optimizer_:
            self.optimizer_[space_id] = self._make_optimizer(search_space)
        optimizer = self.optimizer_[space_id]

        # get parameter values to evaluate
        params = optimizer.ask(n_points=n_jobs)
        params_dict = [point_asdict(search_space, p) for p in params]

        # self.cv_results_ is reset at every call to _fit, keep current
        all_cv_results = self.cv_results_

        # record performances with different points
        refit = self.refit
        self.refit = False  # do not fit yet - will be fit later

        # this adds compatibility with different versions of sklearn

        self._fit(X, y, groups, params_dict)

        self.refit = refit

        # merge existing and new cv_results_
        for k in self.cv_results_:
            all_cv_results[k].extend(self.cv_results_[k])

        self.cv_results_ = all_cv_results
        self.best_index_ = np.argmax(self.cv_results_['mean_test_score'])

        # feed the point and objective back into optimizer
        local_results = self.cv_results_['mean_test_score'][-len(params):]

        # optimizer minimizes objective, hence provide negative score
        optimizer.tell(params, [-score for score in local_results])

        # fit the best model if necessary
        if self.refit:
            self._fit_best_model(X, y)

    def fit(self, X, y=None, groups=None):
        """Run fit on the estimator with randomly drawn parameters.

        Parameters
        ----------
        X : array-like, shape = [n_samples, n_features]
            Training vector, where n_samples in the number of samples and
            n_features is the number of features.

        y : array-like, shape = [n_samples] or [n_samples, n_output]
            Target relative to X for classification or regression;

        groups : array-like, with shape (n_samples,), optional
            Group labels for the samples used while splitting the dataset into
            train/test set.
        """

        # check if the list of parameter spaces is provided. If not, then
        # only step in manual mode can be used.

        if len(self.search_spaces_) == 0:
            raise ValueError(
                "Please provide search space using `add_spaces` first before"
                "calling fit method."
            )

        n_jobs = self.n_jobs

        # account for case n_jobs < 0
        if n_jobs < 0:
            n_jobs = max(1, cpu_count() + n_jobs + 1)

        for space_id in sorted(self.search_spaces_.keys()):
            elem = self.search_spaces_[space_id]

            # if not provided with search subspace, n_iter is taken as
            # self.n_iter
            if isinstance(elem, tuple):
                space, n_iter = elem
            else:
                n_iter = self.n_iter

            # do the optimization for particular search space
            while n_iter > 0:
                # when n_iter < n_jobs points left for evaluation
                n_jobs_adjusted = min(n_iter, self.n_jobs)

                self.step(
                    X, y, space_id,
                    groups=groups, n_jobs=n_jobs_adjusted
                )
                n_iter -= n_jobs

Ancestors (in MRO)

  • BayesSearchCV
  • sklearn.model_selection._search.BaseSearchCV
  • abc.NewBase
  • sklearn.base.BaseEstimator
  • sklearn.base.MetaEstimatorMixin
  • builtins.object

Static methods

def __init__(

self, estimator, search_spaces=None, optimizer_kwargs=None, n_iter=50, scoring=None, fit_params=None, n_jobs=1, iid=True, refit=True, cv=None, verbose=0, pre_dispatch='2*n_jobs', random_state=None, error_score='raise', return_train_score=True)

Initialize self. See help(type(self)) for accurate signature.

def __init__(self, estimator, search_spaces=None, optimizer_kwargs=None,
             n_iter=50, scoring=None, fit_params=None, n_jobs=1,
             iid=True, refit=True, cv=None, verbose=0,
             pre_dispatch='2*n_jobs', random_state=None,
             error_score='raise', return_train_score=True):
    # set of space name: space dict. Stored as a dict, in order
    # to make it easy to add or remove search spaces, in case
    # user wants to use `step` function directly.
    self.search_spaces_ = {}
    # can be None if user intends to provide spaces later manually
    if search_spaces is not None:
        # account for the case when search space is a dict
        if isinstance(search_spaces, dict):
            search_spaces = [search_spaces]
        self.add_spaces(list(range(len(search_spaces))), search_spaces)
    self.n_iter = n_iter
    self.random_state = random_state
    if optimizer_kwargs is None:
        self.optimizer_kwargs = {}
    else:
        self.optimizer_kwargs = optimizer_kwargs
    # this dict is used in order to keep track of skopt Optimizer
    # instances for different search spaces. `str(space)` is used as key
    # as `space` is a dict, which is unhashable. however, str(space)
    # is fixed and unique if space is not changed.
    self.optimizer_ = {}
    self.cv_results_ = defaultdict(list)
    self.best_index_ = None
    self.multimetric_ = False
    super(BayesSearchCV, self).__init__(
         estimator=estimator, scoring=scoring, fit_params=fit_params,
         n_jobs=n_jobs, iid=iid, refit=refit, cv=cv, verbose=verbose,
         pre_dispatch=pre_dispatch, error_score=error_score,
         return_train_score=return_train_score)

def add_spaces(

self, names, search_spaces)

Add a parameter search space over which to search for parameter values. Naming of search subspaces is necessary in order to specify for the step function over which subspace to perform search step.

names: str or list of str Define names for the parameter search subspaces. if search_spaces is a single dict, then names should be str representing name of the single search subspace. if search_spaces is a list of dicts, defines names for every search subspace in the list.

search_spaces : dict, list of dict or list of tuple (dict, int) Define search subspaces over which to search for parameters of the model. One of 3 following cases: 1. dictionary, where keys are parameter names (strings) and values are skopt.space.Dimension instances (Real, Integer or Categorical) or any other valid value that defines skopt dimension (see skopt.Optimizer docs). Represents search space over parameters of the provided estimator. 2. list of dictionaries: a list of dictionaries, where every dictionary fits the description given in case 1 above. If a list of dictionary objects is given, then the search is performed sequentially for every parameter space with maximum number of evaluations set to self.n_iter. 3. list of (dict, int > 0): an extension of case 2 above, where first element of every tuple is a dictionary representing some search subspace, similarly as in case 2, and second element is a number of iterations that will be spent optimizing over this subspace.

def add_spaces(self, names, search_spaces):
    """
    Add a parameter search space over which to search for parameter
    values. Naming of search subspaces is necessary in order to specify
    for the `step` function over which subspace to perform search step.
    names: str or list of str
        Define names for the parameter search subspaces.
        if search_spaces is a single dict, then names should be str
        representing name of the single search subspace.
        if search_spaces is a list of dicts, defines names for every
        search subspace in the list.
    search_spaces : dict, list of dict or list of tuple (dict, int)
        Define search subspaces over which to search for parameters
        of the model.
        One of 3 following cases:
        1. dictionary, where keys are parameter names (strings)
        and values are skopt.space.Dimension instances (Real, Integer
        or Categorical) or any other valid value that defines skopt
        dimension (see skopt.Optimizer docs). Represents search space
        over parameters of the provided estimator.
        2. list of dictionaries: a list of dictionaries, where every
        dictionary fits the description given in case 1 above.
        If a list of dictionary objects is given, then the search is
        performed sequentially for every parameter space with maximum
        number of evaluations set to self.n_iter.
        3. list of (dict, int > 0): an extension of case 2 above,
        where first element of every tuple is a dictionary representing
        some search subspace, similarly as in case 2, and second element
        is a number of iterations that will be spent optimizing over
        this subspace.
    """
    self._check_search_space(search_spaces)
    if not isinstance(search_spaces, list):
        search_spaces = [search_spaces]
    if not isinstance(names, list):
        names = [names]
    # first check whether space already exits ...
    for space, name in zip(search_spaces, names):
        if name in self.search_spaces_:
            raise ValueError("Search space %s already exists!" % name)
    for space, name in zip(search_spaces, names):
        self.search_spaces_[name] = space

def fit(

self, X, y=None, groups=None)

Run fit on the estimator with randomly drawn parameters.

Parameters

X : array-like, shape = [n_samples, n_features] Training vector, where n_samples in the number of samples and n_features is the number of features.

y : array-like, shape = [n_samples] or [n_samples, n_output] Target relative to X for classification or regression;

groups : array-like, with shape (n_samples,), optional Group labels for the samples used while splitting the dataset into train/test set.

def fit(self, X, y=None, groups=None):
    """Run fit on the estimator with randomly drawn parameters.
    Parameters
    ----------
    X : array-like, shape = [n_samples, n_features]
        Training vector, where n_samples in the number of samples and
        n_features is the number of features.
    y : array-like, shape = [n_samples] or [n_samples, n_output]
        Target relative to X for classification or regression;
    groups : array-like, with shape (n_samples,), optional
        Group labels for the samples used while splitting the dataset into
        train/test set.
    """
    # check if the list of parameter spaces is provided. If not, then
    # only step in manual mode can be used.
    if len(self.search_spaces_) == 0:
        raise ValueError(
            "Please provide search space using `add_spaces` first before"
            "calling fit method."
        )
    n_jobs = self.n_jobs
    # account for case n_jobs < 0
    if n_jobs < 0:
        n_jobs = max(1, cpu_count() + n_jobs + 1)
    for space_id in sorted(self.search_spaces_.keys()):
        elem = self.search_spaces_[space_id]
        # if not provided with search subspace, n_iter is taken as
        # self.n_iter
        if isinstance(elem, tuple):
            space, n_iter = elem
        else:
            n_iter = self.n_iter
        # do the optimization for particular search space
        while n_iter > 0:
            # when n_iter < n_jobs points left for evaluation
            n_jobs_adjusted = min(n_iter, self.n_jobs)
            self.step(
                X, y, space_id,
                groups=groups, n_jobs=n_jobs_adjusted
            )
            n_iter -= n_jobs

def step(

self, X, y, space_id, groups=None, n_jobs=1)

Generate n_jobs parameters and evaluate them in parallel.

Having a separate function for a single step for search allows to save easily checkpoints for the parameter search and restore from possible failures.

Parameters

X : array-like or sparse matrix, shape = [n_samples, n_features] The training input samples. Internally, it will be converted to dtype=np.float32 and if a sparse matrix is provided to a sparse csc_matrix.

y : array-like, shape = [n_samples] or [n_samples, n_outputs] The target values (class labels) as integers or strings.

space_id : hashable Identifier of parameter search space. Add search spaces with

groups : array-like, with shape (n_samples,), optional Group labels for the samples used while splitting the dataset into train/test set.

n_jobs : int, default=1 Number of parameters to evaluate in parallel.

Returns

params_dict: dictionary with parameter values.

def step(self, X, y, space_id, groups=None, n_jobs=1):
    """Generate n_jobs parameters and evaluate them in parallel.
    Having a separate function for a single step for search allows to
    save easily checkpoints for the parameter search and restore from
    possible failures.
    Parameters
    ----------
    X : array-like or sparse matrix, shape = [n_samples, n_features]
        The training input samples. Internally, it will be converted to
        ``dtype=np.float32`` and if a sparse matrix is provided
        to a sparse ``csc_matrix``.
    y : array-like, shape = [n_samples] or [n_samples, n_outputs]
        The target values (class labels) as integers or strings.
    space_id : hashable
        Identifier of parameter search space. Add search spaces with
    groups : array-like, with shape (n_samples,), optional
        Group labels for the samples used while splitting the dataset into
        train/test set.
    n_jobs : int, default=1
        Number of parameters to evaluate in parallel.
    Returns
    -------
    params_dict: dictionary with parameter values.
    """
    # convert n_jobst to int > 0 if necessary
    if n_jobs < 0:
        n_jobs = max(1, cpu_count() + n_jobs + 1)
    # use the cached optimizer for particular parameter space
    if space_id not in self.search_spaces_:
        raise ValueError("Unknown space %s" % space_id)
    # get the search space for a step
    search_space = self.search_spaces_[space_id]
    if isinstance(search_space, tuple):
        search_space, _ = search_space
    # create optimizer if not created already
    if space_id not in self.optimizer_:
        self.optimizer_[space_id] = self._make_optimizer(search_space)
    optimizer = self.optimizer_[space_id]
    # get parameter values to evaluate
    params = optimizer.ask(n_points=n_jobs)
    params_dict = [point_asdict(search_space, p) for p in params]
    # self.cv_results_ is reset at every call to _fit, keep current
    all_cv_results = self.cv_results_
    # record performances with different points
    refit = self.refit
    self.refit = False  # do not fit yet - will be fit later
    # this adds compatibility with different versions of sklearn
    self._fit(X, y, groups, params_dict)
    self.refit = refit
    # merge existing and new cv_results_
    for k in self.cv_results_:
        all_cv_results[k].extend(self.cv_results_[k])
    self.cv_results_ = all_cv_results
    self.best_index_ = np.argmax(self.cv_results_['mean_test_score'])
    # feed the point and objective back into optimizer
    local_results = self.cv_results_['mean_test_score'][-len(params):]
    # optimizer minimizes objective, hence provide negative score
    optimizer.tell(params, [-score for score in local_results])
    # fit the best model if necessary
    if self.refit:
        self._fit_best_model(X, y)

Instance variables

var best_index_

var best_params_

var best_score_

var classes_

var cv_results_

var grid_scores_

var multimetric_

var n_iter

var optimizer_

var random_state

var search_spaces_

class Optimizer

Run bayesian optimisation loop.

An Optimizer represents the steps of a bayesian optimisation loop. To use it you need to provide your own loop mechanism. The various optimisers provided by skopt use this class under the hood.

Use this class directly if you want to control the iterations of your bayesian optimisation loop.

Parameters

  • dimensions [list, shape=(n_dims,)]: List of search space dimensions. Each search dimension can be defined either as

    • a (upper_bound, lower_bound) tuple (for Real or Integer dimensions),
    • a (upper_bound, lower_bound, "prior") tuple (for Real dimensions),
    • as a list of categories (for Categorical dimensions), or
    • an instance of a Dimension object (Real, Integer or Categorical).
  • base_estimator ["GP", "RF", "ET", "GBRT" or sklearn regressor, default="GP"]: Should inherit from sklearn.base.RegressorMixin. In addition the predict method, should have an optional return_std argument, which returns std(Y | x)`` along withE[Y | x]`. If base_estimator is one of ["GP", "RF", "ET", "GBRT"], a default surrogate model of the corresponding type is used corresponding to what is used in the minimize functions.

  • n_random_starts [int, default=10]: DEPRECATED, use n_initial_points instead.

  • n_initial_points [int, default=10]: Number of evaluations of func with initialization points before approximating it with base_estimator. Points provided as x0 count as initialization points. If len(x0) < n_initial_points additional points are sampled at random.

  • acq_func [string, default="EI"]: Function to minimize over the posterior distribution. Can be either

    • "LCB" for lower confidence bound.
    • "EI" for negative expected improvement.
    • "PI" for negative probability of improvement.
    • "gp_hedge" Probabilistically choose one of the above three acquisition functions at every iteration.
      • The gains g_i are initialized to zero.
      • At every iteration,
        • Each acquisition function is optimised independently to propose an candidate point X_i.
        • Out of all these candidate points, the next point X_best is chosen by $softmax(\eta g_i)$
        • After fitting the surrogate model with (X_best, y_best), the gains are updated such that $g_i -= \mu(X_i)$
    • `"EIps" for negated expected improvement per second to take into account the function compute time. Then, the objective function is assumed to return two values, the first being the objective value and the second being the time taken in seconds.
    • "PIps" for negated probability of improvement per second. The return type of the objective function is assumed to be similar to that of `"EIps
  • acq_optimizer [string, "sampling" or "lbfgs", default="auto"]: Method to minimize the acquistion function. The fit model is updated with the optimal value obtained by optimizing acq_func with acq_optimizer.

    • If set to "auto", then acq_optimizer is configured on the basis of the base_estimator and the space searched over. If the space is Categorical or if the estimator provided based on tree-models then this is set to be "sampling"`.
    • If set to "sampling", then acq_func is optimized by computing acq_func at n_points randomly sampled points.
    • If set to "lbfgs", then acq_func is optimized by
      • Sampling n_restarts_optimizer points randomly.
      • "lbfgs" is run for 20 iterations with these points as initial points to find local minima.
      • The optimal of these local minima is used to update the prior.
  • random_state [int, RandomState instance, or None (default)]: Set random state to something other than None for reproducible results.

  • acq_func_kwargs [dict]: Additional arguments to be passed to the acquistion function.

  • acq_optimizer_kwargs [dict]: Additional arguments to be passed to the acquistion optimizer.

Attributes

  • Xi [list]: Points at which objective has been evaluated.
  • yi [scalar]: Values of objective at corresponding points in Xi.
  • models [list]: Regression models used to fit observations and compute acquisition function.
  • space An instance of skopt.space.Space. Stores parameter search space used to sample points, bounds, and type of parameters.
class Optimizer(object):
    """Run bayesian optimisation loop.

    An `Optimizer` represents the steps of a bayesian optimisation loop. To
    use it you need to provide your own loop mechanism. The various
    optimisers provided by `skopt` use this class under the hood.

    Use this class directly if you want to control the iterations of your
    bayesian optimisation loop.

    Parameters
    ----------
    * `dimensions` [list, shape=(n_dims,)]:
        List of search space dimensions.
        Each search dimension can be defined either as

        - a `(upper_bound, lower_bound)` tuple (for `Real` or `Integer`
          dimensions),
        - a `(upper_bound, lower_bound, "prior")` tuple (for `Real`
          dimensions),
        - as a list of categories (for `Categorical` dimensions), or
        - an instance of a `Dimension` object (`Real`, `Integer` or
          `Categorical`).

    * `base_estimator` ["GP", "RF", "ET", "GBRT" or sklearn regressor, default="GP"]:
        Should inherit from `sklearn.base.RegressorMixin`.
        In addition the `predict` method, should have an optional `return_std`
        argument, which returns `std(Y | x)`` along with `E[Y | x]`.
        If base_estimator is one of ["GP", "RF", "ET", "GBRT"], a default
        surrogate model of the corresponding type is used corresponding to what
        is used in the minimize functions.

    * `n_random_starts` [int, default=10]:
        DEPRECATED, use `n_initial_points` instead.

    * `n_initial_points` [int, default=10]:
        Number of evaluations of `func` with initialization points
        before approximating it with `base_estimator`. Points provided as
        `x0` count as initialization points. If len(x0) < n_initial_points
        additional points are sampled at random.

    * `acq_func` [string, default=`"EI"`]:
        Function to minimize over the posterior distribution. Can be either

        - `"LCB"` for lower confidence bound.
        - `"EI"` for negative expected improvement.
        - `"PI"` for negative probability of improvement.
        - `"gp_hedge"` Probabilistically choose one of the above three
          acquisition functions at every iteration.
            - The gains `g_i` are initialized to zero.
            - At every iteration,
                - Each acquisition function is optimised independently to
                  propose an candidate point `X_i`.
                - Out of all these candidate points, the next point `X_best` is
                  chosen by $softmax(\eta g_i)$
                - After fitting the surrogate model with `(X_best, y_best)`,
                  the gains are updated such that $g_i -= \mu(X_i)$
        - `"EIps" for negated expected improvement per second to take into
          account the function compute time. Then, the objective function is
          assumed to return two values, the first being the objective value and
          the second being the time taken in seconds.
        - `"PIps"` for negated probability of improvement per second. The
          return type of the objective function is assumed to be similar to
          that of `"EIps

    * `acq_optimizer` [string, `"sampling"` or `"lbfgs"`, default=`"auto"`]:
        Method to minimize the acquistion function. The fit model
        is updated with the optimal value obtained by optimizing `acq_func`
        with `acq_optimizer`.

        - If set to `"auto"`, then `acq_optimizer` is configured on the
          basis of the base_estimator and the space searched over.
          If the space is Categorical or if the estimator provided based on
          tree-models then this is set to be "sampling"`.
        - If set to `"sampling"`, then `acq_func` is optimized by computing
          `acq_func` at `n_points` randomly sampled points.
        - If set to `"lbfgs"`, then `acq_func` is optimized by
              - Sampling `n_restarts_optimizer` points randomly.
              - `"lbfgs"` is run for 20 iterations with these points as initial
                points to find local minima.
              - The optimal of these local minima is used to update the prior.

    * `random_state` [int, RandomState instance, or None (default)]:
        Set random state to something other than None for reproducible
        results.

    * `acq_func_kwargs` [dict]:
        Additional arguments to be passed to the acquistion function.

    * `acq_optimizer_kwargs` [dict]:
        Additional arguments to be passed to the acquistion optimizer.


    Attributes
    ----------
    * `Xi` [list]:
        Points at which objective has been evaluated.
    * `yi` [scalar]:
        Values of objective at corresponding points in `Xi`.
    * `models` [list]:
        Regression models used to fit observations and compute acquisition
        function.
    * `space`
        An instance of `skopt.space.Space`. Stores parameter search space used
        to sample points, bounds, and type of parameters.

    """
    def __init__(self, dimensions, base_estimator="gp",
                 n_random_starts=None, n_initial_points=10,
                 acq_func="gp_hedge",
                 acq_optimizer="auto",
                 random_state=None, acq_func_kwargs=None,
                 acq_optimizer_kwargs=None):
        # Arguments that are just stored not checked
        self.acq_func = acq_func
        self.rng = check_random_state(random_state)
        self.acq_func_kwargs = acq_func_kwargs

        allowed_acq_funcs = ["gp_hedge", "EI", "LCB", "PI", "EIps", "PIps"]
        if self.acq_func not in allowed_acq_funcs:
            raise ValueError("expected acq_func to be in %s, got %s" %
                             (",".join(allowed_acq_funcs), self.acq_func))
        if self.acq_func == "gp_hedge":
            self.cand_acq_funcs_ = ["EI", "LCB", "PI"]
            self.gains_ = np.zeros(3)
        else:
            self.cand_acq_funcs_ = [self.acq_func]

        if acq_func_kwargs is None:
            acq_func_kwargs = dict()
        self.eta = acq_func_kwargs.get("eta", 1.0)

        if acq_optimizer_kwargs is None:
            acq_optimizer_kwargs = dict()

        self.n_points = acq_optimizer_kwargs.get("n_points", 10000)
        self.n_restarts_optimizer = acq_optimizer_kwargs.get(
            "n_restarts_optimizer", 5)
        n_jobs = acq_optimizer_kwargs.get("n_jobs", 1)
        self.acq_optimizer_kwargs = acq_optimizer_kwargs

        if n_random_starts is not None:
            warnings.warn(("n_random_starts will be removed in favour of "
                           "n_initial_points."),
                          DeprecationWarning)
            n_initial_points = n_random_starts

        self._check_arguments(base_estimator, n_initial_points, acq_optimizer,
                              dimensions)

        if isinstance(self.base_estimator_, GaussianProcessRegressor):
            dimensions = normalize_dimensions(dimensions)

        self.space = Space(dimensions)
        self.models = []
        self.Xi = []
        self.yi = []

        self._cat_inds = []
        self._non_cat_inds = []
        for ind, dim in enumerate(self.space.dimensions):
            if isinstance(dim, Categorical):
                self._cat_inds.append(ind)
            else:
                self._non_cat_inds.append(ind)

        self.n_jobs = n_jobs

        # The cache of responses of `ask` method for n_points not None.
        # This ensures that multiple calls to `ask` with n_points set
        # return same sets of points.
        # The cache is reset to {} at every call to `tell`.
        self.cache_ = {}

    def _check_arguments(self, base_estimator, n_initial_points,
                         acq_optimizer, dimensions):
        """Check arguments for sanity."""

        if isinstance(base_estimator, str):
            base_estimator = cook_estimator(
                base_estimator, space=dimensions, random_state=self.rng)

        if not is_regressor(base_estimator) and base_estimator is not None:
            raise ValueError(
                "%s has to be a regressor." % base_estimator)

        if "ps" in self.acq_func:
            self.base_estimator_ = MultiOutputRegressor(base_estimator)
        else:
            self.base_estimator_ = base_estimator

        if n_initial_points < 0:
            raise ValueError(
                "Expected `n_initial_points` >= 0, got %d" % n_initial_points)
        self._n_initial_points = n_initial_points
        self.n_initial_points_ = n_initial_points

        if acq_optimizer == "auto":
            if has_gradients(self.base_estimator_):
                acq_optimizer = "lbfgs"
            else:
                acq_optimizer = "sampling"

        if acq_optimizer not in ["lbfgs", "sampling"]:
            raise ValueError("Expected acq_optimizer to be 'lbfgs' or "
                             "'sampling', got {0}".format(acq_optimizer))

        if (not has_gradients(self.base_estimator_) and
            acq_optimizer != "sampling"):
            raise ValueError("The regressor {0} should run with "
                             "acq_optimizer"
                             "='sampling'.".format(type(base_estimator)))

        self.acq_optimizer = acq_optimizer

    def copy(self, random_state=None):
        """Create a shallow copy of an instance of the optimizer.

        Parameters
        ----------
        * `random_state` [int, RandomState instance, or None (default)]:
            Set the random state of the copy.
        """

        optimizer = Optimizer(
            dimensions=self.space.dimensions,
            base_estimator=self.base_estimator_,
            n_initial_points=self.n_initial_points_,
            acq_func=self.acq_func,
            acq_optimizer=self.acq_optimizer,
            acq_func_kwargs=self.acq_func_kwargs,
            acq_optimizer_kwargs=self.acq_optimizer_kwargs,
            random_state=random_state,
        )

        if hasattr(self, "gains_"):
            optimizer.gains_ = np.copy(self.gains_)

        if self.Xi:
            optimizer.tell(self.Xi, self.yi)

        return optimizer

    def ask(self, n_points=None, strategy="cl_min"):
        """Query point or multiple points at which objective should be evaluated.

        * `n_points` [int or None, default=None]:
            Number of points returned by the ask method.
            If the value is None, a single point to evaluate is returned.
            Otherwise a list of points to evaluate is returned of size
            n_points. This is useful if you can evaluate your objective in
            parallel, and thus obtain more objective function evaluations per
            unit of time.

        * `strategy` [string, default=`"cl_min"`]:
            Method to use to sample multiple points (see also `n_points`
            description). This parameter is ignored if n_points = None.
            Supported options are `"cl_min"`, `"cl_mean"` or `"cl_max"`.

            - If set to `"cl_min"`, then constant liar strtategy is used
               with lie objective value being minimum of observed objective
               values. `"cl_mean"` and `"cl_max"` means mean and max of values
               respectively. For details on this strategy see:

               https://hal.archives-ouvertes.fr/hal-00732512/document

               With this strategy a copy of optimizer is created, which is
               then asked for a point, and the point is told to the copy of
               optimizer with some fake objective (lie), the next point is
               asked from copy, it is also told to the copy with fake
               objective and so on. The type of lie defines different
               flavours of `cl_x` strategies.

        """
        if n_points is None:
            return self._ask()

        supported_strategies = ["cl_min", "cl_mean", "cl_max"]

        if not (isinstance(n_points, int) and n_points > 0):
            raise ValueError(
                "n_points should be int > 0, got " + str(n_points)
            )

        if strategy not in supported_strategies:
            raise ValueError(
                "Expected parallel_strategy to be one of " +
                str(supported_strategies) + ", " + "got %s" % strategy
            )

        # Caching the result with n_points not None. If some new parameters
        # are provided to the ask, the cache_ is not used.
        if (n_points, strategy) in self.cache_:
            return self.cache_[(n_points, strategy)]

        # Copy of the optimizer is made in order to manage the
        # deletion of points with "lie" objective (the copy of
        # oiptimizer is simply discarded)
        opt = self.copy()

        X = []
        for i in range(n_points):
            x = opt.ask()
            X.append(x)
            if strategy == "cl_min":
                y_lie = np.min(opt.yi) if opt.yi else 0.0  # CL-min lie
            elif strategy == "cl_mean":
                y_lie = np.mean(opt.yi) if opt.yi else 0.0  # CL-mean lie
            else:
                y_lie = np.max(opt.yi) if opt.yi else 0.0  # CL-max lie
            opt.tell(x, y_lie)  # lie to the optimizer

        self.cache_ = {(n_points, strategy): X}  # cache_ the result

        return X

    def _ask(self):
        """Suggest next point at which to evaluate the objective.

        Return a random point while not at least `n_initial_points`
        observations have been `tell`ed, after that `base_estimator` is used
        to determine the next point.
        """
        if self._n_initial_points > 0 or self.base_estimator_ is None:
            # this will not make a copy of `self.rng` and hence keep advancing
            # our random state.
            return self.space.rvs(random_state=self.rng)[0]

        else:
            if not self.models:
                raise RuntimeError("Random evaluations exhausted and no "
                                   "model has been fit.")

            next_x = self._next_x
            min_delta_x = min([self.space.distance(next_x, xi)
                               for xi in self.Xi])
            if abs(min_delta_x) <= 1e-8:
                warnings.warn("The objective has been evaluated "
                              "at this point before.")

            # return point computed from last call to tell()
            return next_x

    def tell(self, x, y, fit=True):
        """Record an observation (or several) of the objective function.

        Provide values of the objective function at points suggested by `ask()`
        or other points. By default a new model will be fit to all
        observations. The new model is used to suggest the next point at
        which to evaluate the objective. This point can be retrieved by calling
        `ask()`.

        To add observations without fitting a new model set `fit` to False.

        To add multiple observations in a batch pass a list-of-lists for `x`
        and a list of scalars for `y`.

        Parameters
        ----------
        * `x` [list or list-of-lists]:
            Point at which objective was evaluated.

        * `y` [scalar or list]:
            Value of objective at `x`.

        * `fit` [bool, default=True]
            Fit a model to observed evaluations of the objective. A model will
            only be fitted after `n_initial_points` points have been told to
            the optimizer irrespective of the value of `fit`.
        """
        check_x_in_space(x, self.space)

        if "ps" in self.acq_func:
            if is_2Dlistlike(x):
                if np.ndim(y) == 2 and np.shape(y)[1] == 2:
                    y = [[val, log(t)] for (val, t) in y]
                    self.Xi.extend(x)
                    self.yi.extend(y)
                else:
                    raise TypeError("expcted y to be a list of (func_val, t)")
                self._n_initial_points -= len(y)
            elif is_listlike(x):
                if np.ndim(y) == 1 and len(y) == 2:
                    y = list(y)
                    y[1] = log(y[1])
                    self.Xi.append(x)
                    self.yi.append(y)
                else:
                    raise TypeError("expected y to be (func_val, t)")
                self._n_initial_points -= 1

        # if y isn't a scalar it means we have been handed a batch of points
        elif is_listlike(y) and is_2Dlistlike(x):
            self.Xi.extend(x)
            self.yi.extend(y)
            self._n_initial_points -= len(y)

        elif is_listlike(x):
            if isinstance(y, Number):
                self.Xi.append(x)
                self.yi.append(y)
                self._n_initial_points -= 1
            else:
                raise ValueError("`func` should return a scalar")

        else:
            raise ValueError("Type of arguments `x` (%s) and `y` (%s) "
                             "not compatible." % (type(x), type(y)))

        # optimizer learned somethnig new - discard cache
        self.cache_ = {}

        # after being "told" n_initial_points we switch from sampling
        # random points to using a surrogate model
        if (fit and self._n_initial_points <= 0 and
           self.base_estimator_ is not None):
            transformed_bounds = np.array(self.space.transformed_bounds)
            est = clone(self.base_estimator_)

            with warnings.catch_warnings():
                warnings.simplefilter("ignore")
                est.fit(self.space.transform(self.Xi), self.yi)

            if hasattr(self, "next_xs_") and self.acq_func == "gp_hedge":
                self.gains_ -= est.predict(np.vstack(self.next_xs_))
            self.models.append(est)

            # even with BFGS as optimizer we want to sample a large number
            # of points and then pick the best ones as starting points
            X = self.space.transform(self.space.rvs(
                n_samples=self.n_points, random_state=self.rng))

            self.next_xs_ = []
            for cand_acq_func in self.cand_acq_funcs_:
                values = _gaussian_acquisition(
                    X=X, model=est, y_opt=np.min(self.yi),
                    acq_func=cand_acq_func,
                    acq_func_kwargs=self.acq_func_kwargs)
                # Find the minimum of the acquisition function by randomly
                # sampling points from the space
                if self.acq_optimizer == "sampling":
                    next_x = X[np.argmin(values)]

                # Use BFGS to find the mimimum of the acquisition function, the
                # minimization starts from `n_restarts_optimizer` different
                # points and the best minimum is used
                elif self.acq_optimizer == "lbfgs":
                    x0 = X[np.argsort(values)[:self.n_restarts_optimizer]]

                    with warnings.catch_warnings():
                        warnings.simplefilter("ignore")
                        results = Parallel(n_jobs=self.n_jobs)(
                            delayed(fmin_l_bfgs_b)(
                                gaussian_acquisition_1D, x,
                                args=(est, np.min(self.yi), cand_acq_func,
                                      self.acq_func_kwargs),
                                bounds=self.space.transformed_bounds,
                                approx_grad=False,
                                maxiter=20)
                            for x in x0)

                    cand_xs = np.array([r[0] for r in results])
                    cand_acqs = np.array([r[1] for r in results])
                    next_x = cand_xs[np.argmin(cand_acqs)]

                # lbfgs should handle this but just in case there are
                # precision errors.
                if not self.space.is_categorical:
                    next_x = np.clip(
                        next_x, transformed_bounds[:, 0],
                        transformed_bounds[:, 1])
                self.next_xs_.append(next_x)

            if self.acq_func == "gp_hedge":
                logits = np.array(self.gains_)
                logits -= np.max(logits)
                exp_logits = np.exp(self.eta * logits)
                probs = exp_logits / np.sum(exp_logits)
                next_x = self.next_xs_[np.argmax(self.rng.multinomial(1,
                                                                      probs))]
            else:
                next_x = self.next_xs_[0]

            # note the need for [0] at the end
            self._next_x = self.space.inverse_transform(
                next_x.reshape((1, -1)))[0]

        # Pack results
        return create_result(self.Xi, self.yi, self.space, self.rng,
                             models=self.models)

    def run(self, func, n_iter=1):
        """Execute ask() + tell() `n_iter` times"""
        for _ in range(n_iter):
            x = self.ask()
            self.tell(x, func(x))

        return create_result(self.Xi, self.yi, self.space, self.rng,
                             models=self.models)

Ancestors (in MRO)

Static methods

def __init__(

self, dimensions, base_estimator='gp', n_random_starts=None, n_initial_points=10, acq_func='gp_hedge', acq_optimizer='auto', random_state=None, acq_func_kwargs=None, acq_optimizer_kwargs=None)

Initialize self. See help(type(self)) for accurate signature.

def __init__(self, dimensions, base_estimator="gp",
             n_random_starts=None, n_initial_points=10,
             acq_func="gp_hedge",
             acq_optimizer="auto",
             random_state=None, acq_func_kwargs=None,
             acq_optimizer_kwargs=None):
    # Arguments that are just stored not checked
    self.acq_func = acq_func
    self.rng = check_random_state(random_state)
    self.acq_func_kwargs = acq_func_kwargs
    allowed_acq_funcs = ["gp_hedge", "EI", "LCB", "PI", "EIps", "PIps"]
    if self.acq_func not in allowed_acq_funcs:
        raise ValueError("expected acq_func to be in %s, got %s" %
                         (",".join(allowed_acq_funcs), self.acq_func))
    if self.acq_func == "gp_hedge":
        self.cand_acq_funcs_ = ["EI", "LCB", "PI"]
        self.gains_ = np.zeros(3)
    else:
        self.cand_acq_funcs_ = [self.acq_func]
    if acq_func_kwargs is None:
        acq_func_kwargs = dict()
    self.eta = acq_func_kwargs.get("eta", 1.0)
    if acq_optimizer_kwargs is None:
        acq_optimizer_kwargs = dict()
    self.n_points = acq_optimizer_kwargs.get("n_points", 10000)
    self.n_restarts_optimizer = acq_optimizer_kwargs.get(
        "n_restarts_optimizer", 5)
    n_jobs = acq_optimizer_kwargs.get("n_jobs", 1)
    self.acq_optimizer_kwargs = acq_optimizer_kwargs
    if n_random_starts is not None:
        warnings.warn(("n_random_starts will be removed in favour of "
                       "n_initial_points."),
                      DeprecationWarning)
        n_initial_points = n_random_starts
    self._check_arguments(base_estimator, n_initial_points, acq_optimizer,
                          dimensions)
    if isinstance(self.base_estimator_, GaussianProcessRegressor):
        dimensions = normalize_dimensions(dimensions)
    self.space = Space(dimensions)
    self.models = []
    self.Xi = []
    self.yi = []
    self._cat_inds = []
    self._non_cat_inds = []
    for ind, dim in enumerate(self.space.dimensions):
        if isinstance(dim, Categorical):
            self._cat_inds.append(ind)
        else:
            self._non_cat_inds.append(ind)
    self.n_jobs = n_jobs
    # The cache of responses of `ask` method for n_points not None.
    # This ensures that multiple calls to `ask` with n_points set
    # return same sets of points.
    # The cache is reset to {} at every call to `tell`.
    self.cache_ = {}

def ask(

self, n_points=None, strategy='cl_min')

Query point or multiple points at which objective should be evaluated.

  • n_points [int or None, default=None]: Number of points returned by the ask method. If the value is None, a single point to evaluate is returned. Otherwise a list of points to evaluate is returned of size n_points. This is useful if you can evaluate your objective in parallel, and thus obtain more objective function evaluations per unit of time.

  • strategy [string, default="cl_min"]: Method to use to sample multiple points (see also n_points description). This parameter is ignored if n_points = None. Supported options are "cl_min", "cl_mean" or "cl_max".

    • If set to "cl_min", then constant liar strtategy is used with lie objective value being minimum of observed objective values. "cl_mean" and "cl_max" means mean and max of values respectively. For details on this strategy see:

    https://hal.archives-ouvertes.fr/hal-00732512/document

    With this strategy a copy of optimizer is created, which is then asked for a point, and the point is told to the copy of optimizer with some fake objective (lie), the next point is asked from copy, it is also told to the copy with fake objective and so on. The type of lie defines different flavours of cl_x strategies.

def ask(self, n_points=None, strategy="cl_min"):
    """Query point or multiple points at which objective should be evaluated.
    * `n_points` [int or None, default=None]:
        Number of points returned by the ask method.
        If the value is None, a single point to evaluate is returned.
        Otherwise a list of points to evaluate is returned of size
        n_points. This is useful if you can evaluate your objective in
        parallel, and thus obtain more objective function evaluations per
        unit of time.
    * `strategy` [string, default=`"cl_min"`]:
        Method to use to sample multiple points (see also `n_points`
        description). This parameter is ignored if n_points = None.
        Supported options are `"cl_min"`, `"cl_mean"` or `"cl_max"`.
        - If set to `"cl_min"`, then constant liar strtategy is used
           with lie objective value being minimum of observed objective
           values. `"cl_mean"` and `"cl_max"` means mean and max of values
           respectively. For details on this strategy see:
           https://hal.archives-ouvertes.fr/hal-00732512/document
           With this strategy a copy of optimizer is created, which is
           then asked for a point, and the point is told to the copy of
           optimizer with some fake objective (lie), the next point is
           asked from copy, it is also told to the copy with fake
           objective and so on. The type of lie defines different
           flavours of `cl_x` strategies.
    """
    if n_points is None:
        return self._ask()
    supported_strategies = ["cl_min", "cl_mean", "cl_max"]
    if not (isinstance(n_points, int) and n_points > 0):
        raise ValueError(
            "n_points should be int > 0, got " + str(n_points)
        )
    if strategy not in supported_strategies:
        raise ValueError(
            "Expected parallel_strategy to be one of " +
            str(supported_strategies) + ", " + "got %s" % strategy
        )
    # Caching the result with n_points not None. If some new parameters
    # are provided to the ask, the cache_ is not used.
    if (n_points, strategy) in self.cache_:
        return self.cache_[(n_points, strategy)]
    # Copy of the optimizer is made in order to manage the
    # deletion of points with "lie" objective (the copy of
    # oiptimizer is simply discarded)
    opt = self.copy()
    X = []
    for i in range(n_points):
        x = opt.ask()
        X.append(x)
        if strategy == "cl_min":
            y_lie = np.min(opt.yi) if opt.yi else 0.0  # CL-min lie
        elif strategy == "cl_mean":
            y_lie = np.mean(opt.yi) if opt.yi else 0.0  # CL-mean lie
        else:
            y_lie = np.max(opt.yi) if opt.yi else 0.0  # CL-max lie
        opt.tell(x, y_lie)  # lie to the optimizer
    self.cache_ = {(n_points, strategy): X}  # cache_ the result
    return X

def copy(

self, random_state=None)

Create a shallow copy of an instance of the optimizer.

Parameters

  • random_state [int, RandomState instance, or None (default)]: Set the random state of the copy.
def copy(self, random_state=None):
    """Create a shallow copy of an instance of the optimizer.
    Parameters
    ----------
    * `random_state` [int, RandomState instance, or None (default)]:
        Set the random state of the copy.
    """
    optimizer = Optimizer(
        dimensions=self.space.dimensions,
        base_estimator=self.base_estimator_,
        n_initial_points=self.n_initial_points_,
        acq_func=self.acq_func,
        acq_optimizer=self.acq_optimizer,
        acq_func_kwargs=self.acq_func_kwargs,
        acq_optimizer_kwargs=self.acq_optimizer_kwargs,
        random_state=random_state,
    )
    if hasattr(self, "gains_"):
        optimizer.gains_ = np.copy(self.gains_)
    if self.Xi:
        optimizer.tell(self.Xi, self.yi)
    return optimizer

def run(

self, func, n_iter=1)

Execute ask() + tell() n_iter times

def run(self, func, n_iter=1):
    """Execute ask() + tell() `n_iter` times"""
    for _ in range(n_iter):
        x = self.ask()
        self.tell(x, func(x))
    return create_result(self.Xi, self.yi, self.space, self.rng,
                         models=self.models)

def tell(

self, x, y, fit=True)

Record an observation (or several) of the objective function.

Provide values of the objective function at points suggested by ask() or other points. By default a new model will be fit to all observations. The new model is used to suggest the next point at which to evaluate the objective. This point can be retrieved by calling ask().

To add observations without fitting a new model set fit to False.

To add multiple observations in a batch pass a list-of-lists for x and a list of scalars for y.

Parameters

  • x [list or list-of-lists]: Point at which objective was evaluated.

  • y [scalar or list]: Value of objective at x.

  • fit [bool, default=True] Fit a model to observed evaluations of the objective. A model will only be fitted after n_initial_points points have been told to the optimizer irrespective of the value of fit.

def tell(self, x, y, fit=True):
    """Record an observation (or several) of the objective function.
    Provide values of the objective function at points suggested by `ask()`
    or other points. By default a new model will be fit to all
    observations. The new model is used to suggest the next point at
    which to evaluate the objective. This point can be retrieved by calling
    `ask()`.
    To add observations without fitting a new model set `fit` to False.
    To add multiple observations in a batch pass a list-of-lists for `x`
    and a list of scalars for `y`.
    Parameters
    ----------
    * `x` [list or list-of-lists]:
        Point at which objective was evaluated.
    * `y` [scalar or list]:
        Value of objective at `x`.
    * `fit` [bool, default=True]
        Fit a model to observed evaluations of the objective. A model will
        only be fitted after `n_initial_points` points have been told to
        the optimizer irrespective of the value of `fit`.
    """
    check_x_in_space(x, self.space)
    if "ps" in self.acq_func:
        if is_2Dlistlike(x):
            if np.ndim(y) == 2 and np.shape(y)[1] == 2:
                y = [[val, log(t)] for (val, t) in y]
                self.Xi.extend(x)
                self.yi.extend(y)
            else:
                raise TypeError("expcted y to be a list of (func_val, t)")
            self._n_initial_points -= len(y)
        elif is_listlike(x):
            if np.ndim(y) == 1 and len(y) == 2:
                y = list(y)
                y[1] = log(y[1])
                self.Xi.append(x)
                self.yi.append(y)
            else:
                raise TypeError("expected y to be (func_val, t)")
            self._n_initial_points -= 1
    # if y isn't a scalar it means we have been handed a batch of points
    elif is_listlike(y) and is_2Dlistlike(x):
        self.Xi.extend(x)
        self.yi.extend(y)
        self._n_initial_points -= len(y)
    elif is_listlike(x):
        if isinstance(y, Number):
            self.Xi.append(x)
            self.yi.append(y)
            self._n_initial_points -= 1
        else:
            raise ValueError("`func` should return a scalar")
    else:
        raise ValueError("Type of arguments `x` (%s) and `y` (%s) "
                         "not compatible." % (type(x), type(y)))
    # optimizer learned somethnig new - discard cache
    self.cache_ = {}
    # after being "told" n_initial_points we switch from sampling
    # random points to using a surrogate model
    if (fit and self._n_initial_points <= 0 and
       self.base_estimator_ is not None):
        transformed_bounds = np.array(self.space.transformed_bounds)
        est = clone(self.base_estimator_)
        with warnings.catch_warnings():
            warnings.simplefilter("ignore")
            est.fit(self.space.transform(self.Xi), self.yi)
        if hasattr(self, "next_xs_") and self.acq_func == "gp_hedge":
            self.gains_ -= est.predict(np.vstack(self.next_xs_))
        self.models.append(est)
        # even with BFGS as optimizer we want to sample a large number
        # of points and then pick the best ones as starting points
        X = self.space.transform(self.space.rvs(
            n_samples=self.n_points, random_state=self.rng))
        self.next_xs_ = []
        for cand_acq_func in self.cand_acq_funcs_:
            values = _gaussian_acquisition(
                X=X, model=est, y_opt=np.min(self.yi),
                acq_func=cand_acq_func,
                acq_func_kwargs=self.acq_func_kwargs)
            # Find the minimum of the acquisition function by randomly
            # sampling points from the space
            if self.acq_optimizer == "sampling":
                next_x = X[np.argmin(values)]
            # Use BFGS to find the mimimum of the acquisition function, the
            # minimization starts from `n_restarts_optimizer` different
            # points and the best minimum is used
            elif self.acq_optimizer == "lbfgs":
                x0 = X[np.argsort(values)[:self.n_restarts_optimizer]]
                with warnings.catch_warnings():
                    warnings.simplefilter("ignore")
                    results = Parallel(n_jobs=self.n_jobs)(
                        delayed(fmin_l_bfgs_b)(
                            gaussian_acquisition_1D, x,
                            args=(est, np.min(self.yi), cand_acq_func,
                                  self.acq_func_kwargs),
                            bounds=self.space.transformed_bounds,
                            approx_grad=False,
                            maxiter=20)
                        for x in x0)
                cand_xs = np.array([r[0] for r in results])
                cand_acqs = np.array([r[1] for r in results])
                next_x = cand_xs[np.argmin(cand_acqs)]
            # lbfgs should handle this but just in case there are
            # precision errors.
            if not self.space.is_categorical:
                next_x = np.clip(
                    next_x, transformed_bounds[:, 0],
                    transformed_bounds[:, 1])
            self.next_xs_.append(next_x)
        if self.acq_func == "gp_hedge":
            logits = np.array(self.gains_)
            logits -= np.max(logits)
            exp_logits = np.exp(self.eta * logits)
            probs = exp_logits / np.sum(exp_logits)
            next_x = self.next_xs_[np.argmax(self.rng.multinomial(1,
                                                                  probs))]
        else:
            next_x = self.next_xs_[0]
        # note the need for [0] at the end
        self._next_x = self.space.inverse_transform(
            next_x.reshape((1, -1)))[0]
    # Pack results
    return create_result(self.Xi, self.yi, self.space, self.rng,
                         models=self.models)

Instance variables

var Xi

var acq_func

var acq_func_kwargs

var acq_optimizer_kwargs

var cache_

var eta

var models

var n_jobs

var n_points

var n_restarts_optimizer

var rng

var space

var yi

class Space

Search space.

class Space(object):
    """Search space."""

    def __init__(self, dimensions):
        """Initialize a search space from given specifications.

        Parameters
        ----------
        * `dimensions` [list, shape=(n_dims,)]:
            List of search space dimensions.
            Each search dimension can be defined either as

            - a `(lower_bound, upper_bound)` tuple (for `Real` or `Integer`
              dimensions),
            - a `(lower_bound, upper_bound, "prior")` tuple (for `Real`
              dimensions),
            - as a list of categories (for `Categorical` dimensions), or
            - an instance of a `Dimension` object (`Real`, `Integer` or
              `Categorical`).

            NOTE: The upper and lower bounds are inclusive for `Integer`
            dimensions.
        """
        self.dimensions = [check_dimension(dim) for dim in dimensions]

    def __eq__(self, other):
        return all([a == b for a, b in zip(self.dimensions, other.dimensions)])

    def __repr__(self):
        if len(self.dimensions) > 31:
            dims = self.dimensions[:15] + [_Ellipsis()] + self.dimensions[-15:]
        else:
            dims = self.dimensions
        return "Space([{}])".format(',\n       '.join(map(str, dims)))

    def __iter__(self):
        return iter(self.dimensions)

    @property
    def is_real(self):
        """
        Returns true if all dimensions are Real
        """
        return all([isinstance(dim, Real) for dim in self.dimensions])

    def rvs(self, n_samples=1, random_state=None):
        """Draw random samples.

        The samples are in the original space. They need to be transformed
        before being passed to a model or minimizer by `space.transform()`.

        Parameters
        ----------
        * `n_samples` [int, default=1]:
            Number of samples to be drawn from the space.

        * `random_state` [int, RandomState instance, or None (default)]:
            Set random state to something other than None for reproducible
            results.

        Returns
        -------
        * `points`: [list of lists, shape=(n_points, n_dims)]
           Points sampled from the space.
        """
        rng = check_random_state(random_state)

        # Draw
        columns = []

        for dim in self.dimensions:
            if sp_version < (0, 16):
                columns.append(dim.rvs(n_samples=n_samples))
            else:
                columns.append(dim.rvs(n_samples=n_samples, random_state=rng))

        # Transpose
        rows = []

        for i in range(n_samples):
            r = []
            for j in range(self.n_dims):
                r.append(columns[j][i])

            rows.append(r)

        return rows

    def transform(self, X):
        """Transform samples from the original space into a warped space.

        Note: this transformation is expected to be used to project samples
              into a suitable space for numerical optimization.

        Parameters
        ----------
        * `X` [list of lists, shape=(n_samples, n_dims)]:
            The samples to transform.

        Returns
        -------
        * `Xt` [array of floats, shape=(n_samples, transformed_n_dims)]
            The transformed samples.
        """
        # Pack by dimension
        columns = []
        for dim in self.dimensions:
            columns.append([])

        for i in range(len(X)):
            for j in range(self.n_dims):
                columns[j].append(X[i][j])

        # Transform
        for j in range(self.n_dims):
            columns[j] = self.dimensions[j].transform(columns[j])

        # Repack as an array
        Xt = np.hstack([np.asarray(c).reshape((len(X), -1)) for c in columns])

        return Xt

    def inverse_transform(self, Xt):
        """Inverse transform samples from the warped space back to the
           original space.

        Parameters
        ----------
        * `Xt` [array of floats, shape=(n_samples, transformed_n_dims)]:
            The samples to inverse transform.

        Returns
        -------
        * `X` [list of lists, shape=(n_samples, n_dims)]
            The original samples.
        """
        # Inverse transform
        columns = []
        start = 0

        for j in range(self.n_dims):
            dim = self.dimensions[j]
            offset = dim.transformed_size

            if offset == 1:
                columns.append(dim.inverse_transform(Xt[:, start]))
            else:
                columns.append(
                    dim.inverse_transform(Xt[:, start:start+offset]))

            start += offset

        # Transpose
        rows = []

        for i in range(len(Xt)):
            r = []
            for j in range(self.n_dims):
                r.append(columns[j][i])

            rows.append(r)

        return rows

    @property
    def n_dims(self):
        """The dimensionality of the original space."""
        return len(self.dimensions)

    @property
    def transformed_n_dims(self):
        """The dimensionality of the warped space."""
        return sum([dim.transformed_size for dim in self.dimensions])

    @property
    def bounds(self):
        """The dimension bounds, in the original space."""
        b = []

        for dim in self.dimensions:
            if dim.size == 1:
                b.append(dim.bounds)
            else:
                b.extend(dim.bounds)

        return b

    def __contains__(self, point):
        """Check that `point` is within the bounds of the space."""
        for component, dim in zip(point, self.dimensions):
            if component not in dim:
                return False
        return True

    @property
    def transformed_bounds(self):
        """The dimension bounds, in the warped space."""
        b = []

        for dim in self.dimensions:
            if dim.transformed_size == 1:
                b.append(dim.transformed_bounds)
            else:
                b.extend(dim.transformed_bounds)

        return b

    @property
    def is_categorical(self):
        """Space contains exclusively categorical dimensions"""
        return all([isinstance(dim, Categorical) for dim in self.dimensions])

    def distance(self, point_a, point_b):
        """Compute distance between two points in this space.

        Parameters
        ----------
        * `a` [array]
            First point.

        * `b` [array]
            Second point.
        """
        distance = 0.
        for a, b, dim in zip(point_a, point_b, self.dimensions):
            distance += dim.distance(a, b)

        return distance

Ancestors (in MRO)

Static methods

def __init__(

self, dimensions)

Initialize a search space from given specifications.

Parameters

  • dimensions [list, shape=(n_dims,)]: List of search space dimensions. Each search dimension can be defined either as

    • a (lower_bound, upper_bound) tuple (for Real or Integer dimensions),
    • a (lower_bound, upper_bound, "prior") tuple (for Real dimensions),
    • as a list of categories (for Categorical dimensions), or
    • an instance of a Dimension object (Real, Integer or Categorical).

    NOTE: The upper and lower bounds are inclusive for Integer dimensions.

def __init__(self, dimensions):
    """Initialize a search space from given specifications.
    Parameters
    ----------
    * `dimensions` [list, shape=(n_dims,)]:
        List of search space dimensions.
        Each search dimension can be defined either as
        - a `(lower_bound, upper_bound)` tuple (for `Real` or `Integer`
          dimensions),
        - a `(lower_bound, upper_bound, "prior")` tuple (for `Real`
          dimensions),
        - as a list of categories (for `Categorical` dimensions), or
        - an instance of a `Dimension` object (`Real`, `Integer` or
          `Categorical`).
        NOTE: The upper and lower bounds are inclusive for `Integer`
        dimensions.
    """
    self.dimensions = [check_dimension(dim) for dim in dimensions]

def distance(

self, point_a, point_b)

Compute distance between two points in this space.

Parameters

  • a [array] First point.

  • b [array] Second point.

def distance(self, point_a, point_b):
    """Compute distance between two points in this space.
    Parameters
    ----------
    * `a` [array]
        First point.
    * `b` [array]
        Second point.
    """
    distance = 0.
    for a, b, dim in zip(point_a, point_b, self.dimensions):
        distance += dim.distance(a, b)
    return distance

def inverse_transform(

self, Xt)

Inverse transform samples from the warped space back to the original space.

Parameters

  • Xt [array of floats, shape=(n_samples, transformed_n_dims)]: The samples to inverse transform.

Returns

  • X [list of lists, shape=(n_samples, n_dims)] The original samples.
def inverse_transform(self, Xt):
    """Inverse transform samples from the warped space back to the
       original space.
    Parameters
    ----------
    * `Xt` [array of floats, shape=(n_samples, transformed_n_dims)]:
        The samples to inverse transform.
    Returns
    -------
    * `X` [list of lists, shape=(n_samples, n_dims)]
        The original samples.
    """
    # Inverse transform
    columns = []
    start = 0
    for j in range(self.n_dims):
        dim = self.dimensions[j]
        offset = dim.transformed_size
        if offset == 1:
            columns.append(dim.inverse_transform(Xt[:, start]))
        else:
            columns.append(
                dim.inverse_transform(Xt[:, start:start+offset]))
        start += offset
    # Transpose
    rows = []
    for i in range(len(Xt)):
        r = []
        for j in range(self.n_dims):
            r.append(columns[j][i])
        rows.append(r)
    return rows

def rvs(

self, n_samples=1, random_state=None)

Draw random samples.

The samples are in the original space. They need to be transformed before being passed to a model or minimizer by space.transform().

Parameters

  • n_samples [int, default=1]: Number of samples to be drawn from the space.

  • random_state [int, RandomState instance, or None (default)]: Set random state to something other than None for reproducible results.

Returns

  • points: [list of lists, shape=(n_points, n_dims)] Points sampled from the space.
def rvs(self, n_samples=1, random_state=None):
    """Draw random samples.
    The samples are in the original space. They need to be transformed
    before being passed to a model or minimizer by `space.transform()`.
    Parameters
    ----------
    * `n_samples` [int, default=1]:
        Number of samples to be drawn from the space.
    * `random_state` [int, RandomState instance, or None (default)]:
        Set random state to something other than None for reproducible
        results.
    Returns
    -------
    * `points`: [list of lists, shape=(n_points, n_dims)]
       Points sampled from the space.
    """
    rng = check_random_state(random_state)
    # Draw
    columns = []
    for dim in self.dimensions:
        if sp_version < (0, 16):
            columns.append(dim.rvs(n_samples=n_samples))
        else:
            columns.append(dim.rvs(n_samples=n_samples, random_state=rng))
    # Transpose
    rows = []
    for i in range(n_samples):
        r = []
        for j in range(self.n_dims):
            r.append(columns[j][i])
        rows.append(r)
    return rows

def transform(

self, X)

Transform samples from the original space into a warped space.

Note: this transformation is expected to be used to project samples into a suitable space for numerical optimization.

Parameters

  • X [list of lists, shape=(n_samples, n_dims)]: The samples to transform.

Returns

  • Xt [array of floats, shape=(n_samples, transformed_n_dims)] The transformed samples.
def transform(self, X):
    """Transform samples from the original space into a warped space.
    Note: this transformation is expected to be used to project samples
          into a suitable space for numerical optimization.
    Parameters
    ----------
    * `X` [list of lists, shape=(n_samples, n_dims)]:
        The samples to transform.
    Returns
    -------
    * `Xt` [array of floats, shape=(n_samples, transformed_n_dims)]
        The transformed samples.
    """
    # Pack by dimension
    columns = []
    for dim in self.dimensions:
        columns.append([])
    for i in range(len(X)):
        for j in range(self.n_dims):
            columns[j].append(X[i][j])
    # Transform
    for j in range(self.n_dims):
        columns[j] = self.dimensions[j].transform(columns[j])
    # Repack as an array
    Xt = np.hstack([np.asarray(c).reshape((len(X), -1)) for c in columns])
    return Xt

Instance variables

var bounds

The dimension bounds, in the original space.

var dimensions

var is_categorical

Space contains exclusively categorical dimensions

var is_real

Returns true if all dimensions are Real

var n_dims

The dimensionality of the original space.

var transformed_bounds

The dimension bounds, in the warped space.

var transformed_n_dims

The dimensionality of the warped space.