.. only:: html
.. note::
:class: sphx-glr-download-link-note
Click :ref:`here ` to download the full example code or to run this example in your browser via Binder
.. rst-class:: sphx-glr-example-title
.. _sphx_glr_auto_examples_exploration-vs-exploitation.py:
===========================
Exploration vs exploitation
===========================
Sigurd Carlen, September 2019.
Reformatted by Holger Nahrstaedt 2020
.. currentmodule:: skopt
We can control how much the acqusition function favors exploration and
exploitation by tweaking the two parameters kappa and xi. Higher values
means more exploration and less exploitation and vice versa with low values.
kappa is only used if acq_func is set to "LCB". xi is used when acq_func is
"EI" or "PI". By default the acqusition function is set to "gp_hedge" which
chooses the best of these three. Therefore I recommend not using gp_hedge
when tweaking exploration/exploitation, but instead choosing "LCB",
"EI" or "PI".
The way to pass kappa and xi to the optimizer is to use the named argument
"acq_func_kwargs". This is a dict of extra arguments for the aqcuisition
function.
If you want opt.ask() to give a new acquisition value immediately after
tweaking kappa or xi call opt.update_next(). This ensures that the next
value is updated with the new acquisition parameters.
This example uses :class:`plots.plot_gaussian_process` which is available
since version 0.8.
.. code-block:: default
print(__doc__)
import numpy as np
np.random.seed(1234)
import matplotlib.pyplot as plt
from skopt.learning import ExtraTreesRegressor
from skopt import Optimizer
from skopt.plots import plot_gaussian_process
Toy example
-----------
First we define our objective like in the ask-and-tell example notebook and
define a plotting function. We do however only use on initial random point.
All points after the first one is therefore chosen by the acquisition
function.
.. code-block:: default
noise_level = 0.1
# Our 1D toy problem, this is the function we are trying to
# minimize
def objective(x, noise_level=noise_level):
return np.sin(5 * x[0]) * (1 - np.tanh(x[0] ** 2)) +\
np.random.randn() * noise_level
def objective_wo_noise(x):
return objective(x, noise_level=0)
.. code-block:: default
opt = Optimizer([(-2.0, 2.0)], "GP", n_initial_points=3,
acq_optimizer="sampling")
Plotting parameters
.. code-block:: default
plot_args = {"objective": objective_wo_noise,
"noise_level": noise_level, "show_legend": True,
"show_title": True, "show_next_point": False,
"show_acq_func": True}
We run a an optimization loop with standard settings
.. code-block:: default
for i in range(30):
next_x = opt.ask()
f_val = objective(next_x)
opt.tell(next_x, f_val)
# The same output could be created with opt.run(objective, n_iter=30)
_ = plot_gaussian_process(opt.get_result(), **plot_args)
.. image:: /auto_examples/images/sphx_glr_exploration-vs-exploitation_001.png
:alt: x* = -0.2913, f(x*) = -1.0409
:class: sphx-glr-single-img
We see that some minima is found and "exploited"
Now lets try to set kappa and xi using'to other values and
pass it to the optimizer:
.. code-block:: default
acq_func_kwargs = {"xi": 10000, "kappa": 10000}
.. code-block:: default
opt = Optimizer([(-2.0, 2.0)], "GP", n_initial_points=3,
acq_optimizer="sampling",
acq_func_kwargs=acq_func_kwargs)
.. code-block:: default
opt.run(objective, n_iter=20)
_ = plot_gaussian_process(opt.get_result(), **plot_args)
.. image:: /auto_examples/images/sphx_glr_exploration-vs-exploitation_002.png
:alt: x* = -0.3083, f(x*) = -0.7990
:class: sphx-glr-single-img
We see that the points are more random now.
This works both for kappa when using acq_func="LCB":
.. code-block:: default
opt = Optimizer([(-2.0, 2.0)], "GP", n_initial_points=3,
acq_func="LCB", acq_optimizer="sampling",
acq_func_kwargs=acq_func_kwargs)
.. code-block:: default
opt.run(objective, n_iter=20)
_ = plot_gaussian_process(opt.get_result(), **plot_args)
.. image:: /auto_examples/images/sphx_glr_exploration-vs-exploitation_003.png
:alt: x* = -0.1829, f(x*) = -0.8271
:class: sphx-glr-single-img
And for xi when using acq_func="EI": or acq_func="PI":
.. code-block:: default
opt = Optimizer([(-2.0, 2.0)], "GP", n_initial_points=3,
acq_func="PI", acq_optimizer="sampling",
acq_func_kwargs=acq_func_kwargs)
.. code-block:: default
opt.run(objective, n_iter=20)
_ = plot_gaussian_process(opt.get_result(), **plot_args)
.. image:: /auto_examples/images/sphx_glr_exploration-vs-exploitation_004.png
:alt: x* = -0.3877, f(x*) = -0.8487
:class: sphx-glr-single-img
We can also favor exploitaton:
.. code-block:: default
acq_func_kwargs = {"xi": 0.000001, "kappa": 0.001}
.. code-block:: default
opt = Optimizer([(-2.0, 2.0)], "GP", n_initial_points=3,
acq_func="LCB", acq_optimizer="sampling",
acq_func_kwargs=acq_func_kwargs)
.. code-block:: default
opt.run(objective, n_iter=20)
_ = plot_gaussian_process(opt.get_result(), **plot_args)
.. image:: /auto_examples/images/sphx_glr_exploration-vs-exploitation_005.png
:alt: x* = -0.4020, f(x*) = -0.9123
:class: sphx-glr-single-img
.. code-block:: default
opt = Optimizer([(-2.0, 2.0)], "GP", n_initial_points=3,
acq_func="EI", acq_optimizer="sampling",
acq_func_kwargs=acq_func_kwargs)
.. code-block:: default
opt.run(objective, n_iter=20)
_ = plot_gaussian_process(opt.get_result(), **plot_args)
.. image:: /auto_examples/images/sphx_glr_exploration-vs-exploitation_006.png
:alt: x* = 0.8760, f(x*) = -0.4154
:class: sphx-glr-single-img
.. code-block:: default
opt = Optimizer([(-2.0, 2.0)], "GP", n_initial_points=3,
acq_func="PI", acq_optimizer="sampling",
acq_func_kwargs=acq_func_kwargs)
.. code-block:: default
opt.run(objective, n_iter=20)
_ = plot_gaussian_process(opt.get_result(), **plot_args)
.. image:: /auto_examples/images/sphx_glr_exploration-vs-exploitation_007.png
:alt: x* = -0.2782, f(x*) = -1.1051
:class: sphx-glr-single-img
Note that negative values does not work with the "PI"-acquisition function
but works with "EI":
.. code-block:: default
acq_func_kwargs = {"xi": -1000000000000}
.. code-block:: default
opt = Optimizer([(-2.0, 2.0)], "GP", n_initial_points=3,
acq_func="PI", acq_optimizer="sampling",
acq_func_kwargs=acq_func_kwargs)
.. code-block:: default
opt.run(objective, n_iter=20)
_ = plot_gaussian_process(opt.get_result(), **plot_args)
.. image:: /auto_examples/images/sphx_glr_exploration-vs-exploitation_008.png
:alt: x* = 0.8093, f(x*) = -0.4621
:class: sphx-glr-single-img
.. code-block:: default
opt = Optimizer([(-2.0, 2.0)], "GP", n_initial_points=3,
acq_func="EI", acq_optimizer="sampling",
acq_func_kwargs=acq_func_kwargs)
.. code-block:: default
opt.run(objective, n_iter=20)
_ = plot_gaussian_process(opt.get_result(), **plot_args)
.. image:: /auto_examples/images/sphx_glr_exploration-vs-exploitation_009.png
:alt: x* = 0.8341, f(x*) = -0.4331
:class: sphx-glr-single-img
Changing kappa and xi on the go
-------------------------------
If we want to change kappa or ki at any point during our optimization
process we just replace opt.acq_func_kwargs. Remember to call
`opt.update_next()` after the change, in order for next point to be
recalculated.
.. code-block:: default
acq_func_kwargs = {"kappa": 0}
.. code-block:: default
opt = Optimizer([(-2.0, 2.0)], "GP", n_initial_points=3,
acq_func="LCB", acq_optimizer="sampling",
acq_func_kwargs=acq_func_kwargs)
.. code-block:: default
opt.acq_func_kwargs
.. rst-class:: sphx-glr-script-out
Out:
.. code-block:: none
{'kappa': 0}
.. code-block:: default
opt.run(objective, n_iter=20)
_ = plot_gaussian_process(opt.get_result(), **plot_args)
.. image:: /auto_examples/images/sphx_glr_exploration-vs-exploitation_010.png
:alt: x* = -0.2982, f(x*) = -1.0610
:class: sphx-glr-single-img
.. code-block:: default
acq_func_kwargs = {"kappa": 100000}
.. code-block:: default
opt.acq_func_kwargs = acq_func_kwargs
opt.update_next()
.. code-block:: default
opt.run(objective, n_iter=20)
_ = plot_gaussian_process(opt.get_result(), **plot_args)
.. image:: /auto_examples/images/sphx_glr_exploration-vs-exploitation_011.png
:alt: x* = -0.2982, f(x*) = -1.0610
:class: sphx-glr-single-img
.. rst-class:: sphx-glr-timing
**Total running time of the script:** ( 0 minutes 29.821 seconds)
**Estimated memory usage:** 8 MB
.. _sphx_glr_download_auto_examples_exploration-vs-exploitation.py:
.. only :: html
.. container:: sphx-glr-footer
:class: sphx-glr-footer-example
.. container:: binder-badge
.. image:: /../../miniconda/envs/testenv/lib/python3.8/site-packages/sphinx_gallery/_static/binder_badge_logo.svg
:target: https://mybinder.org/v2/gh/scikit-optimize/scikit-optimize/master?urlpath=lab/tree/notebooks/auto_examples/exploration-vs-exploitation.ipynb
:width: 150 px
.. container:: sphx-glr-download sphx-glr-download-python
:download:`Download Python source code: exploration-vs-exploitation.py `
.. container:: sphx-glr-download sphx-glr-download-jupyter
:download:`Download Jupyter notebook: exploration-vs-exploitation.ipynb `
.. only:: html
.. rst-class:: sphx-glr-signature
`Gallery generated by Sphinx-Gallery `_