Merge pull request #32 from numericalalgorithmsgroup/convex_constraints

Convex constraints

Author: lindonroberts

README.rst

@@ -28,8 +28,9 @@ More details about Py-BOBYQA and its enhancements over BOBYQA can be found in ou
 
 1. Coralia Cartis, Jan Fiala, Benjamin Marteau and Lindon Roberts, `Improving the Flexibility and Robustness of Model-Based Derivative-Free Optimization Solvers <https://doi.org/10.1145/3338517>`_, *ACM Transactions on Mathematical Software*, 45:3 (2019), pp. 32:1-32:41 [`arXiv preprint: 1804.00154 <https://arxiv.org/abs/1804.00154>`_] 
 2. Coralia Cartis, Lindon Roberts and Oliver Sheridan-Methven, `Escaping local minima with derivative-free methods: a numerical investigation <https://doi.org/10.1080/02331934.2021.1883015>`_, *Optimization*, 71:8 (2022), pp. 2343-2373. [`arXiv preprint: 1812.11343 <https://arxiv.org/abs/1812.11343>`_] 
+3. Lindon Roberts, `Model Construction for Convex-Constrained Derivative-Free Optimization <https://arxiv.org/abs/2403.14960>`_, *arXiv preprint arXiv:2403.14960* (2024).
 
-Please cite [1] when using Py-BOBYQA for local optimization, and [1,2] when using Py-BOBYQA's global optimization heuristic functionality. For reproducibility of all figures, please feel free to contact the authors.
+Please cite [1] when using Py-BOBYQA for local optimization, [1,2] when using Py-BOBYQA's global optimization heuristic functionality, and [1,3] if using the general convex constraints functionality.
 
 The original paper by Powell is: M. J. D. Powell, The BOBYQA algorithm for bound constrained optimization without derivatives, technical report DAMTP 2009/NA06, University of Cambridge (2009), and the original Fortran implementation is available `here <http://mat.uc.pt/~zhang/software.html>`_.
 
@@ -41,13 +42,13 @@ See manual.pdf or the `online manual <https://numericalalgorithmsgroup.github.io
 
 Citation
 --------
-If you use Py-BOBYQA in a paper, please cite:
+Full details of the Py-BOBYQA algorithm are given in our papers: 
 
-Cartis, C., Fiala, J., Marteau, B. and Roberts, L., Improving the Flexibility and Robustness of Model-Based Derivative-Free Optimization Solvers, *ACM Transactions on Mathematical Software*, 45:3 (2019), pp. 32:1-32:41.
+1. Coralia Cartis, Jan Fiala, Benjamin Marteau and Lindon Roberts, `Improving the Flexibility and Robustness of Model-Based Derivative-Free Optimization Solvers <https://doi.org/10.1145/3338517>`_, *ACM Transactions on Mathematical Software*, 45:3 (2019), pp. 32:1-32:41 [`preprint <https://arxiv.org/abs/1804.00154>`_] 
+2. Coralia Cartis, Lindon Roberts and Oliver Sheridan-Methven, `Escaping local minima with derivative-free methods: a numerical investigation <https://doi.org/10.1080/02331934.2021.1883015>`_, *Optimization*, 71:8 (2022), pp. 2343-2373. [`arXiv preprint: 1812.11343 <https://arxiv.org/abs/1812.11343>`_]
+3. Lindon Roberts, `Model Construction for Convex-Constrained Derivative-Free Optimization <https://arxiv.org/abs/2403.14960>`_, *arXiv preprint arXiv:2403.14960* (2024).
 
-If you use Py-BOBYQA's global optimization heuristic, please cite the above and also
-
-Cartis, C., Roberts, L. and Sheridan-Methven, O., Escaping local minima with derivative-free methods: a numerical investigation, Optimization, 71:8 (2022), pp. 2343-2373. 
+Please cite [1] when using Py-BOBYQA for local optimization, [1,2] when using Py-BOBYQA's global optimization heuristic functionality, and [1,3] if using the general convex constraints functionality.
 
 Requirements
 ------------
@@ -65,31 +66,17 @@ Additionally, the following python packages should be installed (these will be i
 
 Installation using pip
 ----------------------
-For easy installation, use `pip <http://www.pip-installer.org/>`_ as root:
-
- .. code-block:: bash
-
-    $ [sudo] pip install Py-BOBYQA
-
-or alternatively *easy_install*:
-
- .. code-block:: bash
-
-    $ [sudo] easy_install Py-BOBYQA
-
-If you do not have root privileges or you want to install Py-BOBYQA for your private use, you can use:
+For easy installation, use `pip <http://www.pip-installer.org/>`_:
 
  .. code-block:: bash
 
-    $ pip install --user Py-BOBYQA
-
-which will install Py-BOBYQA in your home directory.
+    $ pip install Py-BOBYQA
 
 Note that if an older install of Py-BOBYQA is present on your system you can use:
 
  .. code-block:: bash
 
-    $ [sudo] pip install --upgrade Py-BOBYQA
+    $ pip install --upgrade Py-BOBYQA
 
 to upgrade Py-BOBYQA to the latest version.
 
@@ -106,13 +93,7 @@ Py-BOBYQA is written in pure Python and requires no compilation. It can be insta
 
  .. code-block:: bash
 
-    $ [sudo] pip install .
-
-If you do not have root privileges or you want to install Py-BOBYQA for your private use, you can use:
-
- .. code-block:: bash
-
-    $ pip install --user .
+    $ pip install .
 
 instead.
 
@@ -121,7 +102,7 @@ To upgrade Py-BOBYQA to the latest version, navigate to the top-level directory
  .. code-block:: bash
 
     $ git pull
-    $ [sudo] pip install .  # with admin privileges
+    $ pip install .
 
 Testing
 -------
@@ -144,7 +125,7 @@ If Py-BOBYQA was installed using *pip* you can uninstall as follows:
 
  .. code-block:: bash
 
-    $ [sudo] pip uninstall Py-BOBYQA
+    $ pip uninstall Py-BOBYQA
 
 If Py-BOBYQA was installed manually you have to remove the installed files by hand (located in your python site-packages directory).
 

docs/advanced.rst

@@ -21,9 +21,9 @@ Logging and Output
 
 Initialization of Points
 ------------------------
-* :code:`init.random_initial_directions` - Build the initial interpolation set using random directions (as opposed to coordinate directions). Default is :code:`True`.
-* :code:`init.random_directions_make_orthogonal` - If building initial interpolation set with random directions, whether or not these should be orthogonalized. Default is :code:`True`.
-* :code:`init.run_in_parallel` - If using random directions, whether or not to ask for all :code:`objfun` to be evaluated at all points without any intermediate processing. Default is :code:`False`.
+* :code:`init.random_initial_directions` - Build the initial interpolation set using random directions (as opposed to coordinate directions). Default is :code:`True`. Not used if general convex constraints provided.
+* :code:`init.random_directions_make_orthogonal` - If building initial interpolation set with random directions, whether or not these should be orthogonalized. Default is :code:`True`. Not used if general convex constraints provided.
+* :code:`init.run_in_parallel` - If using random directions, whether or not to ask for all :code:`objfun` to be evaluated at all points without any intermediate processing. Default is :code:`False`. Not used if general convex constraints provided.
 
 Trust Region Management
 -----------------------
@@ -74,6 +74,12 @@ Multiple Restarts
 * :code:`restarts.auto_detect.min_chg_model_slope` - Minimum rate of increase of :math:`\log(\|g_k-g_{k-1}\|)` and :math:`\log(\|H_k-H_{k-1}\|_F)` over the past iterations to cause a restart. Default is 0.015.
 * :code:`restarts.auto_detect.min_correl` - Minimum correlation of the data sets :math:`(k, \log(\|g_k-g_{k-1}\|))` and :math:`(k, \log(\|H_k-H_{k-1}\|_F))` required to cause a restart. Default is 0.1.
 
+General Convex Constraints
+--------------------------
+* :code:`projections.dykstra.d_tol` - termination tolerance for Dykstra's algorithm for computing the projection onto the intersection of all convex constraints. Default is :math:`10^{-10}`.
+* :code:`projections.dykstra.max_iters` - maximum iterations of Dykstra's algorithm for computing the projection onto the intersection of all convex constraints. Default is 100.
+* :code:`projections.feasible_tol` - tolerance for checking feasibility of initial points with respect to general convex constraints. Default is :math:`10^{-10}`.
+* :code:`projections.pgd_tol` - termination tolerance for trust-region and geometry-improving subproblems. Default is :math:`10^{-8}`.
 
 References
 ----------

docs/build/doctrees/advanced.doctree

@@ -1,4 +1,4 @@
 # Sphinx build info version 1
 # This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
-config: 71b7b5e9dd47a27f370b73edfaf0d9fa
+config: 23c30e3852e30f05e3745555cbeb2562
 tags: 645f666f9bcd5a90fca523b33c5a78b7

docs/build/doctrees/diagnostic.doctree

@@ -21,9 +21,9 @@ Logging and Output
 
 Initialization of Points
 ------------------------
-* :code:`init.random_initial_directions` - Build the initial interpolation set using random directions (as opposed to coordinate directions). Default is :code:`True`.
-* :code:`init.random_directions_make_orthogonal` - If building initial interpolation set with random directions, whether or not these should be orthogonalized. Default is :code:`True`.
-* :code:`init.run_in_parallel` - If using random directions, whether or not to ask for all :code:`objfun` to be evaluated at all points without any intermediate processing. Default is :code:`False`.
+* :code:`init.random_initial_directions` - Build the initial interpolation set using random directions (as opposed to coordinate directions). Default is :code:`True`. Not used if general convex constraints provided.
+* :code:`init.random_directions_make_orthogonal` - If building initial interpolation set with random directions, whether or not these should be orthogonalized. Default is :code:`True`. Not used if general convex constraints provided.
+* :code:`init.run_in_parallel` - If using random directions, whether or not to ask for all :code:`objfun` to be evaluated at all points without any intermediate processing. Default is :code:`False`. Not used if general convex constraints provided.
 
 Trust Region Management
 -----------------------
@@ -74,6 +74,12 @@ Multiple Restarts
 * :code:`restarts.auto_detect.min_chg_model_slope` - Minimum rate of increase of :math:`\log(\|g_k-g_{k-1}\|)` and :math:`\log(\|H_k-H_{k-1}\|_F)` over the past iterations to cause a restart. Default is 0.015.
 * :code:`restarts.auto_detect.min_correl` - Minimum correlation of the data sets :math:`(k, \log(\|g_k-g_{k-1}\|))` and :math:`(k, \log(\|H_k-H_{k-1}\|_F))` required to cause a restart. Default is 0.1.
 
+General Convex Constraints
+--------------------------
+* :code:`projections.dykstra.d_tol` - termination tolerance for Dykstra's algorithm for computing the projection onto the intersection of all convex constraints. Default is :math:`10^{-10}`.
+* :code:`projections.dykstra.max_iters` - maximum iterations of Dykstra's algorithm for computing the projection onto the intersection of all convex constraints. Default is 100.
+* :code:`projections.feasible_tol` - tolerance for checking feasibility of initial points with respect to general convex constraints. Default is :math:`10^{-10}`.
+* :code:`projections.pgd_tol` - termination tolerance for trust-region and geometry-improving subproblems. Default is :math:`10^{-8}`.
 
 References
 ----------

docs/build/doctrees/environment.pickle

@@ -48,6 +48,10 @@ Version 1.4 (16 May 2023)
 * Bugfix: reset slow iteration counter when doing soft restarts
 
 Version 1.4.1 (11 Apr 2024)
--------------------------
+---------------------------
 * Migrate package setup to pyproject.toml (required for Python version 3.12)
 * Drop support for Python 2.7 and <=3.7 due to new setup process
+
+Version 1.5.0 (16 Sep 2024)
+---------------------------
+* Added support for general convex constraints

docs/build/doctrees/history.doctree

@@ -12,23 +12,26 @@ Py-BOBYQA: Derivative-Free Optimizer for Bound-Constrained Minimization
 
 **Author:** `Lindon Roberts <lindon.roberts@sydney.edu.au>`_
 
-Py-BOBYQA is a flexible package for finding local solutions to nonlinear, nonconvex minimization problems (with optional bound constraints), without requiring any derivatives of the objective. Py-BOBYQA is a Python implementation of the `BOBYQA <http://mat.uc.pt/~zhang/software.html#powell_software>`_ solver by Powell (documentation `here <http://www.damtp.cam.ac.uk/user/na/NA_papers/NA2009_06.pdf>`_). It is particularly useful when evaluations of the objective function are expensive and/or noisy.
+Py-BOBYQA is a flexible package for finding local solutions to nonlinear, nonconvex minimization problems (with optional bound and other convex constraints), without requiring any derivatives of the objective. Py-BOBYQA is a Python implementation of the `BOBYQA <http://mat.uc.pt/~zhang/software.html#powell_software>`_ solver by Powell (documentation `here <http://www.damtp.cam.ac.uk/user/na/NA_papers/NA2009_06.pdf>`_). It is particularly useful when evaluations of the objective function are expensive and/or noisy.
 
 That is, Py-BOBYQA solves
 
 .. math::
 
    \min_{x\in\mathbb{R}^n}  &\quad  f(x)\\
-   \text{s.t.} &\quad  a \leq x \leq b
+   \text{s.t.} &\quad  a \leq x \leq b \\
+   &\quad x \in C := C_1 \cap \cdots \cap C_n, \quad \text{all $C_i$ convex}
 
-The upper and lower bounds on the variables are non-relaxable (i.e. Py-BOBYQA will never ask to evaluate a point outside the bounds).
+If provided, the constraints the variables are non-relaxable (i.e. Py-BOBYQA will never ask to evaluate a point outside the bounds),
+although the general :math:`x \in C` constraint may be slightly violated from rounding errors.
 
 Full details of the Py-BOBYQA algorithm are given in our papers: 
 
 1. Coralia Cartis, Jan Fiala, Benjamin Marteau and Lindon Roberts, `Improving the Flexibility and Robustness of Model-Based Derivative-Free Optimization Solvers <https://doi.org/10.1145/3338517>`_, *ACM Transactions on Mathematical Software*, 45:3 (2019), pp. 32:1-32:41 [`preprint <https://arxiv.org/abs/1804.00154>`_] 
-2. Coralia Cartis, Lindon Roberts and Oliver Sheridan-Methven, `Escaping local minima with derivative-free methods: a numerical investigation <https://doi.org/10.1080/02331934.2021.1883015>`_, *Optimization*, 71:8 (2022), pp. 2343-2373. [`arXiv preprint: 1812.11343 <https://arxiv.org/abs/1812.11343>`_] 
+2. Coralia Cartis, Lindon Roberts and Oliver Sheridan-Methven, `Escaping local minima with derivative-free methods: a numerical investigation <https://doi.org/10.1080/02331934.2021.1883015>`_, *Optimization*, 71:8 (2022), pp. 2343-2373. [`arXiv preprint: 1812.11343 <https://arxiv.org/abs/1812.11343>`_]
+3. Lindon Roberts, `Model Construction for Convex-Constrained Derivative-Free Optimization <https://arxiv.org/abs/2403.14960>`_, *arXiv preprint arXiv:2403.14960* (2024).
 
-Please cite [1] when using Py-BOBYQA for local optimization, and [1,2] when using Py-BOBYQA's global optimization heuristic functionality.
+Please cite [1] when using Py-BOBYQA for local optimization, [1,2] when using Py-BOBYQA's global optimization heuristic functionality, and [1,3] if using the general convex constraints :math:`x \in C` functionality.
 
 If you are interested in solving least-squares minimization problems, you may wish to try `DFO-LS <https://github.com/numericalalgorithmsgroup/dfols>`_, which has the same features as Py-BOBYQA (plus some more), and exploits the least-squares problem structure, so performs better on such problems.
 
@@ -49,5 +52,6 @@ Py-BOBYQA is released under the GNU General Public License. Please `contact NAG
 
 Acknowledgements
 ----------------
-This software was developed under the supervision of `Coralia Cartis <https://www.maths.ox.ac.uk/people/coralia.cartis>`_, and was supported by the EPSRC Centre For Doctoral Training in `Industrially Focused Mathematical Modelling <https://www.maths.ox.ac.uk/study-here/postgraduate-study/industrially-focused-mathematical-modelling-epsrc-cdt>`_ (EP/L015803/1) in collaboration with the `Numerical Algorithms Group <http://www.nag.com/>`_.
+This software was initially developed under the supervision of `Coralia Cartis <https://www.maths.ox.ac.uk/people/coralia.cartis>`_, and was supported by the EPSRC Centre For Doctoral Training in `Industrially Focused Mathematical Modelling <https://www.maths.ox.ac.uk/study-here/postgraduate-study/industrially-focused-mathematical-modelling-epsrc-cdt>`_ (EP/L015803/1) in collaboration with the `Numerical Algorithms Group <http://www.nag.com/>`_.
+Development of Py-BOBYQA has also been supported by the Australian Research Council (DE240100006).
 

docs/build/doctrees/index.doctree

@@ -3,12 +3,13 @@ Overview
 
 When to use Py-BOBYQA
 ---------------------
-Py-BOBYQA is designed to solve the nonlinear least-squares minimization problem (with optional bound constraints)
+Py-BOBYQA is designed to solve the nonlinear least-squares minimization problem (with optional bound and general convex constraints)
 
 .. math::
 
    \min_{x\in\mathbb{R}^n}  &\quad  f(x)\\
-   \text{s.t.} &\quad  a \leq x \leq b
+   \text{s.t.} &\quad  a \leq x \leq b\\
+   &\quad x \in C := C_1 \cap \cdots \cap C_n, \quad \text{all $C_i$ convex}
 
 We call :math:`f(x)` the objective function.
 

docs/build/doctrees/info.doctree

@@ -17,25 +17,17 @@ Additionally, the following python packages should be installed (these will be i
 
 Installation using pip
 ----------------------
-For easy installation, use `pip <http://www.pip-installer.org/>`_ as root:
+For easy installation, use `pip <http://www.pip-installer.org/>`_:
 
  .. code-block:: bash
 
-    $ [sudo] pip install Py-BOBYQA
-
-If you do not have root privileges or you want to install Py-BOBYQA for your private use, you can use:
-
- .. code-block:: bash
-
-    $ pip install --user Py-BOBYQA
-      
-which will install Py-BOBYQA in your home directory.
+    $ pip install Py-BOBYQA
 
 Note that if an older install of Py-BOBYQA is present on your system you can use:
 
  .. code-block:: bash
 
-    $ [sudo] pip install --upgrade Py-BOBYQA
+    $ pip install --upgrade Py-BOBYQA
       
 to upgrade Py-BOBYQA to the latest version.
 
@@ -52,22 +44,14 @@ Py-BOBYQA is written in pure Python and requires no compilation. It can be insta
 
  .. code-block:: bash
 
-    $ [sudo] pip install .
-
-If you do not have root privileges or you want to install Py-BOBYQA for your private use, you can use:
-
- .. code-block:: bash
-
-    $ pip install --user .
-    
-instead.
+    $ pip install .
 
 To upgrade Py-BOBYQA to the latest version, navigate to the top-level directory (i.e. the one containing :code:`setup.py`) and rerun the installation using :code:`pip`, as above:
 
  .. code-block:: bash
 
     $ git pull
-    $ [sudo] pip install .  # with admin privileges
+    $ pip install .
 
 Testing
 -------
@@ -84,7 +68,7 @@ If Py-BOBYQA was installed using `pip <http://www.pip-installer.org/>`_ you can
 
  .. code-block:: bash
 
-    $ [sudo] pip uninstall Py-BOBYQA
+    $ pip uninstall Py-BOBYQA
 
 If Py-BOBYQA was installed manually you have to remove the installed files by hand (located in your python site-packages directory).