Bayesian Optimization

This repo contains my work with Bayesian optimization and related topics. The primary goal of this repo is to develop and curate a set of resources that myself and others can use to better understand (and utilize) Bayesian optimization.

Note

If you find any issues/typos or have any suggestions, feel free to raise an issue and let me know.

📁 docs

This directory contains notes/tutorials on Bayesian optimization and related topics.

• Introduction to Bayesian Optimization
• Bayesian Decision Theory
• Gaussian Processes
• Covariance Functions and Kernels
• Model Evaluation and Selection
• Utility Functions
• Acquisition Functions
• GP Regression
• GP Classification

📁 examples

This directory contains notebooks exploring various examples of Bayesian optimization and its applications.

• BoTorch Tutorials
  This notebook contains tutorials from BoTorch's documentation and tutorials.
• Introduction to GPyTorch and GAUCHE
  This notebook synthesizes information from GPyTorch and GAUCHE's documentation regarding Gaussian processes for machine learning and how to apply them to irregular-structured input representations (i.e., molecular, graph, etc.).

📁 src

This directory contains from-scratch implementations of Bayesian optimization methods and the methods of related topics.

Use the commands below to create a new Conda environment with all of the necessary dependencies:

conda env create -f bo_env.yml
conda activate bo-env

Note that the commands above should also be used to create an environment to run the notebooks in the examples directory.

• branin.py
  This Python script compares the performance of analytic and monte-carlo acquisition functions implemented in BoTorch on the Branin function embedded in higher dimensions. The performance of each acquisition function is automatically tracked using Weights & Biases. To run the script, use this command: python3 branin.py (after activating the bo-env environment).

📚 References

Below is a list of reference texts, papers, and other sources on Bayesian optimization and related topics. The BibTeX entries can be found in the bibliography.bib file.

• Bayesian Optimization by Roman Garnett (2023)
• Bayesian Optimization: Theory and Practice Using Python by Peng Liu (2023)
• Gaussian Processes for Machine Learning by Rasmussen & Williams (2019)