Add Model wrapper that uses Gaussian processes to evaluate

[MichaelClerx]

Possibly based on GPy or GPflow

[martinjrobins]

I can see three options here:

1. Use GPflow or GPy (I'm leaning towards GPy, since it doesn't have tensorflow as a dependency). This is the easiest, but adds another dependency. All that probably would be required is writing an example showing how to use GPy/GPflow to emulate a pints model
2. Write code to implement a GP model in pints. Myself and @sanmitraghosh are planning to do this for a paper, but the code would be (partially) in C++, so would require adding C++ compilation functionality back into pints
3. Same as 2, but split this code into a separate repository in the pints organisation, then the main pints code can add this as a dependency. This might be a pain to develop both code-bases in parallel....

I'm leaning towards (1) for the short term, leading towards (2) in the longer term if the paper with @sanmitraghosh shows some good result for our approach. Any other thoughts?

[martinjrobins]

I've got some good results at using Aboria to solve for the GP, so I'm going to experiment with fitting a GP using this.

[sanmitraghosh]

@martinjrobins Great news !

[martinjrobins]

I'll let you know how it goes. I'm going to stick to the matern 3/2 kernel for the moment, as that seems to be the easiest to solve (Gaussian I haven't managed to do efficiently for dimensions higher than 3)

[sanmitraghosh]

I think you need not worry about Gaussian kernel, matern is much more useful in practical applications.

[martinjrobins]

couple of implementation points, I'm planning to:

1. implement the Adam stochastic gradient descent algorithm in pints, to use for fitting the GP (might be related to Investigate stochastic gradient descent as a type of approximate inference algorithm #764)
2. the GP itself will be derived from pints.log_pdf. Be nice to use it to fit to a posterior from mcmc samples, then use that as a prior for subsequent inference :)
3. for the GP fitting (using Adam), I don't want to evaluate the function to be minimised, only the sensitivities, so I'm implementing Adam so you don't need to evaluate the function. This might be a bit unusual in the pints framework (e.g I'm using abs(sensitivities) instead of fbest).
4. also for the GP fitting, the gradients of the function are noise-corrupted, so the stopping criteria shouldn't be based on the minimum abs(gradient) achieved, but when the change in hyper-parameters have converged