Interfacing to the prior engine

[jgomezdans]

Main idea is to have a wrapper around the prior engine that gathers all the relevant data files for different parameters and relevant dates in the temporal grid, and stores them. The pseudocode for this would be similar to the one that follows

class Prior(object):
    """A class to wrap the MULTIPLY prior engine to the inference engine"""
    def __init__ ( self, prior_engine_config_file):
        """This method sets up the MULTIPLY prior engine using a YAML (e.g.)
        configuration file. It also sets up a mean dictionary that will be index by 
        parameter name, and then by date (each date storing the mean and 
        standard deviation filenames for the given parameter and date)
        """
        self.prior_engine = PriorEngine(prior_engine_config_file)
        self.prior_info_mean = {}
        self.prior_info_mean = {}

    def _get_prior_date(self, parameters, date):
        """This method gets the prior information for a given set of parameters and 
        a particular date. """
        for param in parameters:
            mean, std_dev = self.prior_engine(param, date)
            # mean and std_dev are the full path to the global VRT file
            self.prior_info_mean[param][date] = mean
            self.prior_info_std[param][date] = std_dev

    def get_all_priors(self, parameters, time_grid):
        """A method to go through all dates and query the prior information.
        Takes a list of parameters, and a time grid (some iterator, list possibly)
        and stores the prior in self.prior_info{mean|std}.
        """
        for timestep in time_grid:
            self._get_prior_date(parameters, timestep)

    def process_prior(self, parameters, timestep, state_grid, inv_cov=True):
        """This method reads in the prior parameters for a given `timestep`, 
        processes them to match the state grid, and returns a mean state
        vector and an (inverse) sparse covariance matrix.
        """
        pass

[TonioF]

José, I understood that not the prior engine but the various .vrt-files should be passed to the Inference Engine and that this class would be used to extract the priors as required by the inference engine. From your introduction I am not sure what you mean: Whether it encapsulates the prior engine OR the files - put another way: Whether the prior engine shall be executed within the inference engine, i.e., within the time loop, or beforehand.

The nice thing is that in both cases the interface stays the same so this decision is not (yet) crucial.

[jgomezdans]

I think you can work both ways: you either "preload" the files before you pass this to KaFKA (your "beforehand" case), or you have KaFKA call the Prior engine and get the file names "on demand" as you are processing (within the loop). Both user cases are needed in the long run: the first one is the one we'd use for the workshop, the other one would allow one to do some more sophisticated prior calculations. The second use case we shouldn't worry about for now.

The PriorEngine is shorthand for getting some information from the prior engine, which will be a bunch of file locations indexed by parameter and time. I think you need this (in this form or some other) as there is a strong need for the user to be able to modify the prior that is being used. For the workshop, maybe ?two? prior datasets are given (and these could just be different sets of VRT files). (@RaT0M this means that you can have your CCI SM prior and the "Munich prior" made up of the ground truth, as we discussed yesterday)

The process_prior method (empty here) merely takes the relevant prior files (they can be VRT or something else, this is flexible) for a given time step and set of parameters and turn them into something that KaFKA can then use. If the class has been initialised before, it just means to search the relevant files in the self.prior_info_mean and self.prior_info_std dictionaries. If not (or if you have a dynamic prior), it can call the _get_prior_date method. I think the first mode of operation is the one to consider for now, but the other one will need to be there too.

[tramsauer]

With the interface like

.. you have KaFKA call the Prior engine and get the file names "on demand" as you are processing (within the loop).

being mentioned several times in the workshop, this is what we have set up now. So the PriorEngine would be called for each time step with a list of variables (which have to be named equally across modules; e.g. lai not LAI ..) to be inferred and a date (see interfaces) and return a dictionary with variable: filename structure

At the moment it's more variable: prior_type: filename like in:

{'sm': {'climatology': 'ESA_CCI_SM_CLIM_3.geotiff'}, 'lai': {'database': 'Priors_lai_060_global.vrt'}, 'cab': {'database': 'Priors_cab_060_global.vrt'}}
for
get_mean_state_vector(datestr="2017-03-01", variables=['sm','lai', 'cab']) .

Prior types can be selected via commenting out/leaving in the lines in the config.
Prior information of other types are going to be added in the dictionary beneath (?) the variable name (e.g. {sm: {clim: 'bla.tif', recent:'blabla.tif'}} where the ptype names could also be left out...).

Can this be used or should we rewrite (again)?

The actual master branch should work for my part. @JorisTimmermans also sent some demonstration files making his part work which I'll add to the repo.

[TonioF]

@jgomezdans : Yes, I figured both solutions make sense, so I am now developing an approach which can work both ways.

@RaT0M : The interface as you have it now is what I expected. I will let you know when I run into a problem.

[jgomezdans]

@TonioF Any chance of getting a dummy version of the prior class so I can start debugging? Just empty methods will do. Thanks!

[TonioF]

I'll try to let you have one this afternoon.

[jgomezdans]

@TonioF Thanks! And sorry for the spam earlier on!!! 😬

[TonioF]

@jgomezdans : I have added a class InferencePrior to multiply-org/inference-engine . When you import this, you can create a dummy prior with InferencePrior(, , True) .
I did not implement get_prior_date or get_all_priors from the interface above, as I do not think they were necessary.
I extended the interface for process prior to
`
@AbstractMethod

def process_prior(self, parameters: List[str], time: Union[str, datetime], state_grid: np.array, inv_cov: bool = True) -> List[np.array]:
"""
This method retrieves the requested parameters for the given time and region. For each parameter,
it returns a mean state vector and sparse covariance matrix which might be inverse.
:param parameters: List of parameters for which prior information is requested
:param time: The time for which information is requested.
:param state_grid: A state grid to indicate the region for which information must be retrieved.
Must be given in the form of a 2D numpy array.
:param inv_cov: Whether the covariance matrix shall be inverse.
:return:
"""
`

So, this method expects a 2D state boolean mask matching the output grid. It will return a 1D array with parameter values for each valid pixel (i.e., of length num_pixels * num_parameters) and a 2D array with a matrix for each pixel (of shape (num_pixels * num_parameters) * (num_pixels * num_parameters)). I designed this after the input passed onto the LinearKalman in kafka_test.py . Is this what you expect?

[TonioF]

Okay, I have added the missing inference priors now. You have to pass them a gdal dataset of the grid you want so the prior files can be reprojected accordingly.

The inference priors now return two numpy arrays: One with the mean state vectors (in the state as above) and another one with covariance values (in the SAME state as the mean state vector). Also note that first the parameter values for one pixel are listed, then for the next one, and so on and so on.