Adds model verification testing

[PulkitMishra]

WIP - creates a model validation testing system for the Opensource models

[abishekvashok]

Put in taint models path as well pointing to the repositories stubs? This helps us make the action verify newer models as well.

[abishekvashok]

What output do you get? Can we have a GitHub action that runs this script? We could mimic the pysa action and run this script instead of running the deliberately vulnerable flask app's script.

[PulkitMishra]

yes having a github action is the next step @abishekvashok but first need to find a way to make pyre_configuration.local play well with pyre_configuration at the root folder

[abishekvashok]

If doesn't play well, feel free to delete the .pyre_configuration file at the root of the dir and create a file at the your folder. For an action it should get you by :)

[PulkitMishra]

yeah that's possible and is my last resort. Some clever use of tempfile seems cleaner though

[r0rshark]

@PulkitMishra To avoid the issue with the .pyre_configuration located in the root folder the trick is to create a temporary directory in the global temporary folder (with something like tempfile.gettempdir() ) and create a folder there in which we can store the .pyre_configuration file. We can then pass to the PyreConnection the path of this folder so it will not conflict with the global .pyre_configuration.
We can create a function which does this in https://github.com/facebook/pyre-check/blob/main/api/connection.py and invoke it passing tempfile.gettempdir() as root directory and the path to the virtualenvironment we created as a source dir. Example function:

def create_temporary_pyre_connection(
    root_directory: pathlib.Path
    targets: Optional[List[str]] = None,
    source_directories: Optional[List[str]] = None,
) -> PyreConnection:
  # We can compute the sha1 as folder name to avoid creating a new directory every time and use the same if the source 
  directories and targets do not change
  root_directory.mkdir(exist_ok=True)
  pyre_configuration_directory = (
          root_directory
          / hashlib.sha1(
              ",".join((targets or []) + (source_directories or [])).encode()
          ).hexdigest()
      ) 
    pyre_configuration_directory.mkdir(exist_ok=True)
    with open(
        str(pyre_configuration_directory / ".pyre_configuration.local"), "w"
    ) as configuration:
        configuration_contents: Dict[str, Any] = {}
        if targets:
            configuration_contents["targets"] = targets
        if source_directories:
            configuration_contents["source_directories"] = source_directories
        json.dump(configuration_contents, configuration, indent=2)
    return PyreConnection(pyre_configuration_directory)