Skip to content
This repository was archived by the owner on Feb 1, 2026. It is now read-only.
This repository was archived by the owner on Feb 1, 2026. It is now read-only.

Leave-one-out method doesn't handle test envs correctly #164

@qwer1304

Description

@qwer1304

Two serious bugs in LOO model selection method:

  1. On line 138 test_records = get_test_records(records) is called, which returns None when number of test environments is not 1 which is incompatible with the usage as can be seen on line 149 where number of test envs is required to be 2.
  2. More serious, it assumes on line 142 that the test environment is always the first one in test_envs entry, which is incorrect: the method is called once for the first entry and second time for the second one. Problem is that as it currently stands no indication which environment of the pair is the test one is unavailable: only the test record with test_envs field is.

I have a patches availabe, if you want, so let me know if you want a pull.

PS It's also very inconvenient that it assumes that all environments are provided as validation ones on line 153. But that's a different issue subject of extending sweep to support explicitly specified test environment, which I have patches for too. So, let me know if you want a pull with them too.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions