Skip to content

Comments

Spec tests#246

Closed
Zacholme7 wants to merge 5 commits intosigp:unstablefrom
Zacholme7:spec-tests
Closed

Spec tests#246
Zacholme7 wants to merge 5 commits intosigp:unstablefrom
Zacholme7:spec-tests

Conversation

@Zacholme7
Copy link
Member

Issue Addressed

#39

Proposed Changes

Introductory rough draft to spec tests. This is extremely WIP and any feedback is welcome.

I have attempted to setup a general structure that provides a clean interface for parsing, setting up, and running the spec tests. A key observation here is that spec tests are divided into categories (can discern which category based on .json file name) and each category has a different json structure. We need to be able to read these into an internal format, setup the test, and then run the test.

Test are identified via the SpecTestType enum, which is further subdivided into specifics areas where each inner enum has a variant for each test category.

The core behind this is the SpecTest trait. Each different test category must implement this trait on its respective struct. This provides functionality for setting up the test, running the test, and identifying which test category it is.

All test categories must register itself with the register_test_loaders macro. This allows for a very clean way of parsing and running all of the test categories in a generic way. Without this, we would have to match against all variants and optimistically decode each one in succession.

The whole flow is roughly the following

  1. call run_tests with the variant of test that you would like to run
  2. Walk through the test directory and read in/parse all of the file
  3. Perform any setup, if necessary
  4. Run all all of the tests

There is still a lot of work to be done for setting up the tests and mapping the types into our ssv_types representation, but I think this provides a very extensible and clean framework for implmenting all of the spec tests.

Note: Also included only one testfile for each category for now. Otherwise this pr is 200,000 lines and impossible to review.

@Zacholme7 Zacholme7 closed this Apr 11, 2025
@Zacholme7 Zacholme7 deleted the spec-tests branch April 11, 2025 14:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant