Spec tests Framework

[Zacholme7]

Issue Addressed

#39

Proposed Changes

Introductory rough draft to spec tests. This is extremely WIP and any feedback is welcome.

I have attempted to setup a general structure that provides a clean interface for parsing, setting up, and running the spec tests. A key observation here is that spec tests are divided into categories (can discern which category based on .json file name) and each category has a different json structure. We need to be able to read these into an internal format, setup the test, and then run the test.

Test are identified via the SpecTestType enum, which is further subdivided into specifics areas where each inner enum has a variant for each test category.

The core behind this is the SpecTest trait. Each different test category must implement this trait on its respective struct. This provides functionality for setting up the test, running the test, and identifying which test category it is.

All test categories must register itself with the register_test_loaders macro. This allows for a very clean way of parsing and running all of the test categories in a generic way. Without this, we would have to match against all variants and optimistically decode each one in succession.

The whole flow is roughly the following

1. call run_tests with the variant of test that you would like to run
2. Walk through the test directory and read in/parse all of the file
3. Perform any setup, if necessary
4. Run all all of the tests

We are still missing implementations for the type spec and the ssv spec. It is pretty easy to create the scaffolding for these by following the qbft example.

There is still a lot of work to be done for setting up the tests and mapping the types into our ssv_types representation, but I think this provides a very extensible and clean framework for implmenting all of the spec tests.

[Copilot]

Copilot reviewed 11 out of 13 changed files in this pull request and generated no comments.

Files not reviewed (2)

• .gitmodules: Language not supported
• anchor/spec_tests/src/ssv-spec: Language not supported

Comments suppressed due to low confidence (2)

anchor/spec_tests/src/qbft/round_robin.rs:22

• The RoundRobinTest struct is not declared as public, unlike TimeoutTest. If external access is intended, consider adding the pub modifier.

struct RoundRobinTest {

Cargo.toml:54

• The spec_tests package is registered with an unexpected path. It should likely point to "anchor/spec_tests" rather than "anchor/common/version" to correctly reference the new package.

spec_tests = { path = "anchor/common/version" }

[diegomrsantos]

Could this be a property in the variant?

[Zacholme7]

Not sure what you mean, do you mean on the trait? or the inner enum variant?

[diegomrsantos]

In the enum variant

[diegomrsantos]

Maybe we could split it into run and assert

[Zacholme7]

Yea I want to head in this direction but im holding off for now because I am not sure how the structure of this would work. Need to dig deeper into the other tests first.

Might be a case of setting up tests and then refactoring to an assertion func too

[Copilot]

Pull Request Overview

This PR introduces an initial framework for running spec tests by defining a SpecTest trait and its implementations for various QBFT test types. Key changes include:

• Adding individual test modules for Timeout, RoundRobin, QbftMessage, MessageProcessing, CreateMessage, and Controller.
• Defining the QbftSpecTestType enum and its Display implementation for file-path construction.
• Setting up a test loader registry and a basic test runner in the library.

Reviewed Changes

Copilot reviewed 11 out of 13 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
anchor/spec_tests/src/qbft/timeout.rs	Adds TimeoutTest with SpecTest implementation.
anchor/spec_tests/src/qbft/round_robin.rs	Adds RoundRobinTest with SpecTest implementation.
anchor/spec_tests/src/qbft/qbft_message.rs	Adds QbftMessageTest with SpecTest implementation.
anchor/spec_tests/src/qbft/message_processing.rs	Adds MessageProcessingTest with SpecTest implementation.
anchor/spec_tests/src/qbft/create_message.rs	Adds CreateMessageTest with SpecTest implementation.
anchor/spec_tests/src/qbft/controller.rs	Adds ControllerTest with SpecTest implementation.
anchor/spec_tests/src/qbft/mod.rs	Defines QbftSpecTestType enum and module organization.
anchor/spec_tests/src/lib.rs	Sets up the SpecTest trait, test loader registration, and the test runner.
anchor/spec_tests/Cargo.toml	Declares the package and dependencies for spec tests.
Cargo.toml	Registers the spec_tests crate in the workspace.

Files not reviewed (2)

• .gitmodules: Language not supported
• anchor/spec_tests/src/ssv-spec: Language not supported

[Copilot]

[nitpick] The 'name' field in TimeoutTest is declared with private visibility, while similar test structs in other modules use public visibility. Consider standardizing the field visibility across all test structs for consistency.

Suggested change

	name: String,
	pub name: String,

[Copilot]

Only TimeoutTest is registered in TEST_LOADERS, leaving other SpecTest implementations unregistered. Consider adding registrations for all test types to ensure comprehensive test loading.

Suggested change

	static TEST_LOADERS: LazyLock<Loaders> = register_test_loaders!(TimeoutTest);
	static TEST_LOADERS: LazyLock<Loaders> = register_test_loaders!(TimeoutTest, ProposalTest, CommitTest);

[Zacholme7]

Closing for now