Skip to content

Have nightly tests against nightly generated artifacts#7366

Closed
GaAstDev wants to merge 10 commits intosigp:unstablefrom
GaAstDev:feat/nightly-test-vectors
Closed

Have nightly tests against nightly generated artifacts#7366
GaAstDev wants to merge 10 commits intosigp:unstablefrom
GaAstDev:feat/nightly-test-vectors

Conversation

@GaAstDev
Copy link

Issue Addressed

Hello !
This PR aims to have a new nightly action that pulls the nightly generated artifacts from the CL specs repo and have the unstable branch run against them.

See: https://github.com/ethereum/consensus-specs/actions/workflows/nightly-tests.yml

Proposed Changes

To achieve this goal, this PR:

  1. Creates a new workflow .github/workflows/nightly-tests.yml
  2. Adds a new target in the Makefile run-ef-nightly-tests
  3. Have the retrieval of the artifacts happen in testing/ef_tests/Makefile

Please note, this required a Github Authentication Token, I reused the one existing in the Github secret vault that has repo access: secrets.LIGHTHOUSE_GITHUB_TOKEN

@CLAassistant
Copy link

CLAassistant commented Apr 28, 2025

CLA assistant check
All committers have signed the CLA.

@GaAstDev GaAstDev changed the title Feat/nightly test vectors Have nightly tests against nightly generated artifacts Apr 28, 2025
@GaAstDev GaAstDev changed the base branch from stable to unstable April 28, 2025 19:27
@michaelsproul michaelsproul added the test improvement Improve tests label Apr 29, 2025
@michaelsproul
Copy link
Member

Hi @GabrielAstieres, if you rebase on unstable that should fix all the CI failures (the target branch check won't update until you push a new commit, and the other two are fixed in unstable)

@michaelsproul michaelsproul added waiting-on-author The reviewer has suggested changes and awaits thier implementation. v7.1.0 Post-Electra release labels Apr 29, 2025
@GaAstDev GaAstDev force-pushed the feat/nightly-test-vectors branch from 335154f to 06699e7 Compare April 29, 2025 08:53
@GaAstDev
Copy link
Author

Hi @GabrielAstieres, if you rebase on unstable that should fix all the CI failures (the target branch check won't update until you push a new commit, and the other two are fixed in unstable)

Hey @michaelsproul!
I've succesfully rebased on unstable. Thanks

@michaelsproul
Copy link
Member

Thanks!

@GaAstDev
Copy link
Author

GaAstDev commented May 9, 2025

Hey Michael @michaelsproul, did you get a chance to look at this? :)

@jimmygchen jimmygchen added ready-for-review The code is ready for review and removed waiting-on-author The reviewer has suggested changes and awaits thier implementation. labels May 15, 2025
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's up to the team, but personally I think we should remove this workflow.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jtraglia Are you concerned about noisy breakages, or accidentally revealing some bug on public CI? Both?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just noise. Lots of the time, these tests will fail due to unimplemented spec changes. And I'm planning on aggregating the status of these in a new website (and eventually on Hive).

@michaelsproul
Copy link
Member

Closing to prevent noisy breakages. LH generally won't be able to keep up with nightly test vectors

@jtraglia
Copy link
Contributor

jtraglia commented May 29, 2025

Hey @michaelsproul & @GabrielAstieres, sounds good. Let's open a new PR which just adds the ability to run the nightly reference tests, eg the Makefile stuff from here.

@michaelsproul
Copy link
Member

Sounds good. Maybe even a Github action that we could trigger on demand?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-review The code is ready for review test improvement Improve tests v7.1.0 Post-Electra release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants