Runs-on for linux-build-lib and linux-test (2X faster CI)#20107
Runs-on for linux-build-lib and linux-test (2X faster CI)#20107comphead merged 3 commits intoapache:mainfrom
linux-build-lib and linux-test (2X faster CI)#20107Conversation
github-actions
bot
added
documentation
|
Nice ! |
comphead
left a comment
comphead
left a comment
There was a problem hiding this comment.
Thanks @blaginin lets give it a try
Are you aware if any limits applies for AWS runners, I checked https://docs.github.com/en/actions/reference/limits but cannot say if it is for AWS runners
Yes, those limits do not apply to us. We will be bounded by the ASF AWS Account budget and service quotas, this will be on me and infra team to monitor I will merge this PR soon and we can start testing 🚀 |
|
I think it is a great idea to try one or two jobs initially and then slowly migrate over The biggest concern I have is if there is some difference between the normal runners (that will run on PRs) and the one that will run on main. |
Jefffrey
left a comment
Jefffrey
left a comment
There was a problem hiding this comment.
DataFusion has one of the largest runtimes across Apache projects
😅
|
If everyone is okay, I'm merging the PR |
Those on |
| linux-build-lib: | ||
| name: linux build test | ||
| runs-on: ubuntu-latest | ||
| runs-on: ${{ github.repository_owner == 'apache' && format('runs-on={0},family=m7a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }} |
There was a problem hiding this comment.
github.repository_owner == 'apache'
❤️
|
BTW I don't have data, but this feels like it made a major difference on CI time recently |
|
Yes, it is around twice faster now for the affected CI jobs! |
|
Thanks @blaginin I think we need the same for Comet, as its heavily tested. Appreciate if you can share how to start |
Of course! I'll switch the remaining DF actions and then happy to do the same with Comet. I'll also write a guide for transitioning :) |
thanks and FYI created apache/datafusion-comet#3404 |
6 participants
Which issue does this PR close?
Related to #13813
Thanks to the infra team and @gmcdonald specifically, we now have the ability to use more powerful AWS-provided runners in our CI 🥳
DataFusion has one of the largest runtimes across Apache projects - that's why we're bringing those runners here first. Since we're first to test this, I think it's reasonable to do a gradual transition, so I updated the two most frequently failing actions to be hosted in AWS. The plan is to test that everything works fine and then transition the remaining actions.
What changes are included in this PR?
If the org is
apache, we'll now use ASF-provisioned runners in the ASF infra AWS account. Forks will not have access to those runners, so they will fall back to GitHub-provisioned ones.Are these changes tested?
Yes.
linux-build-libandlinux-testblaginin/datafusion#10Are there any user-facing changes?
No