Skip to content

fix: reschedule nightly and optimize canary runs#1691

Open
Shubhranshu153 wants to merge 1 commit intomainfrom
fix-workflows
Open

fix: reschedule nightly and optimize canary runs#1691
Shubhranshu153 wants to merge 1 commit intomainfrom
fix-workflows

Conversation

@Shubhranshu153
Copy link
Member

@Shubhranshu153 Shubhranshu153 commented Feb 6, 2026

Problem

  1. Nightly builds were scheduled at 9:00 UTC which conflicted with patch reboot times
  2. Deb Canary workflow was running every 10 minutes, creating 144 workflow runs per day and flooding the GitHub Actions page

Solution

1. Reschedule Nightly Builds to 6 AM PST (14:00 UTC)

All nightly builds have been moved to 6 AM PST to avoid conflicts with patch reboot windows.

2. Optimize Deb Canary Workflow

Restructured the canary workflow to run hourly while maintaining the same monitoring frequency:

  • I've reviewed the guidance in CONTRIBUTING.md

License Acceptance

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Signed-off-by: Shubhranshu Mahapatra <shubhum@amazon.com>
@Shubhranshu153 Shubhranshu153 requested a review from a team as a code owner February 6, 2026 14:30
max_attempts: 3
retry_wait_seconds: 30
command: ./scripts/canary-deb.sh
- name: Run canary script (6 checks over ~55 minutes)
Copy link
Contributor

@Swapnanil-Gupta Swapnanil-Gupta Feb 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it a good idea to block this runner for 55-60 minutes when most of that time is spent sleeping?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not really so the other case you are spinning up a runner every 5 mins and it runs for 3 mins and stops. if its a cold start then that is expensive.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually yes the runners are running for only ~10-15 seconds, in that case this can be worse. i will look what other options are there

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Part of the reason for splitting the canary into a separate shellscript was so that we can easily run it on a local machine if we wanted to.

I would prefer if we keep singular canary runs within a singular script. Then call the script multiple times from either a separate bash file or within GitHub. (Or even a Go file?)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can do that or have an option to control it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm also okay with that. But I would like the option to do so easily before approving.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants