Skip to content

Handle resource exhaustion for hhmi in us-west-2#7691

Merged
agoose77 merged 4 commits into2i2c-org:mainfrom
yuvipanda:hhmi-reduce
Feb 24, 2026
Merged

Handle resource exhaustion for hhmi in us-west-2#7691
agoose77 merged 4 commits into2i2c-org:mainfrom
yuvipanda:hhmi-reduce

Conversation

@yuvipanda
Copy link
Member

@yuvipanda yuvipanda commented Feb 17, 2026

  • Spread the load out to all zones, as us-west2 seems much smaller in GCP
  • Ask for a smaller VM, so we have less chances of running out of resources. Also helps with cloud cost
  • Set max VM count to be smaller

Ref #7657

In 2i2c-org#7655,
sometime we run into resource exhaustion issues on the GCP
zone HHMI is in. Since this hub sees infrequent *mass* usage,
we can reduce the chances of resource starvation by requesting
smaller nodes, as well as some cloud cost reduction
A single zone gets exhausted far too often
@yuvipanda
Copy link
Member Author

Not actually applied!

@agoose77
Copy link
Contributor

This also needs the hub node selector to be updated.

@yuvipanda
Copy link
Member Author

I'm not planning on taking this through to completion, so someone else should :)

@agoose77
Copy link
Contributor

@yuvipanda I am! Just took a second to get here 😢

@agoose77 agoose77 self-assigned this Feb 20, 2026
@agoose77 agoose77 moved this to In progress in Product and Services Feb 20, 2026
@github-actions
Copy link
Contributor

github-actions bot commented Feb 20, 2026

Merging this PR will trigger the following deployment actions.

Support deployments

No support upgrades will be triggered

Staging deployments

Cloud Provider Cluster Name Hub Name Reason for Redeploy
gcp hhmi staging Following helm chart values files were modified: staging.values.yaml

Production deployments

Cloud Provider Cluster Name Hub Name Reason for Redeploy
gcp hhmi spyglass Following helm chart values files were modified: spyglass.values.yaml
gcp hhmi binder Following helm chart values files were modified: binder.values.yaml

@agoose77
Copy link
Contributor

I was holding off on this whilst we were dealing with lots of infra work. I will now merge this tomorrow!

@agoose77 agoose77 merged commit 57ce6c7 into 2i2c-org:main Feb 24, 2026
3 of 5 checks passed
@github-project-automation github-project-automation bot moved this from In progress to Done in Product and Services Feb 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants