Skip to content

Conversation

@Denyme24
Copy link

Purpose of this PR

This PR adds resource quota validation support for SparkConnect to prevent silent pod failures and provide clear error messages to users when quota limits are exceeded.

fixes #2790

Proposed changes:

  • Add validating admission webhook for SparkConnect to validate resource quotas at creation time
  • Implement resource quota calculation logic that accounts for server pod + all executor pods
  • Enhance SparkConnect controller to update status with quota validation errors
  • Register SparkConnect webhook in operator startup to enable quota enforcement
  • Add v1alpha1 scheme support to WebhookScheme for proper SparkConnect object handling

WhatsApp Image 2026-01-20 at 9 19 19 PM

  • SparkConnect creation is now rejected with clear error messages when it would exceed namespace resource quotas
  • Status field shows detailed quota validation errors instead of remaining empty
  • Behavior is now consistent with SparkApplication's resource quota enforcement

Change Category

  • Bugfix (non-breaking change which fixes an issue)
  • Feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that could affect existing functionality)
  • Documentation update

Rationale

Checklist

  • I have conducted a self-review of my own code.
  • I have updated documentation accordingly.
  • I have added tests that prove my changes are effective or that my feature works.
  • Existing unit tests pass locally with my changes.

Additional Notes

Signed-off-by: Denyme24 <namanraj24@outlook.com>
@google-oss-prow
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign chenyi015 for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@Denyme24
Copy link
Author

hey @nabuskey @ImpSy, PTAL.

@Denyme24
Copy link
Author

hey @ChenYi015 @Shekharrajak, could you please review these changes.
thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] No resource quota support in Spark Connect

1 participant