-
Notifications
You must be signed in to change notification settings - Fork 16.4k
Add cancel_on_kill support for EMR Serverless deferrable operator #60440
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
|
BasPH
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
first quick scan, few comments
|
@cruseakshay could you fix the failing test? |
Apply prek hook formatting to single-line assertion. Co-authored-by: Cursor <cursoragent@cursor.com>
Replace deprecated session.query().filter() with select().where() and session.scalars() pattern. Co-authored-by: Cursor <cursoragent@cursor.com>
Done |
This PR adds support for cancelling EMR Serverless jobs when an Airflow task is killed while in deferrable mode.
Problem:
When using
EmrServerlessStartJobOperatorwithdeferrable=True, if a user cancels the task in Airflow, the EMR Serverless job continues running in AWS. This leads to orphaned jobs consuming resourcesSolution:
Implements the "cancel_on_kill" (similar to
DataprocSubmitJobTrigger) for EMR Serverless:cancel_on_killparameter toEmrServerlessStartJobOperator(default:True)cancel_on_killparameter toEmrServerlessStartJobTriggersafe_to_cancel()method in the trigger to distinguish between:run()method to catchasyncio.CancelledErrorand cancel the EMR job when safeChanges:
providers/amazon/aws/operators/emr.py: Addedcancel_on_killparameterproviders/amazon/aws/triggers/emr.py: Added cancellation logic withsafe_to_cancel()tests/unit/amazon/aws/triggers/test_emr.py: Added unit tests for cancellation scenariosTesting:
Was generative AI tooling used to co-author this PR?
Generated-by: Claude (Cursor IDE) following the guidelines
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.