Skip to content

Conversation

@cruseakshay
Copy link
Contributor

This PR adds support for cancelling EMR Serverless jobs when an Airflow task is killed while in deferrable mode.

Problem:
When using EmrServerlessStartJobOperator with deferrable=True, if a user cancels the task in Airflow, the EMR Serverless job continues running in AWS. This leads to orphaned jobs consuming resources

Solution:
Implements the "cancel_on_kill" (similar to DataprocSubmitJobTrigger) for EMR Serverless:

  1. Added cancel_on_kill parameter to EmrServerlessStartJobOperator (default: True)
  2. Added cancel_on_kill parameter to EmrServerlessStartJobTrigger
  3. Implemented safe_to_cancel() method in the trigger to distinguish between:
    • User-initiated cancellation (task state changed from DEFERRED) → Cancel the EMR job
    • Triggerer shutdown/restart → Keep the EMR job running
  4. Override run() method to catch asyncio.CancelledError and cancel the EMR job when safe

Changes:

  • providers/amazon/aws/operators/emr.py: Added cancel_on_kill parameter
  • providers/amazon/aws/triggers/emr.py: Added cancellation logic with safe_to_cancel()
  • tests/unit/amazon/aws/triggers/test_emr.py: Added unit tests for cancellation scenarios

Testing:

  • ✅ Unit tests added and passing

Was generative AI tooling used to co-author this PR?

  • Yes - Claude (Cursor IDE)

Generated-by: Claude (Cursor IDE) following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@boring-cyborg
Copy link

boring-cyborg bot commented Jan 13, 2026

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@boring-cyborg boring-cyborg bot added area:providers provider:amazon AWS/Amazon - related issues labels Jan 13, 2026
Copy link
Contributor

@BasPH BasPH left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

first quick scan, few comments

@cruseakshay cruseakshay marked this pull request as ready for review January 16, 2026 07:28
@cruseakshay cruseakshay requested a review from o-nikolas January 20, 2026 04:44
@BasPH
Copy link
Contributor

BasPH commented Feb 4, 2026

@cruseakshay could you fix the failing test?

cruseakshay and others added 2 commits February 4, 2026 18:48
Apply prek hook formatting to single-line assertion.

Co-authored-by: Cursor <cursoragent@cursor.com>
Replace deprecated session.query().filter() with select().where()
and session.scalars() pattern.

Co-authored-by: Cursor <cursoragent@cursor.com>
@akshaykumarsalunke
Copy link

@cruseakshay could you fix the failing test?

Done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:amazon AWS/Amazon - related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants