Fix dag_version FK constraint violation in db cleanup (#61390) #61435
+286
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The airflow db clean command was failing with FK constraint violations when cleaning the dag_version table. This occurred because the cleanup logic only checked the created_at timestamp, without verifying if dag_version rows were still referenced by task_instance or dag_run tables. Root cause: - task_instance.dag_version_id has FK with ondelete=RESTRICT - dag_run.created_dag_version_id has FK with ondelete=set null - Cleanup tried to delete referenced dag_version rows Solution: - Modified _build_query() to exclude dag_version rows still referenced by task_instance or dag_run using NOT IN subquery - Uses SQLAlchemy, handles NULL values, efficient for large DBs Tests added: - test_dag_version_cleanup_respects_task_instance_fk - test_dag_version_cleanup_deletes_unreferenced_versions - test_dag_version_cleanup_respects_dag_run_fk Closes: apache#61390
Replace UNION subquery with multiple NOT IN conditions for better cross-dialect compatibility. The UNION approach was failing in CI across MySQL, Postgres, SQLite, and low dependency environments. Issue: SQLAlchemy's handling of UNION within NOT IN subqueries is inconsistent across dialects and versions. Solution: Use two separate NOT IN conditions (one for task_instance, one for dag_run) following the pattern used in scheduler_job_runner.py for _remove_unreferenced_triggers. This pattern is proven to work across all Airflow-supported databases and SQLAlchemy versions.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
SUMMARY
Fix foreign key constraint failures when running
airflow db cleanon thedag_versiontable.Previously, cleanup logic filtered
dag_versionrows only based on timestamp conditions.This allowed deletion attempts for rows still referenced by
task_instance.dag_version_id,causing foreign key constraint errors during database cleanup.
This change excludes
dag_versionrows that are still referenced by:This ensures cleanup respects referential integrity and prevents runtime failures.
ROOT CAUSE
The cleanup query did not account for existing foreign key relationships:
SOLUTION
Add filtering logic to exclude referenced dag_version rows using a subquery union of:
TESTING
closes: #61390
related: #52147
related: #59474