Skip to content

Conversation

@malka4325
Copy link

Summary & Motivation

Fix collision detection for Asset + AssetSpec with the same AssetKey

Currently, when loading modules with include_specs=True, Dagster raises a
duplicate asset error if an Asset and an AssetSpec share the same AssetKey.
This prevents horizontal imports and proper use of AssetSpecs across files.

This change allows a single executable Asset to coexist with one or more
AssetSpecs for the same key without raising an error, while still raising
errors for duplicate executable Assets.

How I Tested These Changes

Added unit tests covering:

  • Loading an Asset + AssetSpec with the same key does not raise an error.
  • Loading two executable Assets with the same key still raises DagsterInvalidDefinitionError.
  • Ran all existing Dagster tests locally to ensure nothing else is broken.

Changelog

  • Fix collision detection in load_assets_from_modules when include_specs=True.
  • Add tests for Asset + AssetSpec coexistence.

@malka4325
Copy link
Author

@cmpadden can you unblock the buildkite?

Copy link
Contributor

@cmpadden cmpadden left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @malka4325.

There is a potential failure in this implementation (see test below) in that the new collision logic only ignores specs when len(asset_defs) == 1. In this test, the same AssetsDefinition object is imported into two modules, so asset_defs has length 2 but only one distinct id. That passes the "multiple distinct defs" check, but because len(asset_defs) != 1, the code does not skip specs. It then compares the two AssetSpec objects (they differ due to description) and raises a collision, even though an AssetsDefinition exists and should make specs irrelevant.

@pytest.mark.xfail(
    reason=(
        "Asset specs should be ignored when an AssetsDefinition exists, even if the "
        "AssetsDefinition is imported in multiple modules."
    ),
    strict=True,
)
def test_assets_def_imported_twice_with_conflicting_specs_does_not_raise():
    mod1 = types.ModuleType("mod1")
    mod2 = types.ModuleType("mod2")

    @dg.asset
    def asset1():
        return 1

    mod1.asset1 = asset1
    mod2.asset1 = asset1
    mod1.asset1_spec = dg.AssetSpec("asset1", description="mod1")
    mod2.asset1_spec = dg.AssetSpec("asset1", description="mod2")

    dg.load_assets_from_modules([mod1, mod2], include_specs=True)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants