Add TableMetadataSet to AssetDeps produced by BI Integrations#33414
Open
michalcabir-ui wants to merge 3 commits intodagster-io:masterfrom
Open
Add TableMetadataSet to AssetDeps produced by BI Integrations#33414michalcabir-ui wants to merge 3 commits intodagster-io:masterfrom
michalcabir-ui wants to merge 3 commits intodagster-io:masterfrom
Conversation
…r-looker, dagster-sigma, and dagster-powerbi with TableMetadataSet.
Contributor
Author
|
Hi @OwenKephart, I don't have permission to add reviewers, so I'm tagging you here as requested. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary & Motivation
This PR enriches the AssetSpecs produced by several BI integrations (dagster-omni, dagster-tableau, dagster-looker, dagster-sigma, and dagster-powerbi) by including TableMetadataSet.
Extracts a clean table_name from the source object (stripping schema prefixes or internal IDs where necessary) and populates the dagster/table_name metadata key.
For dagster-tableau and dagster-looker, it additionally attempts to extract the underlying dagster/storage_kind (e.g., "snowflake", "bigquery") from the connection definition, falling back to the integration name if unavailable.
For dagster-omni, it implements logic to parse table names from the db_schema__table format.
How I Tested These Changes
dagster-omni: Added tests to verify the _extract_table_name_from_omni_table_name helper correctly splits schema__table strings.
dagster-tableau: Added tests for both published and embedded data sources, and a specific test case to verify dagster/storage_kind is correctly inferred from the connectionType property.
dagster-looker: Updated test_build_defs to assert dagster/table_name exists on Views and Explores, and verified that dagster/storage_kind tags are applied.
dagster-sigma: Updated translator tests to verify dataset metadata.
dagster-powerbi: Added a new test case to verify data source metadata enrichment.
Changelog
[dagster-omni, dagster-tableau, dagster-looker, dagster-sigma, dagster-powerbi] Automatically enrich generated assets with dagster/table_name metadata. Additionally, dagster-tableau and dagster-looker assets now populate dagster/storage_kind based on the upstream connection type.