feature/union-data

.gitignore

@@ -71,3 +71,6 @@ env/
 env.bak/
 venv/
 venv.bak/
+
+# Claude instructions
+union_data_agents.md

CHANGELOG.md

@@ -1,3 +1,19 @@
+# dbt_recharge v1.2.0
+
+## Schema/Data Change
+**1 total change • 0 possible breaking changes**
+
+| Data Model(s) | Change type | Old | New | Notes |
+| ------------- | ----------- | ----| --- | ----- |
+| All models | New column | | `source_relation` | Identifies the source connection when using multiple Recharge connections |
+
+## Feature Update
+- **Union Data Functionality**: This release supports running the package on multiple Recharge source connections. See the [README](https://github.com/fivetran/dbt_recharge/tree/main?tab=readme-ov-file#step-3-define-database-and-schema-variables) for details on how to leverage this feature.
+
+## Tests Update
+- Removes uniqueness tests. The new unioning feature requires combination-of-column tests to consider the new `source_relation` column in addition to the existing primary key, but this is not supported across dbt versions.
+- These tests will be reintroduced once a version-agnostic solution is available.
+
 # dbt_recharge v1.1.0
 This release aligns with the [Fivetran Recharge Connector August 2025](https://fivetran.com/docs/changelog/2025/august-2025#recharge) updates. [PR #35](https://github.com/fivetran/dbt_recharge/pull/35) includes the following updates:
 
@@ -8,7 +24,7 @@ This release aligns with the [Fivetran Recharge Connector August 2025](https://f
 | --------------------- | --------------- | ------- |-------- | --------- |
 | `ORDER` | Source removed | | | Sunset in favor of `ORDERS`, which is now standard across all connections. |
 | [`stg_recharge__discount`](https://fivetran.github.io/dbt_recharge/#!/model/model.recharge.stg_recharge__discount) | Column removed | `applies_to_id` | | Removed due to inconsistent data types (JSON vs. string) in the source. It can be re-enabled via the newly added `recharge__discount_passthrough_columns` variable. See the [Passing Through Additional Columns](https://github.com/fivetran/dbt_recharge/blob/main/README.md#passing-through-additional-columns) section in the README for usage details. |
-| [`stg_recharge__one_time_product`](https://fivetran.github.io/dbt_recharge/#!/model/model.recharge.stg_recharge__one_time_product)<br>[`stg_recharge__order`](https://fivetran.github.io/dbt_recharge/#!/model/model.recharge.stg_recharge__order)<br>[`recharge__billing_history`](https://fivetran.github.io/dbt_recharge/#!/model/model.re-0charge.recharge__billing_history) | Column removed | `is_deleted` | | Removed deprecated column. The same information is already captured in `_fivetran_deleted`. |
+| [`stg_recharge__one_time_product`](https://fivetran.github.io/dbt_recharge/#!/model/model.recharge.stg_recharge__one_time_product)<br>[`stg_recharge__order`](https://fivetran.github.io/dbt_recharge/#!/model/model.recharge.stg_recharge__order)<br>[`recharge__billing_history`](https://fivetran.github.io/dbt_recharge/#!/model/model.recharge.recharge__billing_history) | Column removed | `is_deleted` | | Removed deprecated column. The same information is already captured in `_fivetran_deleted`. |
 
 ## Documentation
 - Removed deprecated source and columns from documentation.

README.md

@@ -64,7 +64,7 @@ Include the following recharge package version in your `packages.yml` file.
 ```yaml
 packages:
   - package: fivetran/recharge
-    version: [">=1.1.0", "<1.2.0"] # we recommend using ranges to capture non-breaking changes automatically
+    version: [">=1.2.0", "<1.3.0"] # we recommend using ranges to capture non-breaking changes automatically
 ```
 > All required sources and staging models are now bundled into this transformation package. Do not include `fivetran/recharge_source` in your `packages.yml` since this package has been deprecated.
 
@@ -77,12 +77,72 @@ dispatch:
     search_order: ['spark_utils', 'dbt_utils']
 ```
 ### Step 3: Define database and schema variables
-By default, this package runs using your destination and the `recharge` schema. If your Recharge data is in a different database or schema (for example, if your Recharge schema is named `recharge_fivetran`), add the following configuration to your root `dbt_project.yml` file:
 
+#### Option A: Single connection
+By default, this package runs using your [destination](https://docs.getdbt.com/docs/running-a-dbt-project/using-the-command-line-interface/configure-your-profile) and the `recharge` schema. If this is not where your Recharge data is (for example, if your Recharge schema is named `recharge_fivetran`), add the following configuration to your root `dbt_project.yml` file:
+
+```yml
+vars:
+  recharge:
+    recharge_database: your_database_name
+    recharge_schema: your_schema_name
+```
+
+#### Option B: Union multiple connections
+If you have multiple Recharge connections in Fivetran and would like to use this package on all of them simultaneously, we have provided functionality to do so. For each source table, the package will union all of the data together and pass the unioned table into the transformations. The `source_relation` column in each model indicates the origin of each record.
+
+To use this functionality, you will need to set the `recharge_sources` variable in your root `dbt_project.yml` file:
+
+```yml
+# dbt_project.yml
+
+vars:
+  recharge:
+    recharge_sources:
+      - database: connection_1_destination_name # Required
+        schema: connection_1_schema_name # Required
+        name: connection_1_source_name # Required only if following the step in the following subsection
+
+      - database: connection_2_destination_name
+        schema: connection_2_schema_name
+        name: connection_2_source_name
+```
+
+##### Recommended: Incorporate unioned sources into DAG
+> *If you are running the package through [Fivetran Transformations for dbt Core™](https://fivetran.com/docs/transformations/dbt#transformationsfordbtcore), the below step is necessary in order to synchronize model runs with your Recharge connections. Alternatively, you may choose to run the package through Fivetran [Quickstart](https://fivetran.com/docs/transformations/quickstart), which would create separate sets of models for each Recharge source rather than one set of unioned models.*
+
+By default, this package defines one single-connection source, called `recharge`, which will be disabled if you are unioning multiple connections. This means that your DAG will not include your Recharge sources, though the package will run successfully.
+
+To properly incorporate all of your Recharge connections into your project's DAG:
+1. Define each of your sources in a `.yml` file in your project. Utilize the following template for the `source`-level configurations, and, **most importantly**, copy and paste the table and column-level definitions from the package's `src_recharge.yml` [file](https://github.com/fivetran/dbt_recharge/blob/main/models/staging/src_recharge.yml).
+
+```yml
+# a .yml file in your root project
+
+version: 2
+
+sources:
+  - name: <name> # ex: Should match name in recharge_sources
+    schema: <schema_name>
+    database: <database_name>
+    loader: fivetran
+    config:
+      loaded_at_field: _fivetran_synced
+      freshness: # feel free to adjust to your liking
+        warn_after: {count: 72, period: hour}
+        error_after: {count: 168, period: hour}
+
+    tables: # copy and paste from recharge/models/staging/src_recharge.yml - see https://support.atlassian.com/bitbucket-cloud/docs/yaml-anchors/ for how to use anchors to only do so once
+```
+
+> **Note**: If there are source tables you do not have (see [Step 4](#step-4-enabledisable-models-and-sources)), you may still include them, as long as you have set the right variables to `False`.
+
+2. Set the `has_defined_sources` variable (scoped to the `recharge` package) to `True`, like such:
 ```yml
+# dbt_project.yml
 vars:
-  recharge_database: your_destination_name
-  recharge_schema: your_schema_name 
+  recharge:
+    has_defined_sources: true
 ```
 
 ### Step 4: Enable/disable models and sources

dbt_project.yml

@@ -1,6 +1,6 @@
 
 name: 'recharge'
-version: '1.1.0'
+version: '1.2.0'
 config-version: 2
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 

integration_tests/dbt_project.yml

@@ -1,13 +1,13 @@
 config-version: 2
 
 name: 'recharge_integration_tests'
-version: '1.1.0'
+version: '1.2.0'
 
 profile: 'integration_tests' 
 
 models:
   +schema: "recharge_{{ var('directed_schema','dev') }}"  ## To be used for validation tests
-
+ 
 vars:
   # recharge__checkout_enabled: true # @docs-include
   recharge:
@@ -210,4 +210,4 @@ seeds:
         _FIVETRAN_SYNCED: "timestamp"
 flags:
   send_anonymous_usage_stats: False
-  use_colors: True
+  use_colors: True

integration_tests/tests/consistency/consistency_billing_history.sql

@@ -3,13 +3,15 @@
     enabled=var('fivetran_validation_tests_enabled', false)
 ) }}
 
+{% set exclude_cols = ['_fivetran_deleted'] + var('consistency_test_exclude_metrics', []) %}
+
 with prod as (
-    select {{ dbt_utils.star(from=ref('recharge__billing_history'), except=['_fivetran_deleted']) }} --To be switched to * after this release.
+    select {{ dbt_utils.star(from=ref('recharge__billing_history'), except=exclude_cols) }}
     from {{ target.schema }}_recharge_prod.recharge__billing_history
 ),
 
 dev as (
-    select {{ dbt_utils.star(from=ref('recharge__billing_history'), except=['_fivetran_deleted']) }} --To be switched to * after this release.
+    select {{ dbt_utils.star(from=ref('recharge__billing_history'), except=exclude_cols) }}
     from {{ target.schema }}_recharge_dev.recharge__billing_history
 ), 
 

integration_tests/tests/consistency/consistency_customer_daily_rollup.sql

@@ -0,0 +1,48 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+{% set exclude_cols = var('consistency_test_exclude_metrics', []) %}
+
+-- this test ensures the recharge__customer_daily_rollup end model matches the prior version
+with prod as (
+    select {{ dbt_utils.star(from=ref('recharge__customer_daily_rollup'), except=exclude_cols) }}
+    from {{ target.schema }}_recharge_prod.recharge__customer_daily_rollup
+),
+
+dev as (
+    select {{ dbt_utils.star(from=ref('recharge__customer_daily_rollup'), except=exclude_cols) }}
+    from {{ target.schema }}_recharge_dev.recharge__customer_daily_rollup
+),
+
+prod_not_in_dev as (
+    -- rows from prod not found in dev
+    select * from prod
+    except distinct
+    select * from dev
+),
+
+dev_not_in_prod as (
+    -- rows from dev not found in prod
+    select * from dev
+    except distinct
+    select * from prod
+),
+
+final as (
+    select
+        *,
+        'from prod' as source
+    from prod_not_in_dev
+
+    union all -- union since we only care if rows are produced
+
+    select
+        *,
+        'from dev' as source
+    from dev_not_in_prod
+)
+
+select *
+from final

integration_tests/tests/consistency/consistency_customer_details.sql

@@ -3,13 +3,15 @@
     enabled=var('fivetran_validation_tests_enabled', false)
 ) }}
 
+{% set exclude_cols = ['_fivetran_deleted'] + var('consistency_test_exclude_metrics', []) %}
+
 with prod as (
-    select {{ dbt_utils.star(from=ref('recharge__customer_details'), except=['_fivetran_deleted']) }}
+    select {{ dbt_utils.star(from=ref('recharge__customer_details'), except=exclude_cols) }}
     from {{ target.schema }}_recharge_prod.recharge__customer_details
 ),
 
 dev as (
-    select {{ dbt_utils.star(from=ref('recharge__customer_details'), except=['_fivetran_deleted']) }}
+    select {{ dbt_utils.star(from=ref('recharge__customer_details'), except=exclude_cols) }}
     from {{ target.schema }}_recharge_dev.recharge__customer_details
 ), 
 

integration_tests/tests/consistency/consistency_line_item_enhanced.sql

@@ -0,0 +1,48 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+{% set exclude_cols = var('consistency_test_exclude_metrics', []) %}
+
+-- this test ensures the recharge__line_item_enhanced end model matches the prior version
+with prod as (
+    select {{ dbt_utils.star(from=ref('recharge__line_item_enhanced'), except=exclude_cols) }}
+    from {{ target.schema }}_recharge_prod.recharge__line_item_enhanced
+),
+
+dev as (
+    select {{ dbt_utils.star(from=ref('recharge__line_item_enhanced'), except=exclude_cols) }}
+    from {{ target.schema }}_recharge_dev.recharge__line_item_enhanced
+),
+
+prod_not_in_dev as (
+    -- rows from prod not found in dev
+    select * from prod
+    except distinct
+    select * from dev
+),
+
+dev_not_in_prod as (
+    -- rows from dev not found in prod
+    select * from dev
+    except distinct
+    select * from prod
+),
+
+final as (
+    select
+        *,
+        'from prod' as source
+    from prod_not_in_dev
+
+    union all -- union since we only care if rows are produced
+
+    select
+        *,
+        'from dev' as source
+    from dev_not_in_prod
+)
+
+select *
+from final

integration_tests/tests/consistency/consistency_monthly_recurring_revenue.sql

@@ -0,0 +1,48 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+{% set exclude_cols = var('consistency_test_exclude_metrics', []) %}
+
+-- this test ensures the recharge__monthly_recurring_revenue end model matches the prior version
+with prod as (
+    select {{ dbt_utils.star(from=ref('recharge__monthly_recurring_revenue'), except=exclude_cols) }}
+    from {{ target.schema }}_recharge_prod.recharge__monthly_recurring_revenue
+),
+
+dev as (
+    select {{ dbt_utils.star(from=ref('recharge__monthly_recurring_revenue'), except=exclude_cols) }}
+    from {{ target.schema }}_recharge_dev.recharge__monthly_recurring_revenue
+),
+
+prod_not_in_dev as (
+    -- rows from prod not found in dev
+    select * from prod
+    except distinct
+    select * from dev
+),
+
+dev_not_in_prod as (
+    -- rows from dev not found in prod
+    select * from dev
+    except distinct
+    select * from prod
+),
+
+final as (
+    select
+        *,
+        'from prod' as source
+    from prod_not_in_dev
+
+    union all -- union since we only care if rows are produced
+
+    select
+        *,
+        'from dev' as source
+    from dev_not_in_prod
+)
+
+select *
+from final

integration_tests/tests/consistency/consistency_subscription_overview.sql

@@ -0,0 +1,48 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+{% set exclude_cols = var('consistency_test_exclude_metrics', []) %}
+
+-- this test ensures the recharge__subscription_overview end model matches the prior version
+with prod as (
+    select {{ dbt_utils.star(from=ref('recharge__subscription_overview'), except=exclude_cols) }}
+    from {{ target.schema }}_recharge_prod.recharge__subscription_overview
+),
+
+dev as (
+    select {{ dbt_utils.star(from=ref('recharge__subscription_overview'), except=exclude_cols) }}
+    from {{ target.schema }}_recharge_dev.recharge__subscription_overview
+),
+
+prod_not_in_dev as (
+    -- rows from prod not found in dev
+    select * from prod
+    except distinct
+    select * from dev
+),
+
+dev_not_in_prod as (
+    -- rows from dev not found in prod
+    select * from dev
+    except distinct
+    select * from prod
+),
+
+final as (
+    select
+        *,
+        'from prod' as source
+    from prod_not_in_dev
+
+    union all -- union since we only care if rows are produced
+
+    select
+        *,
+        'from dev' as source
+    from dev_not_in_prod
+)
+
+select *
+from final