Merge pull request #1633 from bruin-data/docs/introduce-lakehouses

Lakehouse Docs (Initial)

Author: aryahassibi

docs/.vitepress/config.mjs

@@ -92,6 +92,7 @@ j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src=
                             {text: "Environments", link: "/getting-started/devenv"},
                             {text: "Variables", link: "/getting-started/pipeline-variables"},
                             {text: "Bruin MCP", link: "/getting-started/bruin-mcp"},
+                            {text: "Lakehouse Support", link: "/getting-started/lakehouse"},
                         ]
                     },
                     {text: "Concepts", link: "/getting-started/concepts"},

docs/.vitepress/theme/custom.css

@@ -56,4 +56,13 @@ html:not(.dark) .vp-code-dark {
 
 .VPSidebarItem .text {
   padding-top: 0.1rem !important;
-}
+}
+
+.lh-check {
+  color: #22c55e;
+  font-weight: 600;
+}
+
+.lh-check::before {
+  content: "\2713";
+}

docs/getting-started/lakehouse.md

@@ -0,0 +1,102 @@
+# Lakehouse Support <Badge type="warning" text="beta" />
+
+> [!WARNING]
+> Lakehouse support is currently in **beta**. APIs and configuration may change in future releases.
+
+Bring lakehouse tables directly into your Bruin pipelines. Query Iceberg and DuckLake data on cloud object storage with a catalog-backed metadata layer, all from the same workflows you already use. This page summarizes supported engines, catalogs, and storage backends.
+
+## Engines and formats
+
+DuckDB and Trino are the engines Bruin supports. In each section, you can discover the lakehouse format + catalog/storage combination supported by Bruin. Visit [DuckDB](../platforms/duckdb.md#lakehouse-support) or [Trino](../platforms/trino.md#lakehouse-support) for Bruin configurations.
+
+### DuckDB [↗](../platforms/duckdb.md#lakehouse-support)
+
+[Iceberg](https://duckdb.org/docs/extensions/iceberg) and [DuckLake](https://duckdb.org/docs/extensions/ducklake) format are natively supported in Bruin.
+
+#### DuckLake
+
+DuckLake uses a DuckDB, SQLite, or Postgres catalog. The table shows supported storage + catalog combinations.
+For more guidance, see DuckLake's [choosing a catalog database](https://ducklake.select/docs/stable/duckdb/usage/choosing_a_catalog_database).
+
+
+| Catalog | S3 |
+|-------------------|----|
+| DuckDB| <span class="lh-check" aria-label="supported"></span> |
+| SQLite | <span class="lh-check" aria-label="supported"></span> |
+| Postgres| <span class="lh-check" aria-label="supported"></span> |
+| MySQL    | Planned |
+
+
+
+#### Iceberg
+
+Iceberg uses the AWS Glue Data Catalog ([AWS Glue Data Catalog](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-data-catalog.html)). The table shows supported storage + catalog combinations.
+
+| Catalog | S3 |
+|-------------------|----|
+| Glue | <span class="lh-check" aria-label="supported"></span> |
+
+
+### Trino [↗](../platforms/trino.md#lakehouse-support)
+
+Trino supports lakehouse access via the [Iceberg connector](https://trino.io/docs/current/connector/iceberg.html) with Glue and [Nessie](https://projectnessie.org/) catalogs. Detailed setup guides are coming soon. Check out [Trino](../platforms/trino.md#lakehouse-support) for Bruin configuration.
+
+| Catalog | S3 |
+|-------------------|----|
+| Glue | <span class="lh-check" aria-label="supported"></span> |
+| Nessie | <span class="lh-check" aria-label="supported"></span> |
+
+## What is a Lakehouse?
+
+A lakehouse combines the scalability of data lakes with the reliability of data warehouses. Data is stored in open formats on object storage (S3, GCS, Azure Blob) while metadata catalogs track schema, partitions, and table history.
+
+<!-- Architecture -->
+
+```mermaid
+%%{init: {"flowchart": {"useMaxWidth": true, "nodeSpacing": 180, "rankSpacing": 80}}}%%
+flowchart TB
+  QE["Query Engine<br/>(DuckDB, Trino, ...)<br/>&nbsp;"]
+  Catalog["**Catalog**<br/>(Glue, REST, ...)<br/><br/>Table metadata, Schema info, Partition info<br/>&nbsp;"]
+  Storage["**Storage**<br/>(S3, GCS, ...)<br/>**Format**(Iceberg, DuckLake)<br/><br/>Parquet files, Manifest files, Data files<br/>&nbsp;"]
+
+  QE --> Catalog
+  QE --> Storage
+```
+
+## Quick Start
+
+Let's add a DuckLake lakehouse configuration to your DuckDB connection (DuckDB catalog + S3 storage):
+
+```yaml
+connections:
+  duckdb:
+    - name: "analytics"
+      path: "./path/to/duckdb.db"
+      lakehouse:
+        format: ducklake
+        catalog:
+          type: duckdb
+          path: "metadata.ducklake"
+        storage:
+          type: s3
+          path: "s3://my-ducklake-warehouse/path"
+          region: "us-east-1"
+          auth:
+            access_key: "AKIA..."
+            secret_key: "..."
+```
+
+Then query your Iceberg tables (defaults to the `main` schema):
+
+```Bruin-sql
+/* @Bruin
+name: lakehouse_users
+type: duckdb.sql
+connection: analytics
+@Bruin */
+
+
+SELECT * FROM users;
+```
+
+See the engine-specific pages [DuckDB](../platforms/duckdb.md#lakehouse-support) or [Trino](../platforms/trino.md#lakehouse-support) for detailed configuration options.

docs/platforms/duckdb.md

@@ -130,3 +130,163 @@ name,networking_through,position,contact_date
 Y,LinkedIn,SDE,2024-01-01
 B,LinkedIn,SDE 2,2024-01-01
 ```
+
+
+## Lakehouse Support <Badge type="warning" text="beta" />
+
+DuckDB can query [Iceberg](https://duckdb.org/docs/extensions/iceberg) and [DuckLake](https://duckdb.org/docs/extensions/ducklake) tables through its native extensions. DuckLake supports DuckDB, SQLite, or Postgres catalogs with S3-backed storage.
+
+### Connection
+
+Add the `lakehouse` block to your DuckDB connection in `.bruin.yml`:
+
+```yaml
+connections:
+  duckdb:
+    - name: "example-conn"
+      path: "./path/to/duckdb.db"
+      lakehouse:
+        format: <iceberg|ducklake>
+        catalog:
+          type: <glue|postgres|duckdb|sqlite>
+          auth: { ... } # optional
+        storage:
+          type: <s3>
+          auth: { ... } # optional
+```
+
+<br>
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `format` | string | Yes | Table format: `iceberg` or `ducklake` |
+| `catalog` | object | Yes | Catalog configuration (Glue for Iceberg, DuckDB/SQLite/Postgres for DuckLake) |
+| `storage` | object | No | Storage configuration (required for DuckLake) |
+
+---
+### Supported Lakehouse Formats
+
+
+
+#### DuckLake
+
+| Catalog | S3 |
+|-------------------|----|
+| DuckDB | <span class="lh-check" aria-label="supported"></span> |
+| SQLite | <span class="lh-check" aria-label="supported"></span> |
+| Postgres | <span class="lh-check" aria-label="supported"></span> |
+| MySQL    | Planned |
+
+
+
+#### Iceberg
+
+| Catalog | S3 |
+|-------------------|----|
+| Glue | <span class="lh-check" aria-label="supported"></span> |
+
+
+For background, see DuckDB's [lakehouse format overview](https://duckdb.org/docs/stable/lakehouse_formats).
+
+---
+### Catalog Options
+For guidance, see DuckLake's [choosing a catalog database](https://ducklake.select/docs/stable/duckdb/usage/choosing_a_catalog_database).
+
+
+#### Glue
+
+```yaml
+catalog:
+  type: glue
+  catalog_id: "123456789012"
+  region: "us-east-1"
+  auth:
+    access_key: "${AWS_ACCESS_KEY_ID}"
+    secret_key: "${AWS_SECRET_ACCESS_KEY}"
+    session_token: "${AWS_SESSION_TOKEN}" # optional
+```
+
+#### Postgres
+
+
+```yaml
+catalog:
+  type: postgres
+  host: "localhost"
+  port: 5432 # optional - default: 5432
+  database: "ducklake_catalog"
+  auth:
+    username: "ducklake_user"
+    password: "ducklake_password"
+```
+
+#### DuckDB
+
+```yaml
+catalog:
+  type: duckdb
+  path: "metadata.ducklake"
+```
+
+`catalog.path` should point to the DuckLake metadata file.
+
+Note that if you are using DuckDB as your catalog database, you're limited to a single client.
+
+#### SQLite
+
+```yaml
+catalog:
+  type: sqlite
+  path: "metadata.sqlite"
+```
+
+
+---
+### Storage Options
+
+#### S3
+
+Bruin currently only supports explicit AWS credentials in the `auth` block.
+Session tokens are supported for temporary credentials (AWS STS).
+
+```yaml
+storage:
+  type: s3
+  path: "s3://my-ducklake-warehouse/path" # required for DuckLake, optional for Iceberg
+  region: "us-east-1"
+  auth:
+    access_key: "${AWS_ACCESS_KEY_ID}"
+    secret_key: "${AWS_SECRET_ACCESS_KEY}"
+    session_token: "${AWS_SESSION_TOKEN}" # optional
+```
+
+---
+### Usage
+
+Bruin makes the lakehouse catalog active for your session and ensures a default `main` schema is available (cannot create Iceberg on S3 schemas/tables, so they must already exist). You can query tables with or without a schema:
+
+```sql
+SELECT * FROM my_table;
+```
+
+You can also use the fully qualified path:
+
+```sql
+SELECT * FROM iceberg_catalog.main.my_table;
+```
+
+> [!NOTE]
+> Unqualified table names resolve to the `main` schema of the active catalog. Use `<schema>.<table>` to target non-main schemas.
+
+#### Example Asset
+
+```bruin-sql
+/* @bruin
+name: lakehouse_example
+type: duckdb.sql
+connection: example-conn
+@bruin */
+
+SELECT SUM(amount) as total_sales
+FROM orders;
+```

docs/platforms/trino.md

@@ -81,3 +81,14 @@ type: trino.sensor.query
 parameters:
     query: select exists(select 1 from upstream_table where inserted_at > '{{ end_timestamp }}')
 ```
+
+
+## Lakehouse Support
+
+> [!WARNING]
+> Trino lakehouse support is available. Detailed setup guides are coming soon.
+
+| Catalog \ Storage | S3 |
+|-------------------|----|
+| Glue | <span class="lh-check" aria-label="supported"></span> |
+| Nessie | <span class="lh-check" aria-label="supported"></span> |