ClickPipes: add get started guide for ABS #5364

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open

Blargian wants to merge 5 commits into ClickHouse:main from Blargian:azure_docs

docs/cloud/guides/data_sources/04_accessing-azure-data-securely.md

-Original file line number
+Diff line change
@@ -0,0 +1,116 @@
+    ---
+    slug: /cloud/data-sources/secure-azure
+    sidebar_label: 'Accessing Azure data securely'
+    title: 'Connecting ClickHouse Cloud to Azure Blob Storage'
+    description: 'This article demonstrates how ClickHouse Cloud customers can access their Azure data securely'
+    keywords: ['ABS', 'azure blob storage']
+    doc_type: 'guide'
+    ---
+    This guide covers how to securely connect ClickHouse Cloud to Azure Blob Storage for data ingestion, external tables, and other integration scenarios.
+    ## Overview {#overview}
+    ClickHouse Cloud can connect to Azure Blob Storage using several authentication methods.
+    This guide will help you choose the right approach and configure your connection securely.
+    Supported use cases:
+    - Reading data from Azure Blob Storage using the [azureBlobStorage table function](/sql-reference/table-functions/azureBlobStorage)
+    - Creating external tables with the [AzureBlobStorage table engine](/engines/table-engines/integrations/azureBlobStorage)
+    - Ingesting data via ClickPipes
+    - [Storing backups in Azure Blob Storage](/cloud/manage/backups/backup-restore-via-ui#azure)
+    :::warning Important Network Limitation
+    When your ClickHouse Cloud service and Azure Blob Storage container are deployed in the same Azure region, IP address whitelisting does not work.
+    This happens because Azure routes same-region traffic through its internal network (VNet + Service Endpoints), bypassing public internet and NAT gateways.
+    As a result, your Azure Storage Account firewall rules based on public IP addresses will not be applied.
+    IP whitelisting works when:
+    - Your ClickHouse Cloud service is in a different Azure region than storage account
+    - Your ClickHouse Cloud service is on AWS/GCP connecting to Azure storage
+    IP whitelisting fails when:
+    - Your ClickHouse Cloud service and storage are in the same Azure region. Use [Shared Access Signatures (SAS)](/integrations/clickpipes/object-storage/abs/overview#authentication) via connection string instead of IP whitelisting or deploy ABS and ClickHouse in different regions.
+    :::
+    ## Network Configuration (Cross-Region Only) {#network-config}
+    :::warning Cross-Region Only
+    This section applies only when your ClickHouse Cloud service and Azure Blob Storage container are in different Azure regions, or when ClickHouse Cloud is on AWS/GCP.
+    For same-region deployments, use SAS tokens instead.
+    :::
+    <VerticalStepper headerLevel="h3">
+    ### Find Your ClickHouse Cloud Egress IPs {#find-egress-ips}
+    To configure IP-based firewall rules, you need to allowlist the egress IP addresses for your ClickHouse Cloud region.
+    Run the following command to retrieve a list of egress and ingress ips per region.
+    Replace `eastus` below with your region to filter out other regions:
+    ```bash
+    # For Azure regions
+    curl https://api.clickhouse.cloud/static-ips.json | jq '.azure[] | select(.region == "westus")'
+    ```
+    You will see something similar to:
+    ```response
+    {
+      "egress_ips": [
+        "20.14.94.21",
+        "20.150.217.205",
+        "20.38.32.164"
+      ],
+      "ingress_ips": [
+        "4.227.34.126"
+      ],
+      "region": "westus3"
+    }
+    ```
+    :::tip
+    See [Azure regions](/cloud/reference/supported-regions#azure-regions) for a list of supported Cloud regions,
+    and the "Programmatic name" column of [Azure regions list](https://learn.microsoft.com/en-us/azure/reliability/regions-list#azure-regions-list-1)
+    for which name to use.
+    :::
+    See ["Cloud IP addresses"](/manage/data-sources/cloud-endpoints-api) for more details.
+    ### Configure Azure Storage Firewall {#configure-firewall}
+    Navigate to your Storage Account in Azure Portal
+. Go to **Networking** → **Firewalls and virtual networks**
+. Select **Enabled from selected virtual networks and IP addresses**
+. Add each ClickHouse Cloud egress IP address obtained in the previous step to the Address range field
+    :::warning
+    Do not add ClickHouse Cloud private IPs (10.x.x.x addresses)
+    :::
+. Click Save
+    See [Configure Azure Storage firewalls docs](https://learn.microsoft.com/en-us/azure/storage/common/storage-network-security?tabs=azure-portal) for more details.
+    </VerticalStepper>
+    ## ClickPipes Configuration {#clickpipes-config}
+    When using [ClickPipes](/integrations/clickpipes) with Azure Blob Storage, you need to configure authentication in the ClickPipes UI.
+    See ["Creating your first Azure ClickPipe"](/integrations/clickpipes/object-storage/azure-blob-storage/get-started) for more details.
+    :::note
+    ClickPipes uses separate static IP addresses for outbound connections.
+    These IPs must be allowlisted if you're using IP-based firewall rules.
+    See ["List of Static IPs"](/integrations/clickpipes#list-of-static-ips)
+    :::
+    :::tip
+    The same-region IP whitelisting limitation mentioned at the start of this document applies to ClickPipes as well.
+    If your ClickPipes service and Azure Blob Storage are in the same region, use SAS token authentication instead of IP whitelisting.
+    :::

...s/data-ingestion/clickpipes/object-storage/azure-blob-storage/02_get-started.md

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -3,20 +3,94 @@ sidebar_label: 'Get started'
  
    description: 'Step-by-step guide to create your first Azure Blob Storage (ABS) ClickPipe.'

    slug: /integrations/clickpipes/object-storage/azure-blob-storage/get-started

    sidebar_position: 1

    title: 'Creating your first Azure Blob Storage ClickPipe'

    title: 'Create your first Azure Blob Storage ClickPipe'

    doc_type: 'guide'

    ---

    import Image from '@theme/IdealImage';

    import cp_step0 from '@site/static/images/integrations/data-ingestion/clickpipes/cp_step0.png'

    import navigateToDatasources from '@site/static/images/integrations/data-ingestion/clickpipes/object-storage/azure-blob-storage/01-navigate-to-datasources.png'

    import createClickpipe from '@site/static/images/integrations/data-ingestion/clickpipes/object-storage/azure-blob-storage/02-create-clickpipe.png'

    import selectBlobStorage from '@site/static/images/integrations/data-ingestion/clickpipes/object-storage/azure-blob-storage/03-select-blob-storage.png'

    import configurationDetails from '@site/static/images/integrations/data-ingestion/clickpipes/object-storage/azure-blob-storage/04-configuration-details.png'

    import chooseDataFormat from '@site/static/images/integrations/data-ingestion/clickpipes/object-storage/azure-blob-storage/05-choose-data-format.png'

    import parseInformation from '@site/static/images/integrations/data-ingestion/clickpipes/object-storage/azure-blob-storage/06-parse-information.png'

    import permissions from '@site/static/images/integrations/data-ingestion/clickpipes/object-storage/azure-blob-storage/07-permissions.png'

    # Creating your first Azure Blob Storage ClickPipe {#creating-your-first-azure-blob-storage-clickpipe}

    **Prerequisites**

    <VerticalStepper type="numbered" headerLevel="h2">

    To follow this guide, you will need:

    - An Azure Blob Storage account

    - [Azure connection string](/integrations/azure-data-factory/table-function#acquiring-azure-blob-storage-access-keys)

    - Container name

    - A running ClickHouse Cloud service

    ## Navigate to data sources {#1-load-sql-console}

    <VerticalStepper headerLevel="h2">

    Select the `Data Sources` button on the left-side menu and click on "Set up a ClickPipe".

    <Image img={cp_step0} alt="Select imports" size="md"/>

    ## Navigate to data sources {#navigate-to-data-sources}

    From the home page of your service, click **Data sources** in the left hand menu.

    Expand the **ClickPipes** dropdown and click **Create ClickPipe**

    <Image img={navigateToDatasources} alt="Navigate to Data Sources" size="md"/>

    <Image img={createClickpipe} alt="Create ClickPipe" size="md"/>

    ## Select the data source {#select-data-source}

    Select **Azure Blob Storage** as data type

    <Image img={selectBlobStorage} alt="Select Azure Blob Storage" size="md"/>

    ## Setup your ClickPipe connection {#setup-connection}

    1. Give your ClickPipe a descriptive name

    2. Select **Connection String** from the authentication method dropdown

    3. Paste your Azure connection string in the field for **Connection string**

    4. Enter your container name

    5. Enter your Azure Blob Storage file path, using wildcards if you want to ingest multiple files

    Optionally, enable continuous ingestion. See ["Continous Ingestion"](/integrations/clickpipes/object-storage/abs/overview#continuous-ingestion) for more details.

    Finally, click **Incoming data**

    <Image img={configurationDetails} alt="Configuration Details" size="md"/>

    ## Select data format {#select-data-format}

    1. Select file type

    2. File compression (`detect automatically`, `none`, `gzip`, `brotli`, `xz` or `zstd`)

    3. Complete additional format specific configuration such as delimiter used for comma-separated formats

    4. Click **Parse information**

    <Image img={chooseDataFormat} alt="Choose Data Format" size="md"/>

    ## Configure table, schema and settings {#configure-table-schema}

    Now you need to create a new table or select an existing one where the incoming data can be stored.

    1. Select whether to upload data to a new table or an existing table

    2. Select the database to use, and which name to give the table if it is a new table

    3. Choose a sorting key or keys

    4. Define any mappings from the source file to destination table for column name, column type, default value and nullability

    5. Finally, specify advanced settings such as the engine type you wish to use, the expression to partition by and the primary key

    <Image img={parseInformation} alt="Parse Information" size="md"/>

    When you have finished configuring your table, schema and settings, click **Details and settings**

    ## Configure permissions {#configure-permissions}

    ClickPipes will set up a dedicated database user for data writing.

    You can select a role for this user.

    For materialized views or dictionary access from the destination table, opt for "Full access".

    <Image img={permissions} alt="Configure Permissions" size="md"/>

    ## Complete setup {#complete-setup}

    Click **Create ClickPipe** to complete setup

    You should now see your ClickPipe in a provisioning status.

    After a few moments it will change from **provisioning** to **completed**.

    </VerticalStepper>

sidebars.js

-Original file line number
+Diff line change
@@ Expand Up / @@ -636,6 +636,7 @@ const sidebars = { @@
                   collapsible: true,
                   items: [
                     'integrations/data-ingestion/clickpipes/object-storage/azure-blob-storage/overview',
+                    'integrations/data-ingestion/clickpipes/object-storage/azure-blob-storage/get-started',
                   ],
                 },
               ],
@@ Expand Down @@

...ion/clickpipes/object-storage/azure-blob-storage/01-navigate-to-datasources.png

Sorry, something went wrong. Reload?