You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
* Update S3 connector docs with GA status
* chore: small updates for language and to match ui.
* last updates
* chore: small edit.
---------
Signed-off-by: Iain Cox <iain@timescale.com>
Co-authored-by: billy-the-fish <iain@tigerdata.com>
Co-authored-by: Iain Cox <iain@timescale.com>
Co-authored-by: atovpeko <anastasiatverdokhleb@gmail.com>
import PrereqCloud from "versionContent/_partials/_prereqs-cloud-only.mdx";
11
-
import EarlyAccessNoRelease from "versionContent/_partials/_early_access.mdx";
12
11
import NotSupportedAzure from "versionContent/_partials/_not-supported-for-azure.mdx";
13
12
14
13
# Sync data from S3
@@ -22,7 +21,7 @@ You can use the $S3_CONNECTOR to synchronize your existing and new data. Here's
22
21
* Sync data from an S3 bucket instance to a $SERVICE_LONG:
23
22
- Use glob patterns to identify the objects to sync.
24
23
- Watch an S3 bucket for new files and import them automatically. It runs on a configurable schedule and tracks processed files.
25
-
-**Important**: The connector processes files in [lexicographical order][lex-order]. It uses the name of the last file processed as a marker and fetches only files later in the alphabet in subsequent queries. Files added with names earlier in the alphabet than the marker are skipped and never synced. For example, if you add the file Bob when the marker is at Elephant, Bob is never processed.
24
+
-**Important**: the connector processes files in [lexicographical order][lex-order]. It uses the name of the last file processed as a marker and fetches only files later in the alphabet in subsequent queries. Files added with names earlier in the alphabet than the marker are skipped and never synced. For example, if you add the file Bob when the marker is at Elephant, Bob is never processed.
26
25
- For large backlogs, check every minute until caught up.
27
26
28
27
* Sync data from multiple file formats:
@@ -37,6 +36,9 @@ The $S3_CONNECTOR continuously imports data from an Amazon S3 bucket into your d
37
36
38
37
**Note**: the connector currently only syncs existing and new files—it does not support updating or deleting records based on updates and deletes from S3 to tables in a $SERVICE_LONG.
39
38
39
+
The $S3_CONNECTOR is **Generally Available** and production ready. If you have any questions or feedback, join us in
40
+
the [$COMPANY community][livesync-tiger-community].
41
+
40
42
## Prerequisites
41
43
42
44
<PrereqCloud />
@@ -129,31 +131,52 @@ To sync data from your S3 bucket to your $SERVICE_LONG using $CONSOLE:
129
131
130
132
1. Choose `Create a new table for your data` or `Ingest data to an existing table`.
131
133
1. Choose the `Data type` for each column, then click `Continue`.
132
-
1. Choose the interval. This can be a minute, an hour, or use a [cron expression][cron-expression].
134
+
1. Configure the insert behavior when there is a conflict, then click `Continue`.
135
+
1. Choose the polling interval. This can be a minute, an hour, or a [cron expression][cron-expression].
133
136
1. Click `Start Connector`.
134
137
135
138
$CONSOLE starts the connection between the source database and the target $SERVICE_SHORT and displays the progress.
136
139
137
140
1.**Monitor synchronization**
138
141
139
-
1. To view the amount of data replicated, click `Connectors`. The diagram in `Connector data flow` gives you an overview of the connectors you have created, their status, and how much data has been replicated.
142
+
The $S3_CONNECTOR provides comprehensive observability that gives you maximum visibility on connector performance.
143
+
This includes summarized insights into connector state, quick actions, filtering and search to easily find specific
144
+
files, and detailed lifecycle tracking as each file is imported.
145
+
146
+
1. To view the amount of data replicated, click `Connectors`.
147
+
148
+
The diagram in `CONNECTOR DATA FLOW` shows the connectors you have created, their status, and how much data has been replicated.
1. To edit the connector, click `Connectors` > `Source connectors`. Open the three-dot menu next to your connector in the table, then click `Edit` and scroll down to `Modify your Connector`. You must pause the connector before editing it.
177
+
1. To edit the connector, click `Connectors` > `Source connectors`. Open the three-dot menu next to your connector in the table, then click `Edit`. Select `Connector settings`. You must pause the connector before editing it.
1. To pause or delete the connector, click `Connectors` > `Source connectors`, then open the three-dot menu on the right and select an option. You must pause the connector before deleting it.
0 commit comments