Skip to content

Commit 2989e7d

Browse files
Document known limitations for Kafka Avro data format (timescale#4434)
* Document known limitations for Kafka Avro data format Signed-off-by: Vineeth Pothulapati <vineethpothulapati@outlook.com> * review --------- Signed-off-by: Vineeth Pothulapati <vineethpothulapati@outlook.com> Co-authored-by: atovpeko <anastasiatverdokhleb@gmail.com>
1 parent e4c024e commit 2989e7d

File tree

1 file changed

+106
-3
lines changed

1 file changed

+106
-3
lines changed

migrate/livesync-for-kafka.md

Lines changed: 106 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ import EarlyAccessNoRelease from "versionContent/_partials/_early_access.mdx";
1111

1212
# Stream data from Kafka
1313

14-
You use the Kafka source connector in $CLOUD_LONG to stream events from Kafka into your $SERVICE_SHORT. $CLOUD_LONG connects to your Confluent Cloud Kafka cluster and Schema Registry using SASL/SCRAM authentication and service account–based API keys. Only the Avro format is currently supported.
14+
You use the Kafka source connector in $CLOUD_LONG to stream events from Kafka into your $SERVICE_SHORT. $CLOUD_LONG connects to your Confluent Cloud Kafka cluster and Schema Registry using SASL/SCRAM authentication and service account–based API keys. Only the Avro format is currently supported [with some limitations][limitations].
1515

1616
This page explains how to connect $CLOUD_LONG to your Confluence Cloud Kafka cluster.
1717

@@ -130,7 +130,7 @@ Take the following steps to create a Kafka source connector in $CONSOLE_LONG.
130130
downloaded, then click `Authenticate`.
131131
1. **Set up the Schema Registry**
132132

133-
Enter the Service account ID and the information from the second `api-key-*.txt` that you
133+
Enter the service account ID and the information from the second `api-key-*.txt` that you
134134
downloaded, then click `Authenticate`.
135135
1. **Select topics to sync**
136136

@@ -141,10 +141,113 @@ Your Kafka connector is configured and ready to stream events.
141141

142142
</Procedure>
143143

144+
## Known limitations and unsupported types
144145

146+
The following Avro schema types are not supported:
147+
148+
### Union types
149+
150+
All union types are blocked, including simple nullable fields.
151+
152+
Examples:
153+
154+
- Simple nullable field:
155+
156+
```
157+
{
158+
"type": "record",
159+
"name": "User",
160+
"fields": [
161+
{"name": "id", "type": "string"},
162+
{"name": "age", "type": ["null", "int"]}
163+
]
164+
}
165+
```
166+
167+
- Multiple type union:
168+
169+
```
170+
{
171+
"type": "record",
172+
"name": "Message",
173+
"fields": [
174+
{"name": "content", "type": ["string", "bytes", "null"]}
175+
]
176+
}
177+
```
178+
179+
- Union as root schema:
180+
181+
```
182+
["null", "string"]
183+
```
184+
185+
### Reference types (named type references)
186+
187+
Referencing a previously defined named type by name, instead of inline, is not supported.
188+
189+
Examples:
190+
191+
- Named type definition:
192+
193+
```
194+
{
195+
"type": "record",
196+
"name": "Address",
197+
"fields": [
198+
{"name": "street", "type": "string"},
199+
{"name": "city", "type": "string"}
200+
]
201+
}
202+
```
203+
204+
- Failing reference:
205+
206+
```
207+
{
208+
"type": "record",
209+
"name": "Person",
210+
"fields": [
211+
{"name": "name", "type": "string"},
212+
{"name": "address", "type": "Address"}
213+
]
214+
}
215+
```
216+
217+
### Unsupported logical types
218+
219+
Only the logical types in the hardcoded supported list are supported. This includes:
220+
221+
* decimal, date, time-millis, time-micros
222+
223+
* timestamp-millis, timestamp-micros, timestamp-nanos
224+
225+
* local-timestamp-millis, local-timestamp-micros, local-timestamp-nanos
226+
227+
* uuid, duration
228+
229+
Unsupported examples:
230+
231+
```
232+
{
233+
"type": "int",
234+
"logicalType": "date-time"
235+
}
236+
237+
{
238+
"type": "string",
239+
"logicalType": "json"
240+
}
241+
242+
{
243+
"type": "bytes",
244+
"logicalType": "custom-type"
245+
}
246+
```
145247
146248
[confluent-cloud]: https://confluent.cloud/
147249
[connection-info]: /integrations/:currentVersion:/find-connection-details/
148250
[confluence-signup]: https://www.confluent.io/get-started/
149251
[create-kafka-cluster]: https://docs.confluent.io/cloud/current/clusters/create-cluster.html
150-
[console]: https://console.cloud.timescale.com/dashboard/services
252+
[console]: https://console.cloud.timescale.com/dashboard/services
253+
[limitations]: /migrate/:currentVersion:/livesync-for-kafka/#known-limitations-and-unsupported-types

0 commit comments

Comments
 (0)