debezium/dbz#1569 Add a blogpost for platform support for connection#1215
debezium/dbz#1569 Add a blogpost for platform support for connection#1215indraraj wants to merge 1 commit intodebezium:developfrom
Conversation
Signed-off-by: indraraj <indraraj14@gmail.com>
|
🎊 PR Preview has been successfully built and deployed to https://debezium-debezium-github-io-preview-pr-1215.surge.sh |
| Since introducing the Debezium Management Platform (Debezium Platform), our goal has been to simplify how you build and manage CDC data pipelines—so you can focus on how your data flows, rather than repeatedly configuring the same infrastructure details. | ||
|
|
||
| Earlier, connection-related properties (such as hostnames, ports, credentials, or authentication details) were defined directly as part of the Source or Destination configuration. | ||
| While this worked, it often led to duplication, harder maintenance of the same connection details across multiple instances. |
There was a problem hiding this comment.
| While this worked, it often led to duplication, harder maintenance of the same connection details across multiple instances. | |
| While this worked, it often led to duplication and harder maintenance of the same connection details across multiple instances. |
| author: indrashukla | ||
| --- | ||
|
|
||
| Since introducing the Debezium Management Platform (Debezium Platform), our goal has been to simplify how you build and manage CDC data pipelines—so you can focus on how your data flows, rather than repeatedly configuring the same infrastructure details. |
There was a problem hiding this comment.
| Since introducing the Debezium Management Platform (Debezium Platform), our goal has been to simplify how you build and manage CDC data pipelines—so you can focus on how your data flows, rather than repeatedly configuring the same infrastructure details. | |
| The Debezium Platform aims to simplify the building and management of CDC data pipelines by streamlining infrastructure setup. |
|
|
||
| Since introducing the Debezium Management Platform (Debezium Platform), our goal has been to simplify how you build and manage CDC data pipelines—so you can focus on how your data flows, rather than repeatedly configuring the same infrastructure details. | ||
|
|
||
| Earlier, connection-related properties (such as hostnames, ports, credentials, or authentication details) were defined directly as part of the Source or Destination configuration. |
There was a problem hiding this comment.
| Earlier, connection-related properties (such as hostnames, ports, credentials, or authentication details) were defined directly as part of the Source or Destination configuration. | |
| Earlier, connection-related properties (such as hostnames, ports, credentials, and authentication details) were defined directly in the Source or Destination configuration. |
| Earlier, connection-related properties (such as hostnames, ports, credentials, or authentication details) were defined directly as part of the Source or Destination configuration. | ||
| While this worked, it often led to duplication, harder maintenance of the same connection details across multiple instances. | ||
|
|
||
| To address this, we are introducing **Connections** as a first-class entity in the Debezium Platform. |
There was a problem hiding this comment.
| To address this, we are introducing **Connections** as a first-class entity in the Debezium Platform. | |
| To address this, we are introducing **Connections** as a first-class citizen in the Debezium Platform, making it easier for you to manage connection details in one place and avoid repetitive setup. |
|
|
||
| To address this, we are introducing **Connections** as a first-class entity in the Debezium Platform. | ||
|
|
||
| Connections allow you to define, validate, and reuse connection details independently, and then reference them from any number of Sources and Destinations. |
There was a problem hiding this comment.
| Connections allow you to define, validate, and reuse connection details independently, and then reference them from any number of Sources and Destinations. | |
| Connections allow you to define, validate, and reuse connection details independently, reducing configuration errors and saving time by referencing them in multiple Sources and Destinations. |
| Instead of embedding these details inside each Source or Destination: | ||
|
|
||
| * You create a Connection once | ||
| * Validate it independently | ||
| * Reuse it across multiple Sources and Destinations | ||
|
|
||
| This makes pipelines easier to manage, safer to modify, and faster to set up. | ||
|
|
||
| == Why introduce Connections? | ||
|
|
||
| Separating connection details from Source and Destination configurations provides several benefits: | ||
|
|
||
| **Reusability**:: | ||
| A single Connection can be reused across multiple Sources or Destinations. | ||
|
|
||
| **Centralized management**:: | ||
| Update credentials or connection parameters in one place without editing every pipeline. | ||
|
|
||
| **Early validation**:: | ||
| Validate a Connection before it is ever used by a Source or Destination. | ||
|
|
||
| **Cleaner configurations**:: | ||
| Source and Destination definitions focus only on CDC and data flow logic, not infrastructure details. |
There was a problem hiding this comment.
The initial 3 bullet points and the section on the why are effectively the same, with the latter just providing a bit more concrete detail. Perhaps the following is sufficient?
| Instead of embedding these details inside each Source or Destination: | |
| * You create a Connection once | |
| * Validate it independently | |
| * Reuse it across multiple Sources and Destinations | |
| This makes pipelines easier to manage, safer to modify, and faster to set up. | |
| == Why introduce Connections? | |
| Separating connection details from Source and Destination configurations provides several benefits: | |
| **Reusability**:: | |
| A single Connection can be reused across multiple Sources or Destinations. | |
| **Centralized management**:: | |
| Update credentials or connection parameters in one place without editing every pipeline. | |
| **Early validation**:: | |
| Validate a Connection before it is ever used by a Source or Destination. | |
| **Cleaner configurations**:: | |
| Source and Destination definitions focus only on CDC and data flow logic, not infrastructure details. | |
| == Why introduce Connections? | |
| Rather than defining connection details inside each Source or Destination, separating them as a first-class citizen provides a host of benefits: | |
| **Reusability**:: | |
| A single Connection can be reused across multiple Sources or Destinations. | |
| **Centralized management**:: | |
| Update credentials or connection parameters in one place without editing every pipeline. | |
| **Early validation**:: | |
| Validate a Connection before it is ever used by a Source or Destination. | |
| **Cleaner configurations**:: | |
| Source and Destination definitions focus only on CDC and data-flow logic, not on infrastructure details. | |
| All these benefits make creating and managing pipelines easier, safer, and faster. |
| * The platform can successfully connect to the target system | ||
| * Authentication and network settings are correct | ||
|
|
||
| This helps catch issues early—before a Source or Destination is created or a pipeline is executed. |
There was a problem hiding this comment.
| This helps catch issues early—before a Source or Destination is created or a pipeline is executed. | |
| This helps catch issues early—before a Source or Destination is created or a pipeline is started. |
| Once a Connection is created and validated, it becomes available for selection when creating Sources and Destinations. | ||
|
|
||
| Instead of entering connection properties again, you simply reference an existing Connection. | ||
| The Source or Destination then uses that Connection at runtime. |
There was a problem hiding this comment.
| Once a Connection is created and validated, it becomes available for selection when creating Sources and Destinations. | |
| Instead of entering connection properties again, you simply reference an existing Connection. | |
| The Source or Destination then uses that Connection at runtime. | |
| When defining a Source or Destination, all validated Connection definitions are available in a drop-down for quick access, making pipeline setup and management fast and straightforward. |
| If you are already familiar with creating Sources and Destinations in the Debezium Platform, the overall workflow remains the same. | ||
|
|
||
| The key difference is that connection-related properties are no longer embedded directly in the resource configuration. | ||
| Instead, they are managed through reusable Connections, keeping configurations modular and easier to evolve over time. |
There was a problem hiding this comment.
| If you are already familiar with creating Sources and Destinations in the Debezium Platform, the overall workflow remains the same. | |
| The key difference is that connection-related properties are no longer embedded directly in the resource configuration. | |
| Instead, they are managed through reusable Connections, keeping configurations modular and easier to evolve over time. | |
| If you know how to create Sources and Destinations in Debezium, the workflow is unchanged, and updating to use Connection definitions is quick and painless. | |
| First, create a Connection definition with all the connection properties your existing pipeline uses. After you create and validate the Connection, simply edit the workflow and select the newly defined Connection definition to link it to the pipeline. The pipeline then uses the reusable Connection definition moving forward. |
|
|
||
| == More coming soon! | ||
|
|
||
| Connections are a foundational step toward making the Debezium Platform more modular, maintainable, and scalable. |
There was a problem hiding this comment.
Connections lay the foundation for making the Debezium Platform more modular, maintainable, and scalable.

No description provided.