Overview
Data Source is the connection entry point for data ingestion tasks. In DataBuddy, data sources are centrally managed by the platform as global objects and are not bound to a specific ingestion task. A single data source definition can be reused by multiple offline synchronization, real-time synchronization, and file upload tasks, as well as external catalogs.
To avoid maintaining data source-related content separately in both the data ingestion module and the platform management module, DataBuddy consolidates common capabilities such as data source creation, editing, connectivity testing, and permission management into the Data Source Management section. This section serves as an entry point from the "Data Ingestion" perspective and guides you to complete specific operations in Platform Management.
How to Use Data Sources in Data Ingestion
Access Scenarios | Role of Data Source in It | Documentation |
Offline data synchronization (single-table/batch/entire-database) | Source or target connection of a sync task | |
Real-time data synchronization (entire-database/sharded-database-and-table) | Source connection of a sync task (the target is typically the built-in DataBuddy data source) | |
Upload files to a table/volume | COS uploads require a COS data source. Local uploads do not require a pre-registered data source. | |
Data Source Types and Read/Write Capabilities | Before configuring a task, confirm whether the selected data source type supports being used as a source/target. | |
Core Data Source Operations (Completed in Platform Management)
All management operations for data sources are performed in Platform Management > Data Source Management. The specific capabilities are as follows:
Capability | Entries |
Create / view / edit / delete data sources; manage ACL permissions and associated tasks. | |
Configure data source connections (data source type, connection parameters, authentication method, and SSM credential hosting). | |
Run connectivity tests on data sources through specified compute resources to verify network reachability and credential validity. | |
Built-in DataBuddy data source (default destination, non-editable) | |
Establish network connectivity for compute resources (when the data source is in a private VPC). | |
Connection Fields for Each Data Source Type
When a data source is created, the Data Source Type drop-down list displays the complete list of types actually supported by the current version. For field descriptions and read/write capabilities of each data source type from the "Data Ingestion" perspective, see the following:
Data Source List and the documentation for each specific data source under it (MySQL / TDSQL-C MySQL / Oracle / PostgreSQL / SQL Server / MongoDB / Kafka, and so on) Recommended Workflow
When connecting to an external data source for the first time, follow this order:
References