This document describes the general workflow for creating, viewing, editing, and deleting data sources in a workspace, and explains how to manage data source details, owners, and associated tasks.
Overview
Data Source can be reused across multiple data ingestion tasks, external catalogs, and other scenarios within a workspace once defined. This document focuses on the general workflow for global data source configuration.
Data source connections are registered at the workspace level. Once registered, they can be used in the following scenarios:
Data ingestion tasks (offline, real-time sync, file-to-table, COS file-to-table, and so on).
The Catalog module creates External catalogs based on connections.
Global data source management (this document).
Key Concepts
To understand data source configuration, you need to be familiar with the following core concepts:
Data source connection (Connection): An object that describes the network address, authentication information, parameters, and other details required to communicate with an external system.
Connection Method: Different data sources support different connection methods. Common methods include Connection String (via JDBC URL or service list) and Tencent Cloud Instance (based on region and instance ID).
Authentication Method: Supports multiple authentication methods such as Username/Password, SSH Key, Kerberos, OAuth 2.0, SAS, and SecretId/SecretKey, depending on the data source type and enterprise security policy.
SSM Credential Hosting: Sensitive credentials such as passwords and tokens can be hosted in Tencent Cloud Credential Management (SSM) and dynamically obtained by the triplet of region, credential name, and credential version. Note that currently only the Custom Credential type of SSM is supported, while the Database Credential type is not supported.
Connectivity Test: Verifies the network reachability and authentication validity between DataBuddy and the target system before a data source is saved or during use.
Data Source Details: Each data source contains information such as Overview, Permissions, and Associated Tasks, facilitating unified management and auditing.
For the complete glossary, see Glossary. Sub-Feature Navigation
Sub-feature | Description | Documentation |
Add or Modify Data Source Connection | Create a data source, configure connection information and authentication, and save or save and create a task. | |
Data source connectivity test | Verify connectivity of the network and authentication during creation/editing, or on the data source details page. | |
Data Source Details | View data source overview, permissions, and associated tasks; edit or delete the data source. | Corresponding section in this document |
Prerequisites
You have joined the target workspace and have the Can Create Data Source role permission. By default, both workspace administrators and workspace members have this permission.
Have the connection information of the target data source ready, including the connection string, Tencent Cloud instance ID, username, password, credentials, and so on.
Entries
You can go to data source configuration from the following entries:
|
Data Integration Entry | Go to the Data Integration module, confirm that you are on the Access Method Tab, and in the Connectors section, click Data Source Management. |
Global Data Source Management | Platform Management > Data Source Management |
Recommended Learning Path
If you are configuring a DataBuddy data source for the first time, read the documentation in the following order:
3. Create Data Ingestion Task: On the data source details page, click Create Task to go to task configuration.
Viewing the Data Source List
The list displays all data sources created in the current workspace:
Field | Description |
Data Source Name | Display name of the data source. Click it to go to the details page. |
Type | Data source type (MySQL / Kafka / COS, and others), with filtering support |
Owner | Owner of the data source, with filtering support |
Creation Time | Data Source Creation Time |
Description | Remarks |
Operation | Edit, Delete, More (Permissions, etc.) |
The following list operations are supported:
Search: Enter a data source name keyword in the search box in the upper-right corner to perform a fuzzy search.
Filter: Click the filter icon in the Type or Owner column header to filter by value.
Refresh: Click the refresh button next to the search box to reload the list.
Create Data Source: Click + Create Data Source in the upper-right corner to enter the creation process.
Creating and Editing a Data Source
Viewing Data Source Details
Click a data source name on the list page to go to the details page. The details page contains the following information and features:
Overview
Field | Description |
Data Source Type | Not editable |
Data Source Name | Not editable |
Integration information | Display the connection string / instance information. Sensitive fields such as passwords and keys are displayed as ***. |
Owner | Defaults to creator |
Creator. | Not editable and will not be cleared when the creator account is deleted. |
Description | Click to enter edit mode and modify data source remarks. |
Connectivity test | Connectivity test information |
Tab Page
Tab | Content. |
Overview | Connection information and basic fields above |
Permissions | |
Associated tasks | List all tasks in the current workspace that reference this data source, including the deduplicated collection of edited, published, and running versions. |
Operation Item
The following operations are supported in the upper-right corner of the details page:
Create Data Ingestion Task: Redirects to the task creation page and uses this data source as the task's data source.
Permissions: Configure permissions for the current data source.
Associated Task List
The following fields are displayed by default in the Associated Tasks Tab:
|
Task Name | Click to go to the latest edited version of the task. |
Task ID | Associated task ID |
Task type | Offline ingestion / real-time ingestion / file upload, and more |
Published Version | Currently published version number; if the current edited version differs from the published version, a ! indicator is displayed. |
Creator. | Creator of the associated task |
Creation Time | Creation time of the associated task |
Last Modified By / Last Modified At | Last edited information |
Note:
If a task is bound to different data sources in different versions (edit, publish, run), this list pulls tasks from all related versions and displays them after deduplication. Click a task name to go to the latest edited version of the task, which may be inconsistent with the current data source.
Delete a data source
Access:
In the Operation column on the list page, click Delete.
Deletion process:
1. A second confirmation window appears, prompting: Are you sure you want to delete the data source? After deletion, services associated with this connection will become abnormal and cannot be recovered.
2. You can select:
Continue to Delete: Force delete this data source.
View Associated Tasks: Redirects to the Associated Tasks Tab on the details page to view which tasks are referencing this data source.
If deletion fails (for example, the current user does not have deletion permission), a message appears: Deletion failed:${failure reason}. Click Details to view the original error message.
Use Limits
Data source type cannot be modified: In edit mode, the data source type field cannot be changed. To switch the type, you must delete the data source and recreate it.
The built-in DataBuddy data source cannot be edited or deleted.
Owner is required: The owner cannot be empty, and at least one owner must be specified. If the owner's account is removed from the workspace, a red indicator appears.
FAQs
Q1: Why can't I see one of my data sources in the data source list?
Data sources use the ACL permission model. If you do not have any permission on a data source (including Can View), it will not be displayed. Contact the data source owner to grant you the Can View or Can Use permission. For details, see Members and Permission Management. Q2: How can multiple workspaces share the same data source?
Data sources are workspace-level configurations and cannot be shared across workspaces. If multiple workspaces need to use the same data source, create it separately in each workspace. We recommend that you store the account and password in SSM and then reference the same SSM credential in each workspace.
Q3: What are the benefits of using SSM credentials?
SSM credentials enable centralized password management. When passwords are rotated, you only need to update the credentials in SSM without modifying the data source configuration in DataBuddy. In addition, passwords are not stored in plaintext in the DataBuddy database, which better meets enterprise security compliance requirements.
References