Overview
You can reference a data source in scenarios such as data ingestion tasks and external catalogs only after you add a data source connection. This document describes how to create a data source connection in DataBuddy, configure authentication information, set advanced parameters, and complete saving and next-step operations.
Prerequisites
Before you start this operation, make sure the following prerequisites are met:
You have activated the DataBuddy service and completed workspace initialization.
You have confirmed that the access address, port, username, password, or other authentication credentials of the target data source are available.
To connect through a Tencent Cloud instance such as Tencent Cloud MySQL, CKafka, or the TCHouse series, make sure that the instance and DataBuddy are in regions with network connectivity.
Use Limits
Limit | Description |
Supported Data Source Types | |
Data Source Name | Consistent with Catalog naming rules; duplicate names are not allowed within the same workspace. |
Credential Management | Password fields support both plaintext passwords and SSM-managed credentials. |
Modifying a Limit | Switching data source type is not allowed when a data source is being edited. |
Operation Steps
The following uses TDSQL-C MySQL access as an example to describe how to create and modify a data source.
Step 1: Go to the Data Source List
2. Switch to the target region and Workspace at the top.
3. Use one of the following entries to go to the data source list:
Entries | Path |
Data Integration Entry | Go to the Data Integration module, confirm that you are on the Access Method Tab, and in the Connectors section, click Data Source Management. |
Global Data Source Management | Platform Management > Data Sources |
Step 2: Select a Data Source Type
1. Click Create Data Source.
2. Select TDSQL-C MySQL from the data source type drop-down list.
Step 3: Configure Connection Information
Go to the connection settings page and enter the information in the following order:
1. Data Source Name: Required.
2. If the selected data source supports multiple connection methods (see the page for supported connection methods), select a Connection Method:
Connection String: Manually enter a JDBC URL, service list, or host port.
Tencent Cloud Instance: Select a region and an instance.
1. Connection Information, using a connection string as an example:
JDBC URL: Required. For example, "jdbc:mysql://168.0.0.1:3306/db_name". Make sure the URL format is correct.
Data Source Version: Required. For example, "8.0.x" indicates support for all 8.0 versions. Because many data source driver versions are available, you can select the closest version.
Step 4: Configure Authentication Information
1. Select the Authentication Method supported by the data source (Username/Password, SSH Key, Kerberos, OAuth 2.0, SAS, SecretId+SecretKey, and so on). If the data source supports only one authentication method, the configuration for that method is displayed directly. For example, if mysql supports only username & password, only the username and password fields are displayed.
2. Fill in the corresponding authentication fields:
Username: Enter the username for the TDSQL-C MySQL database. We strongly recommend that you create a dedicated account in the database before creating the connection to control the final data permissions.
Password: Both regular passwords and SSM hosting are supported. For a regular password, simply enter the password. When editing, you cannot view the previously set password, but you can modify it.
SSM Hosting: If your password is hosted in SSM, you can switch to "Obtain from SSM" and fill in the Region, Credential Name, and Credential Version. DataBuddy dynamically obtains the actual value when in use. Note that SSM provides two modes: custom credentials and database credentials. wedata supports only the custom credential mode.
1. If the data source supports other authentication methods, see the on-screen prompts to complete the configuration.
Step 5: (Optional) Configure Advanced Parameters
Some data sources (such as MySQL, PostgreSQL, Hive, and MongoDB) support configuring extended parameters in Advanced Settings. For specific extended parameters, see the official documentation of each data source. These parameters will be appended to the JDBC URL, for example, ?param1=value1¶m2=value2.
Step 6: Connectivity Test
1. Click Test Connectivity at the bottom of the page.
3. If the test fails, adjust the connection and authentication fields based on the returned failure cause, and test again until it succeeds.
Note:
A connectivity test only verifies the current network reachability and authentication validity. It does not check whether the target database tables exist. A task may still fail at runtime if the permissions or environment of the target database tables change.
Step 7: Save the Data Source
Select one of the following save methods based on your use case:
Create and Ingest Data: Creates a data source and redirects to the ingestion task creation page, using the data source as the source end (recommended for first-time creation).
Create Only: Creates only the data source, allowing users to create sync tasks later.
A success message appears after the data source is saved successfully. If saving fails, the specific failure cause is displayed.
Verification Result
After the data source is created successfully, you can confirm it in the following ways:
1. You can see the newly created data source in the data source list, with the type, name, and owner displayed correctly.
2. Click the data source name to go to its details page. The information in Overview matches what you entered, and the Associated Tasks list is initially empty.
3. When a data ingestion task is created, the Data Source Connection drop-down list for the selected data source type includes the name of the data source connection.
Editing and Deleting Data Sources
Operation | Entries | Description |
Editing an alarm policy | Operation column on the list page Edit; | Switching data source type is not allowed. Changes to other fields take effect immediately after saving. Before editing a data source, confirm that the operation will not affect existing tasks. |
Deleting an alarm policy | Operation column on the list page Delete; | The operation is irreversible. Before deleting a data source, confirm that the operation will not affect existing tasks. |
Warning
Before deleting a data source, check the Associated Tasks tab on the details page to confirm whether any tasks still depend on the data source. After deletion, sync tasks that depend on the data source will report an error at runtime: "Data source connection exception: The data source does not exist or the current account does not have permission to access the data source."
FAQs
Q: Where can I find complete field descriptions given the significant differences in fields among data source types?
A: For details, see the frontend page.
Q: Is it mandatory to use SSM for password field management?
A: No. In regular password mode, you can simply enter the password to use it. If you have compliance or centralized credential management requirements, we recommend using SSM hosting instead to avoid storing passwords in plaintext in the data source configuration.
Q: Why does a task still use the old connection data after I modify the data source information?
A: This mainly occurs in real-time task scenarios. Once a real-time task is started, its running information does not change. The running information is updated only after a restart, so stop the real-time task and then restart it. Offline tasks automatically update data source information during each scheduling.
Q: If the connectivity test fails when a data source is being saved, will the save operation be blocked?
A: No. Connectivity testing and saving are two independent operations. You can save the data source first and then troubleshoot connectivity issues. However, we recommend that you complete the test before saving to avoid task failures at runtime.
References