tencent cloud

Tencent Cloud DataBuddy

DokumentasiTencent Cloud DataBuddyData IntegrationOffline Data SynchronizationBatch Creating Single-Table Sync Tasks

Batch Creating Single-Table Sync Tasks

Unduh
Mode fokus
Ukuran font
Terakhir diperbarui: 2026-10-09 16:10:20
Diterjemahkan oleh AI

Overview

Batch creation of single-table sync tasks allows you to generate multiple independent single-table sync tasks for multiple source tables at once, avoiding the repetitive effort of creating them one by one. It is commonly used to ingest a large number of structured tables into the lake at once, or to periodically load multiple unstructured file paths into the target Catalog / Volume.
The structure of each generated single-table task is exactly the same as that of Create Single-Table Sync Task, and each task can be independently edited, debugged, published, and scheduled. The essence of a batch task is to configure once and generate N single-table tasks in batches.

Applicable Scenarios

Batch ingestion from business databases: 100 business tables in MySQL are synced to the lakehouse at once.
Multi-shard same-source sync: tables with the same name from multiple shards (such as db_0 / db_1 / ... / db_n) are merged into the target table.
Batch sync from multiple file paths: files with the same structure in multiple directories on FTP are loaded into Volume in batches.

Prerequisites

Before you start this operation, make sure the following prerequisites are met:
You have activated the DataBuddy service and completed workspace initialization.
The current account has a role related to data access in the target Workspace. For details, see Members and Permission Management.
Compute resources for data ingestion (platform cluster type) have been created.
A source data source connection has been created and tested. For details, see Add a Data Source Connection.
The target Catalog has been planned. If the target Schema or table does not exist, the batch task can automatically create the table during the creation process, which requires the Catalog creation permission.

Use Limits

Limit
Description
Source Type
Structured data sources (same as single-table tasks) + unstructured files (FTP / SFTP / COS / S3 / Azure Blob / HDFS / Rest API)
Target Type
Table (structured scenario) / Volume (file scenario)
Disabled Target Configurations
After the target Catalog is selected, the target Schema, Table (or Volume / Path / File name) are specified item by item through table mapping, and the target form displays "Set in mapping".
Single-task Concurrency
Channel settings apply to each single-table task, equivalent to batch setting the properties of each single-table task.

Operation Steps

Step 1: Entering the Task Creation Page

1. Log in to the DataBuddy console.
2. Switch to the target region and Workspace at the top.
3. Go to Data Integration > Task Management and click Create Sync Task.
4. Select Sync Method: Offline Batch Sync.

Step 2: Filling in Basic Information

Parameter
Description
Required
Default Value
Task Prefix
Prefix used to generate single-table task names. Final single-table task name format: ${Task Prefix}_${Target Table Name}
Yes
-
Access Mode
Batch offline single-table ingestion
Yes
-
Owner
Applied to all generated single-table tasks by default
Yes
Current Creating User
Task Description
Applied to all generated single-table tasks by default
No
-

Step 3: Configuring the Source

Structured Scenarios

Parameter
Description
Required
Source Connection
Same as single-table tasks
Yes
Select Source Table.
Supports selecting multiple tables; supports search by database/table; displays the structure of selected database tables; can be deleted item by item.
Yes
Split Key
Applied to each single-table task by default
No
Filtering Condition
Applied to each single-table task by default
No
Advanced Settings
Applied to each single-table task by default
No

Unstructured File Scenarios

Parameter
Description
Source Connection
Same as single-table tasks
Read Method
Data sync / file sync (same as single-table)
File format.
Files will be parsed in the specified file format.
Parsing Configuration
Same as single-table tasks; applied to each single-table task by default.
Advanced Settings
Applied to each single-table task by default.

Step 4: Configuring the Target

The target Catalog can be a table type or a Volume type. Configure it based on the target Catalog type.
Select the target Catalog as the table type
Parameter
Description
Required
Schema matching policy
Select the Schema matching policy for the sync target. Options: Same as source database / Custom. For Custom, configure it through matching rules.
Yes
Table matching policy
Select the table matching policy for the sync target. Options: Same as source table / Custom. For Custom, configure it through matching rules.
Yes
Write Mode
append / overwrite / upsert
Yes
Advanced Settings
Applied to each single-table task by default
No
Select the target Catalog as the volume type
Parameter
Description
Required
Write Mode
append / overwrite / upsert
Yes
File format.
Files will be written in the specified file format.
Yes
Format Settings
The written file will be assembled in the specified format.
Yes
Mark Completion File
Generate the .ok file after the file is written.
Yes
CTL File
Generate the .ctl file after the file is written.
Depends on file type.
Advanced Settings
Applied to each single-table task by default
No

Step 5: Configuring Table Mapping

Click "Refresh Source and Target Table Mapping" to refresh the mapping between source and target tables. Single-table tasks will be generated based on this mapping later. The table mapping area is dynamically displayed based on the target Catalog type.

Target as a Table-Type Catalog

Field
Description
Task name / source database / source table
Read-only, follows the source configuration
Target catalog
Specify under Catalog
Target Schema
Display the target Schema name based on the schema matching rule configured on the target. If the Schema does not exist, a red "!" prompt is displayed, and you can choose to replace it or create one.
Target Table
Display the target table name based on the table matching rule configured on the target. If the table does not exist, a red "!" prompt is displayed, and you can choose to replace it or create one.
Field mapping rule
Same Name / Same Row Mapping
Computing Resource
Default unified resource for tasks

Target as a Volume-Type Catalog

Field
Description
Source
You can click Add a Row to generate a single-table task.
Target Schema
Specify under Catalog. You can select an existing one or create a new one.
Target Volume
Specify under Schema. You can select an existing one or create a new one.
Volume Path
Subpath within the Volume
File name.
When the path endpoint is a file, the file name variable is automatically displayed by default.

Table Mapping Operations

Operation
Description
Batch Operations
Configure field mapping rules, configure compute resources, and remove selected rows.
Show unconfigured only.
Filter out rows that have not been configured with targets.
Configure target schema in batches.
Configure target schema in batches.
Configure target volume in batches.
Configure target volume in batches.
List operation - Edit
Open the single-table task editor (for details, see Single-Table Overwrite).
List operation - Remove
Remove this mapping entry.

Step 6: Overwriting a Single Table

The settings are the same as those for a single-table task. For details, see Create a Single-Table Task. Some settings cannot be modified.

Step 7: Configuring Runtime Settings

Step 8: Saving and Creating a Single-Table Task

Click Start Batch Creation. The system returns to the task list page, and a progress pop-up appears in the lower-right corner of the page:
Field
Description
Batch Task Status
Creating / Completed / Terminating / Terminated
Batch Task Name
Display batch task prefix.
Progress Metric
Number of successful tasks / Number of exceptions / Total number of tasks
Deleting an alarm policy
Delete a batch task to remove the batch task, but generated single-table tasks are not affected.
Click the "Task List" area to view the creation details of a specific single-table task:
Field
Description
Task Name
If the task is created successfully, the single-table task name is displayed; otherwise, "-" is displayed.
Source / Target
Source table / target table
Status
Displayed under the All tab; switch by All / Succeeded / Failed.

Verification Result

After the batch task is completed, you can confirm in the following ways:
1. The progress pop-up shows Completed and the number of successful tasks = the total number of tasks.
2. Go to Data Ingestion > Task Management task list, search by task prefix, and view the generated single-table tasks.
3. Open any generated single-table task. Its configuration items are the same as those of the batch task, and it can be debugged, published, and scheduled independently.

FAQs

Q: Can a single-table task still be edited independently after a batch task is created?

A: Yes. Generated single-table tasks have the same status as manually created single-table tasks and can be independently edited, debugged, published, and deleted. Modifying a single-table task does not affect the batch task record.

Q: After a single table is overwritten, will modifying global settings in the batch task overwrite it again?

A: A batch task takes effect immediately after it is saved. Generated single-table tasks will not be overwritten by subsequent configuration changes to the batch task. To update them uniformly, edit each single-table task individually.

Q: What data sources are supported by batch tasks?

A: Same as single-table tasks. Structured data sources support batch table synchronization, while unstructured data sources support batch synchronization of multiple paths (see Limitations).

Q: How does field mapping in a batch task handle multiple heterogeneous tables?

A: Field mappings are not configured at the batch level. Each single-table task is configured independently. If table structures differ significantly, split them into multiple batch tasks to process groups of tables with the same structure separately.

References

Table-to-table synchronization
Table-to-Volume synchronization
File-to-table synchronization
File-to-Volume synchronization
Offline Synchronization Task Ops Monitoring
Topik sebelumnya: File-to-Volume SynchronizationTopik selanjutnya: Ops Monitoring for Offline Sync Tasks

Bantuan dan Dukungan

Apakah halaman ini membantu?

masukan

Search
    Enter keywords to search documentation