Overview
A real-time whole-database multi-table task synchronizes one or more tables from a source database to the DataBuddy data lakehouse in real time through CDC links. A single task configuration can cover the entire database or multiple specified tables, supporting automatic table creation, DDL synchronization, and automatic table addition or removal during runtime. This document describes the complete configuration process of the task.
Prerequisites
Before you start this operation, make sure the following prerequisites are met:
You have activated the DataBuddy service and completed workspace initialization.
A real-time cluster compute resource for real-time synchronization has been created. For details, see Create a Compute Resource. The source has enabled CDC capabilities, such as setting MySQL Binlog to ROW, Oracle LogMiner, and SQL Server CDC. For specific requirements, see the sections for each data source.
The target Catalog has been planned. To enable automatic table creation, the current account must have write and table creation permissions on the target Catalog.
Use Limits
Limit | Description |
Source Type | MySQL, Tencent Cloud MySQL, TDSQL-C MySQL, PostgreSQL, Oracle, SQL Server, Kafka, CKafka, MongoDB, Tencent Cloud MongoDB |
Target Type | Iceberg native tables under the DataBuddy data lakehouse Catalog |
Computing Resource | Only real-time cluster computing resources are supported. |
Resource consumption | Single-task range: 1 to 256 CUs, with a step size of 1 CU |
Maximum Number of Source Tables | A single task can select up to 5,000 tables. |
Automatic table creation | When Kafka is used as the source, automatic table creation is not supported. Tables must be created in the destination in advance. |
Mapping Match Preview | Kafka and MongoDB do not support mapping match preview. |
Automatic table addition and removal | Only the regex matching mode supports automatically adding newly matched tables during running. The specified tables mode requires pausing the task before manually adding or removing tables. |
Operation Steps
A real-time whole-database multi-table task uses a wizard-based configuration, which includes five steps: Basic Information → Source → Target → Data Verification → Runtime Settings.
Step 1: Entering the Task Creation Page
2. From the workspace list, go to the target workspace.
3. Go to Data Integration > Ingestion Tasks, and click Create Task.
4. After selecting the data source type, select Real-time Whole-Database Multi-Table Ingestion as the ingestion method.
Step 2: Configuring Basic Information
After filling in the basic task information, click Next. For parameter details, see Basic Settings. Step 3: Configuring the Source
1. Select a data source connection to run a connectivity test. If the test fails, click Retry to view the specific cause.
2. Select the source table:
Specified Tables: Filter by database name and table name, and then select the specific tables.
Regex-Matched Tables: Enter regular expressions for database names and table names. New tables that match the expressions during runtime are automatically ingested.
1. Select the read mode: Full and Incremental (default) or Incremental Only.
2. Keep at least one of the following in Subscription Type: Insert, Update, or Delete.
3. To configure advanced behaviors such as setting sharding parameters and controlling metadata field writes, expand Advanced Settings.
Tip
Configurations vary by data source type. For details, see the documentation for each data source type listed under Data Sources, such as Kafka Data Source. Step 4: Configuring the Target
1. Basic Configuration: Select the target Catalog and system fields.
2. Table Creation Configuration: Currently, only Iceberg native tables are supported. For Kafka sources, this section is disabled and tables must be created manually.
3. Mapping Match Preview: Check the mapping relationship between each source table and its target table. You can click Edit to adjust field mappings, target Schema/table, and synchronization primary keys.
Note:
After field mappings are modified, DDL changes no longer support adding columns, dropping columns, renaming columns, or modifying column types. If such changes occur in the source table, the task may report an error. Use this only when differentiated mappings are truly needed.
Step 5: Data Reconciliation (Optional)
Step 6: Configuring Runtime Settings
1. Configure the DDL Message Processing Policy: determines whether to automatically synchronize, ignore, or stop the task when a DDL message is received from the source. (Some data sources do not have this configuration. The actual page prevails.)
2. Configure the Task Running Policy: Checkpoint interval, write exception policy, and maximum restart attempts.
3. Configure Alarm Notification: Select an existing alarm channel and enable monitoring metrics. For detailed rules, see Alarm Notification. 4. To adjust pass-through parameters, expand Advanced Settings.
Step 7: Saving or Publishing
Operation | Action | Task Status |
Save only | Save configuration without publishing | Unpublished |
Save and publish | Save and submit for publishing | Pending start |
Parameter Description
Basic Settings
Parameter | Description | Required | Default Value |
Access Type | Select Source Type | Yes | - |
Access Mode | Real-time whole-database multi-table ingestion / Real-time sharded database and table ingestion | Yes | Real-time whole-database multi-table ingestion |
Task Name | Uppercase and lowercase letters, Chinese characters, digits, and underscores, with a length not exceeding 256. | Yes | ${Data Source Type}_YYYYMMDD_HHmmss
|
Computing Resource | Real-time cluster computing resources used for real-time synchronization | Yes | - |
Resource consumption | Number of CUs, ranging from 1 to 256, with a step size of 1. | Yes | - |
Owner | Account used to run the task; must have access permissions on the target end. | Yes | Current Creating User |
Task Description | No more than 200 characters | No | - |
Source Settings
Parameter | Description | Required |
Data source. | Data source connection of a matching type in the current Workspace | Yes |
Source Table | | Yes |
Read Mode | Full + incremental / Incremental only | Yes |
Subscription Type | Insert / Update / Delete, keep at least one. | Yes |
Advanced Settings | Advanced parameters specific to the data source | No |
Two Selection Modes for the Source Table
Mode | Action | Automatic table addition and removal during running |
Specified Tables | Filter by database name and table name, then select; up to 5,000 tables. | ❌ Pause the task before manual editing. |
Regex-matched Tables | Enter the database and table regex; new tables that match during the running period are automatically added. | ✅ Automatic addition and removal; added or removed tables must be within the regular expression scope. |
Target Settings
Complete the basic configuration
Parameter | Description | Required |
Data Location | Select the target Catalog. Local creation is supported (requires Catalog creation permission). | Yes |
Schema matching policy | Control the naming rule of the target Schema. | Yes |
Table matching policy | Control the naming rule of the target table. | Yes |
System Fields | | No |
System Fields (Metadata Fields)
Built-in fields that can be selected and written to the target table:
Field Name | Type | Description |
datasource_wedata_di
| string | Data Source Name |
database_wedata_di
| string | Database name. |
table_wedata_di
| string | Data Table Name |
sequence_id_wedata_di
| string | Incremental event record ID, unique and incrementing. |
type_wedata_di
| string | Operation Type |
ts_wedata_di
| timestamp | Log Time |
processing_time_wedata_di
| timestamp | Processing time (machine local) |
before_image_wedata_di
| string | Whether it is a pre-update record, with values Y / N |
after_image_wedata_di
| string | Whether it is a post-update record, with values Y / N |
Note:
If you deselect a system field after the task has been published and is running, the field in the target table will not be deleted, but subsequent data will be written as null. Confirm this before saving.
Table Creation Configuration
Parameter | Description |
Table creation type | Currently, only native tables (Iceberg) are supported. |
Field mapping | The platform populates field type mappings based on default rules, and supports modifications as needed. |
Mapping Match Preview
Preview the mapping relationship between each source table and its target table. The columns include Sequence Number, Source Database, Source Schema (if any), Source Table, Target Schema, Target Table, Synchronization Primary Key, Table Creation Method, Field Mapping, and Operation.
Item | Description |
Synchronization primary key | Auto-fill: use the primary key of an existing table to fill the primary key of this table; for auto-created tables, fill the primary key of the table to be created. If no primary key exists, select 1+ columns as a joint substitute primary key. |
Field mapping | Default: Default mapping; after Edit is clicked and modifications are made, it changes to Custom mapping (DDL change capability is limited). |
Operation (remove) | Available only in specified tables mode; the remove button is grayed out in regex matching mode. |
Operation (edit) | Open the right drawer to display and allow adjustment of field mappings based on the target DataBuddy type. |
Preview is not supported when the data volume is too large. You can view the actual link details on the Ops details page after the task is published and running.
Running Settings
DDL Message Processing Policy
Processing method when a DDL change is received from the source (for the DDL types supported by different links, see the corresponding data source sections, such as MySQL Data Source): Policy | Action |
Automatic synchronization | Automatically execute the corresponding DDL on the destination. |
Ignored. | Skip DDL; subsequent data may fail due to Schema mismatch. |
Task Execution Policy
Parameter | Description |
Checkpoint Interval | State snapshot write interval |
Write Exception Policy | Handling method for a single write failure |
Maximum Restart Attempts | Maximum number of automatic restarts after task exceptions |
Alarm Notification
Integration with notification channels is supported. After a channel is added in the basic platform, it automatically appears in the options. Alarm Type | Description | Key threshold |
Task failed. | Trigger an alarm when a task enters the failed state | - |
Service Latency | Synchronization link latency exceeds the threshold. | Latency threshold (seconds), duration (minutes) |
Number of Failovers | The number of restarts within the observation window exceeds the threshold. | Observation window (minutes), number of restarts (times) |
DDL Notification | Notify when the selected DDL type occurs. | Multiple types: Add column, Delete column, Rename column, Modify column type, Delete table, Truncate table, Rename table |
Parameter | Description | Value Range | Default Value | Note. |
Latency threshold (seconds) | Lower latency threshold for service latency alarms | 1 to 86400, with a step size of 1 | 120 | Recommended to be at least 2 times the Checkpoint interval. |
Duration (minutes) | Alarm after the latency threshold is continuously exceeded for a specified duration | 1 to 60, with a step size of 1 | 5 | A value that is too short may cause false alarms due to network jitter. |
Observation window (minutes) | Failover Statistics Window | 5 to 60, with a step size of 1 | 10 | Recommended range: 10 to 30 minutes. |
Number of restarts (times) | Alarm threshold for the number of restarts within the observation window | 1 to 20, with a step size of 1 | 3 | Recommended to be less than the upper limit of the number of automatic restarts. |
Task Version Management
You can view the version in the upper-right corner of the task editing page:
Save to generate a version immediately, recording the save time and the user who saved it.
Click a specific version to enter read-only mode. Click Return to This Version to switch to edit mode, and a new version is generated after you edit and save.
The version of a running task is marked with a Production label.
Changes made through the Modify Owner and Modify Alarm entries in the task list do not generate a new version.
Verification Result
After the operation is completed, you can confirm in the following ways:
1. You can see the task in the Data Integration > Access Tasks > Real-Time list, and its status is as expected:
Unpublished: Save only.
Pending Start: Published. You can click Start to enter the running state.
1. After the task is started, its status changes from In Progress to Running. Click the task name to go to the Ops details page and view the running progress. For more information, see Real-Time Synchronization Task Ops Monitoring. 2. View the automatically created target table under the target Catalog. Data is written after the full phase is completed, and changes are continuously appended during the incremental phase.
FAQs
Q: A New Table Was Added at the Source, but the Running Task Did Not Automatically Ingest It?
A: Only in Regex-Matched Tables mode are new tables that fall within the regex scope automatically ingested. In Specified Tables mode, you need to pause the task, edit and add new tables, and then republish and restart it.
Q: Automatic Table Creation Failed?
A: Automatic table creation fails in the following cases:
The current account does not have table creation permission on the target Catalog. Contact the administrator for authorization.
If a source table field type has no corresponding type on the target side, click Edit in Mapping Match Preview to manually adjust the field type and retry.
When Kafka is used as the source, automatic table creation is not supported. You need to create the table on the target side in advance and then select Use Existing Table.
Q: After System Fields Are Deselected, Do the Corresponding Columns Still Exist in the Destination Table?
A: Yes. After a system field is created in the target table, it will not be deleted if you deselect it. Instead, new data will no longer be written to it. To remove it completely, manually adjust the target table Schema outside the data ingestion task.
Q: The Task Reports an Error When a DDL Change Occurs After Field Mapping Is Edited?
A: In custom field mapping mode, DDL changes (adding columns, dropping columns, renaming columns, and modifying column types) are no longer automatically synchronized. Recommendations:
Evaluate whether it can be restored to Default Mapping.
If you must keep the custom mapping, manually adjust the target table Schema and task field mappings when changes occur on the source side.
References