Overview
A real-time sharded database and table task aggregates real-time changes from multiple data source connections, multiple databases, and multiple physical sharded tables into the same target logical table in the DataBuddy data lakehouse. A common scenario is as follows: a business adopts horizontal database and table sharding (distributed across 8 databases and 32 tables by user_id hash) and wants to merge them into a single large table in the data lakehouse for analysis. This document describes the complete configuration process of the task.
Prerequisites
Before you start this operation, make sure the following prerequisites are met:
You have activated the DataBuddy service and completed workspace initialization.
Data source connections have been created for all physical databases to be connected and have passed connectivity tests. For details, see Add a Data Source Connection. CDC has been enabled in the source database.
The schemas of all physical sharded tables involved in the merge must be consistent, with identical field names, types, and primary key constraints.
Use Limits
Limit | Description |
Source Type | MySQL, Tencent Cloud MySQL, TDSQL-C MySQL, PostgreSQL, Oracle, SQL Server |
Target Type | Iceberg native tables under the DataBuddy data lakehouse Catalog |
Computing Resource | Only real-time cluster computing resources are supported. |
Resource consumption | Single-task range: 1 to 256 CUs, with a step size of 1 CU |
Physical table Schema | Schemas of all physical shards corresponding to the same logical table must be consistent. |
Unsupported source type | |
Operation Steps
A real-time sharded database and table task uses a wizard-based configuration, which includes five steps: Basic Information → Source → Target → Data Verification → Runtime Settings. The process is consistent with the whole-database task. This document focuses on the steps that differ, while the parts that are consistent with the whole-database task are reused through links to Real-time Whole-Database Multi-Table Task Configuration. Step 1: Entering the Task Creation Page
2. Go to the workspace.
3. Go to Data Integration > Ingestion Tasks, and click Create Task.
4. After selecting the data source type, select Real-time Sharded Database and Table Ingestion as the ingestion method.
Step 2: Configuring Basic Information
After filling in the basic task information, click Next. For parameter details, see Parameter Description > Basic Information Settings. Step 3: Configuring the Source
1. Data Source: Select multiple data source connections of the same type from the drop-down list.
2. Configure the logical table mapping rule: Declare multiple physical databases and physical tables as the same logical table, which serves as the merge dimension on the target side. For specific rules, see the on-page prompts.
3. Select the read mode: Full + Incremental or Incremental Only.
4. Select the subscription type: Insert, Update, or Delete (at least one).
5. To configure advanced parameters, expand Advanced Settings.
Step 4: Configuring the Target
Basic Configuration: Select the target Catalog, Schema matching policy, table matching policy, and system fields.
Table Creation Configuration: Currently, only Iceberg native tables are supported. The platform generates table creation statements based on default field mapping rules, and users can modify the field mapping as needed.
Mapping Match Preview: The source database and source table of each record are displayed as the logical database and logical table, while the remaining columns are consistent with those in whole-database tasks.
Step 5: Data Reconciliation
Tip
In a sharded database and table scenario, the reconciliation process aggregates the row counts and primary keys of all source physical shards and compares them with the target logical table. For example, if 8 databases and 32 tables on the source are merged into 1 table on the target, the total row counts and primary keys of the 8×32 physical tables are used to compare against the 1 target table.
Step 6: Configuring Runtime Settings
The DDL processing policy, task running policy, alarm notifications, and advanced settings are consistent with those of whole-database tasks. For details, see Run Settings. Step 7: Saving or Publishing
Operation | Action | Task Status |
Create only | Save configuration without publishing. | Unpublished |
Create and publish | Save and submit for publishing. | Pending start |
Parameter Description
Basic Information Settings
Parameter | Description | Required | Default Value |
Access Type | Select Source Type. | Yes | The destination is fixed as DataBuddy. |
Access Mode | Real-time sharded database and table ingestion | Yes | Real-time whole-database multi-table ingestion |
Task Name | Uppercase and lowercase letters, Chinese characters, digits, and underscores, with a length not exceeding 256 | Yes | ${Data Source Type}_YYYYMMDD_HHmmss
|
Computing Resource | Real-time cluster computing resources used for real-time synchronization | Yes | - |
Resource consumption | Number of CUs, ranging from 1 to 256, with a step size of 1 | Yes | - |
Owner | Account used to run the task; must have access permissions on the target end. | Yes | Current Creating User |
Task Description | No more than 200 characters | No | - |
Source Settings
Parameter | Description | Required |
Data source. | Data source connection of a matching type in the current Workspace, supporting multiple selection | Yes |
Logical Tables | Merge multiple physical databases + physical tables into one logical table. | Yes |
Read Mode | Full + incremental / Incremental only | Yes |
Subscription Type | Insert, update, and delete, with at least one retained. | Yes |
Advanced Settings | Pass-through parameters specific to the data source | No |
Tip
The Run Identity of a sharded database and table task must have access permissions to all selected data source connections. Failure to access any data source will cause the task to fail.
Target Settings
Running Settings
Verification Result
After the operation is completed, you can confirm in the following ways:
1. You can see the task in the Data Integration > Access Tasks > Real-Time Synchronization list, and the Task Type column displays Database/Table Sharding.
2. After the task is started, go to the Ops details page. In the link details list, each physical database and physical table corresponds to a mapping record, and the target columns all point to the same logical table.
3. In the data reconciliation overview, the number of tables with consistent data is counted by target logical table. In the details, you can view the differences in total row counts and primary keys between the source and target.
FAQs
Q: Can Multiple Target Tables Be Merged in a Sharding Scenario?
A: The current task uses logical tables as the merge unit. A single task can define multiple logical tables, and each logical table corresponds to a group of sharded databases and tables → one target table. If you need to split different business modules into multiple target tables, you can define multiple logical tables in the same task or split them into multiple tasks.
Q: What If the Schemas of Source Physical Sharded Tables Are Inconsistent? (Some Tables Have Additional Fields)
A: The current task recommends that the schemas of all physical sharded tables involved in the merge be completely consistent. If minor differences exist in individual tables, such as a table having extra fields, the target table will be created based on the union of fields from the source tables. If the schemas of the source tables differ significantly, the following is recommended:
Fill in the differences on the source side to align all sharded tables.
Alternatively, you can split out the differing tables and configure them as a separate task.
Q: What Happens If a Primary/Secondary Switchover Is Performed for the Instance of a Source Data Source?
A: Refer to the "Primary/Secondary Switchover" description in the corresponding data source section (such as MySQL Data Source). General rule: CDC links recover by GTID first. If GTID is not supported, you may need to restart the task and specify a recovery position. Q: Can Tables Be Added to a Running Sharding Task?
A: The "logical table" configuration of a real-time database/table sharding task is fixed once saved. To change the scope, you need to pause the task, edit it, and then republish and restart it.
References