Access Method | Typical Scenario | Entries | Documentation |
Offline Data Synchronization | Periodically synchronize data from relational databases, big data storage, and message queues to the data lakehouse in batches | Data ingestion → Task management → Create synchronization task → Offline single-table synchronization / Offline batch synchronization | |
Real-time Data Synchronization | Synchronize relational database CDC and message queue data in real time, with support for DDL change synchronization. | Data ingestion → Task management → Create synchronization task → Real-time whole-database multi-table synchronization / Real-time sharded database and table synchronization | |
File Upload to Table | Parse local or COS files and load them into structured tables. | Data ingestion → File upload → File to table / COS file to table | |
File Upload to Volume | Upload files to a Volume in their original format while preserving the original file structure. | Data ingestion → File upload → File to volume | |
Batch File Sync to Volume | In an offline synchronization task, write structured data to a Volume in the specified format, or synchronize unstructured files to a Volume in batches. | Data ingestion → Task management → Create synchronization task (select a Volume type Catalog as the destination) |
1. Prepare a data source connection (data source configuration).↓2. Prepare compute resources (use a real-time cluster for real-time ingestion and a platform cluster for offline ingestion).↓3. Create a sync task and select a sync method.↓4. Configure the source (select a database / table / file path, and configure filter conditions and filtering rules if necessary).↓5. Configure the target (Catalog / Schema / Table / Volume, and configure automatic or manual table creation if necessary).↓6. Configure field mappings and runtime parameters (such as dirty data threshold, concurrency, rate limit, and checkpoint interval).↓7. Configure alarms and run policies.↓8. Save / Publish / Start the task.
Dimension | Offline Synchronization | Real-time Synchronization |
Synchronization Granularity | Single table, batch single tables | Whole-database multi-table, sharded database and table |
Triggering Method | Triggered by workflow scheduling or manual debugging | Runs continuously after startup and captures CDC changes from the source |
Source Read Mode | Full read, incremental read (based on filter conditions) | Full + Incremental, Incremental Only |
Write Mode | overwrite / append / upsert | append/upsert |
Automatic table creation | Supports manual table creation | Supports automatic table creation (except for some sources such as Kafka) |
Schema change synchronization | Not involved (based on the Schema at task execution time) | Supports DDL change synchronization (fully supported for MySQL / Tencent Cloud MySQL / TDSQL-C MySQL). |
Data reconciliation | Supports full and incremental data reconciliation by the number of records | Supports full and incremental data reconciliation by the number of records / content. |
Scenario | Scheduled data warehouse refresh, ODS layer loading, report data preparation | Real-time database ingestion into data lake, CDC streaming analysis |
Entries | Description | Supported File Types |
File to Table | After a file is uploaded, it is automatically parsed into a structured table and stored in the Schema under the specified Catalog. | .csv / .tsv / .json / .jsonl / .avro / .parquet / .txt / .xml |
File to Volume | Files are stored in a Volume in their original format, preserving the binary / original structure. | Any type |
COS File to Table | Select files from Tencent Cloud COS and parse them into structured tables. | .csv / .json / .xml / .avro / .parquet / .txt |
Your Requirement | Recommended Solution |
Synchronize data from business databases to the data lakehouse on a daily schedule. | Offline single-table synchronization + Workflow scheduling |
Batch synchronization of multiple tables from the same source to the lakehouse | Offline Batch Synchronization Task |
Real-time CDC ingestion from business databases into the data lake | Real-time whole-database multi-table synchronization task |
Merge multiple sharded databases and tables into one target table. | Real-time sharded database and table synchronization task |
One-time import of historical CSV/Parquet files for analysis | File to Table |
Archive files to the lakehouse in their original format. | File to Volume |
Limit | Description |
Computing Resource | Real-time ingestion tasks support selecting only real-time cluster compute resources; offline ingestion tasks support selecting only platform cluster compute resources. |
Source Type | For the full range of supported data sources, see Supported Data Sources and Read/Write Capabilities |
Target Type | The target of structured tasks is fixed as catalog tables (based on Iceberg native tables); unstructured data writes support catalog Volume volumes. |
File Upload Size | Files to table: up to 10 files per upload, with a total size no greater than 2 GB; files to volume: up to 100 files per upload, with each file no greater than 5 GB. |
Was this page helpful?
You can also Contact sales or Submit a Ticket for help.
Help us improve! Rate your documentation experience in 5 mins.
Feedback