Release Notes
Product Introduction
Purchase Guide
Getting Started
Buddy
- Buddy - Data Analytics Scenarios
- Data Engineering Agent
- Buddy Overview
- Data Governance Agent
- Buddy Data Security
- Overview of Buddy Agent Basic Capabilities
Platform Management
- Workspace Management
- Compute Resource Management
- Data Source Management Overview
Data Integration
- Supported Data Ingestion Solutions
- Supported Data Sources and Read Capabilities
- Data Source Management
- List of Data Sources
- Offline Data Synchronization
- Real-Time Data Synchronization
- File Upload
Data Engineering
- Studio
- Workflow
- CI/CD
Data Science
- Data Science Overview
- Model Experiments
- Feature Management
- Model Management
- Model Service
Data Analysis
- Data Exploration Overview
Data Governance
- Data Catalog Overview
- Data Discovery Overview
- Data Security Overview
- Data Quality Overview
- Semantic Model Overview
- Governance Tag Overview
Data Applications
- Agent Development
- MCP
- Model List
API Documentation
- History
- Introduction
- API Category
- Making API Requests
- Console APIs
- Workspace APIs
- Computational Resource APIs
- Development Studio APIs
- Workflow APIs
- Catalog APIs
- Data Types
- Error Codes
FAQs
Agreements
Contact Us
Glossary

Table-to-Volume Synchronization

Download

Focus Mode

Font Size

Last updated: 2026-10-09 16:10:05

AI-Translated

Overview
Table-to-Volume sync writes table data from structured data sources (such as MySQL, PostgreSQL, Oracle, and SQL Server) to DataBuddy Volume in a specified file format. It is commonly used to export processed data as .ok / .ctl files for downstream systems to consume. This document describes the specific configuration for this pipeline. For general steps, see Create a Single-Table Sync Task.
Applicable Scenarios
Periodically export processed result files to downstream businesses such as SMS, marketing, and external systems (for example, .txt / .csv / .json).
Data delivery: Export lakehouse data to a specified Volume path in the agreed format.
File archiving: writes business database data to a Volume partitioned by time for long-term storage.
Prerequisites
Before you start this operation, make sure the following prerequisites are met:
You have activated the DataBuddy service and completed workspace initialization.
The current account has a role related to data access in the target Workspace. For details, see Members and Permission Management.
Compute resources for data ingestion (platform cluster type) have been created.
A source data source connection has been created and tested. For details, see Add a Data Source Connection.
A Schema of the Volume type already exists in the target Catalog, or the current account has the permission to create one.
Use Limits
Limit
Description
Source Type
Only structured data sources are supported (see Supported Data Sources and Read/Write Capabilities).
Target Type
Volume type Catalog under DataBuddy Catalog
Field mapping
Source can be configured; target cannot be configured (Volume does not perceive field structure).
File Type
Only txt / csv / json are supported.
.ok and .ctl
Marker files supported vary by file type.
Operation Steps
The general process is the same as Table-to-Table Sync. This section only describes the differences in Target and Field Mapping.
Step 1: Entering the Task Creation Page
See the entry for Table-to-Table Sync.
Step 2: Filling in Basic Information and Source
The source fields (data source type, connection, source database, data table, split key, filter conditions, sharding, and advanced settings) are the same as those in Table-to-Table Sync. See Table-to-Table Sync > Step 3.
Step 3: Configuring the Target
When the target is selected as a Volume-type Catalog, the configuration items are as follows:
Parameter
Description
Required
Default Value
Catalog
Select a Volume type Catalog for which you have permission.
Yes
-
Schema
Schema under Catalog
Yes
-
Volume
Select an existing Volume; if it does not exist, create it in the Catalog.
Yes
-
Volume Directory
Subpath within the Volume for file writing; selectable, enterable, supports inserting regular expressions and parameters; using the root path "/" as the subpath is not recommended.
Yes
-
File name.
Name of the file to be written; enterable, supports inserting regular expressions and parameters.
Yes
-
File format.
txt / csv / json; different types have different parsing configurations.
Yes
-
Format Settings
See File Parsing and Inference
Yes
-
Write Mode
Append / Overwrite / Truncate / NonConflict
Yes
Append
Mark Completion File
Generate the .ok file after the file is written.
Yes
Not generated
ctl File
Generate the .ctl file after the file is written.
Depends on file type.
Not generated
Step 4: Format Settings
Click Format Settings to open a dialog box, and configure the write rules based on the selected file type:
TXT / CSV
Configuration Item
Description
Compression Format
gzip / No compression, and so on.
Column delimiter
Field delimiter (such as , ; `)
Row delimiter
Row delimiter (such as \\n \\r\\n custom)
Encoding
UTF-8 / GBK, and so on
Control escaping
Escape special characters or not
Quote Character
String delimiter (such as " ')
Include Header
Write field names in the first row or not
JSON
Configuration Item
Description
Compression Format
gzip / No compression, and so on
Encoding
UTF-8 / GBK, and so on
Null Conversion
Whether to convert null to a specified string
Include Header
Write field names in the first row or not
Step 5: Field Mapping
Link
Source field mapping
Target field mapping
Structured data source → Volume
Configurable
Not configurable
If you do not want a field to be written to a Volume, you can delete the corresponding field mapping row.
When a field mapping row is added, the field type must be Field. Otherwise, an error will be reported.
Step 6: Running Settings, Debugging, Saving, and Publishing
See Table-to-Table Sync > Step 4 / 5.
Supported File Types and Marker Files
File Type
.ok file
ctl File
Text
Yes
Yes
CSV
Yes
Yes
JSON
Yes
None
Tip
The .ok  file is used to notify downstream systems that data has been written. The .ctl  file is used to record data descriptions, such as the number of rows and file list.
Verification Result
After the task is saved and debugged, you can confirm it in the following ways:
1. In Data Catalog, go to the target Volume and view the written files, such as output_20260101.txt and output_20260101.ok.
2. Download or preview the file content to confirm that the field order, delimiter, and encoding are consistent with the configuration.
3. In Offline Sync Task Ops Monitoring, view the task logs to confirm that no errors exist.
FAQs
Q: What Parameters Are Supported for the Target Volume Path or Filename?
A: System time parameters (such as ${yyyyMMdd}) and workflow scheduling parameters are supported. During manual debugging runs, if no parameter values are set, the system uses default values. Define them in Run Parameters in advance to ensure that debugging is repeatable.
Q: Will Existing Files Be Deleted from the Target Volume Path When the Write Mode Is Set to Overwrite?
A: Overwrite only overwrites files with the same name and does not delete other files in the path. Append appends content to the end of a file with the same name, depending on the file type.
Q: How Do I Configure Null Value Conversion for JSON Files?
A: When this feature is enabled, all NULL fields are converted to a specified string, such as an empty string "" or the literal NULL. When this feature is disabled, the literal null is output according to the JSON standard.
Q: Are Columnar Formats Such as .parquet and .avro Supported?
A: Currently, only TEXT / CSV / JSON formats are supported.
References
Table-to-table synchronization
File-to-Volume synchronization
Create Single-Table Sync Tasks in Batches
Offline Synchronization Task Ops Monitoring
Unstructured Data Catalog (File and Volume Management)

Help and Support

Was this page helpful?

You can also Contact sales or Submit a Ticket for help.

Help us improve! Rate your documentation experience in 5 mins.

Feedback

tencent cloud

Tencent Cloud DataBuddy

Table-to-Volume Synchronization

Overview

Applicable Scenarios

Prerequisites

Use Limits

Operation Steps

Step 1: Entering the Task Creation Page

Step 2: Filling in Basic Information and Source

Step 3: Configuring the Target

Step 4: Format Settings

TXT / CSV

JSON

Step 5: Field Mapping

Step 6: Running Settings, Debugging, Saving, and Publishing

Supported File Types and Marker Files

Verification Result

FAQs

Q: What Parameters Are Supported for the Target Volume Path or Filename?

Q: Will Existing Files Be Deleted from the Target Volume Path When the Write Mode Is Set to Overwrite?

Q: How Do I Configure Null Value Conversion for JSON Files?

Q: Are Columnar Formats Such as .parquet and .avro Supported?

References

Help and Support

Limit	Description
Source Type	Only structured data sources are supported (see Supported Data Sources and Read/Write Capabilities).
Target Type	Volume type Catalog under DataBuddy Catalog
Field mapping	Source can be configured; target cannot be configured (Volume does not perceive field structure).
File Type	Only txt / csv / json are supported.
.ok and .ctl	Marker files supported vary by file type.

Parameter	Description	Required	Default Value
Catalog	Select a Volume type Catalog for which you have permission.	Yes	-
Schema	Schema under Catalog	Yes	-
Volume	Select an existing Volume; if it does not exist, create it in the Catalog.	Yes	-
Volume Directory	Subpath within the Volume for file writing; selectable, enterable, supports inserting regular expressions and parameters; using the root path "/" as the subpath is not recommended.	Yes	-
File name.	Name of the file to be written; enterable, supports inserting regular expressions and parameters.	Yes	-
File format.	txt / csv / json; different types have different parsing configurations.	Yes	-
Format Settings	See File Parsing and Inference	Yes	-
Write Mode	Append / Overwrite / Truncate / NonConflict	Yes	Append
Mark Completion File	Generate the .ok file after the file is written.	Yes	Not generated
ctl File	Generate the .ctl file after the file is written.	Depends on file type.	Not generated

Configuration Item	Description
Compression Format	gzip / No compression, and so on.
Column delimiter	Field delimiter (such as `,` `;` `)
Row delimiter	Row delimiter (such as `\\n` `\\r\\n` custom)
Encoding	UTF-8 / GBK, and so on
Control escaping	Escape special characters or not
Quote Character	String delimiter (such as `"` `'`)
Include Header	Write field names in the first row or not

Link	Source field mapping	Target field mapping
Structured data source → Volume	Configurable	Not configurable