Overview
Table-to-Volume sync writes table data from structured data sources (such as MySQL, PostgreSQL, Oracle, and SQL Server) to DataBuddy Volume in a specified file format. It is commonly used to export processed data as .ok / .ctl files for downstream systems to consume. This document describes the specific configuration for this pipeline. For general steps, see Create a Single-Table Sync Task. Applicable Scenarios
Periodically export processed result files to downstream businesses such as SMS, marketing, and external systems (for example, .txt / .csv / .json).
Data delivery: Export lakehouse data to a specified Volume path in the agreed format.
File archiving: writes business database data to a Volume partitioned by time for long-term storage.
Prerequisites
Before you start this operation, make sure the following prerequisites are met:
You have activated the DataBuddy service and completed workspace initialization.
Compute resources for data ingestion (platform cluster type) have been created.
A Schema of the Volume type already exists in the target Catalog, or the current account has the permission to create one.
Use Limits
Limit | Description |
Source Type | |
Target Type | Volume type Catalog under DataBuddy Catalog |
Field mapping | Source can be configured; target cannot be configured (Volume does not perceive field structure). |
File Type | Only txt / csv / json are supported. |
.ok and .ctl | Marker files supported vary by file type. |
Operation Steps
The general process is the same as Table-to-Table Sync. This section only describes the differences in Target and Field Mapping. Step 1: Entering the Task Creation Page
Step 2: Filling in Basic Information and Source
The source fields (data source type, connection, source database, data table, split key, filter conditions, sharding, and advanced settings) are the same as those in Table-to-Table Sync. See Table-to-Table Sync > Step 3. Step 3: Configuring the Target
When the target is selected as a Volume-type Catalog, the configuration items are as follows:
Parameter | Description | Required | Default Value |
Catalog | Select a Volume type Catalog for which you have permission. | Yes | - |
Schema | Schema under Catalog | Yes | - |
Volume | Select an existing Volume; if it does not exist, create it in the Catalog. | Yes | - |
Volume Directory | Subpath within the Volume for file writing; selectable, enterable, supports inserting regular expressions and parameters; using the root path "/" as the subpath is not recommended. | Yes | - |
File name. | Name of the file to be written; enterable, supports inserting regular expressions and parameters. | Yes | - |
File format. | txt / csv / json; different types have different parsing configurations. | Yes | - |
Format Settings | | Yes | - |
Write Mode | Append / Overwrite / Truncate / NonConflict | Yes | Append |
Mark Completion File | Generate the .ok file after the file is written. | Yes | Not generated |
ctl File | Generate the .ctl file after the file is written. | Depends on file type. | Not generated |
Step 4: Format Settings
Click Format Settings to open a dialog box, and configure the write rules based on the selected file type:
TXT / CSV
Configuration Item | Description |
Compression Format | gzip / No compression, and so on. |
Column delimiter | Field delimiter (such as , ; `) |
Row delimiter | Row delimiter (such as \\n \\r\\n custom) |
Encoding | UTF-8 / GBK, and so on |
Control escaping | Escape special characters or not |
Quote Character | String delimiter (such as " ') |
Include Header | Write field names in the first row or not |
JSON
Configuration Item | Description |
Compression Format | gzip / No compression, and so on |
Encoding | UTF-8 / GBK, and so on |
Null Conversion | Whether to convert null to a specified string |
Include Header | Write field names in the first row or not |
Step 5: Field Mapping
Link | Source field mapping | Target field mapping |
Structured data source → Volume | Configurable | Not configurable |
If you do not want a field to be written to a Volume, you can delete the corresponding field mapping row.
When a field mapping row is added, the field type must be Field. Otherwise, an error will be reported.
Step 6: Running Settings, Debugging, Saving, and Publishing
Supported File Types and Marker Files
File Type | .ok file | ctl File |
Text | Yes | Yes |
CSV | Yes | Yes |
JSON | Yes | None |
Tip
The .ok file is used to notify downstream systems that data has been written. The .ctl file is used to record data descriptions, such as the number of rows and file list.
Verification Result
After the task is saved and debugged, you can confirm it in the following ways:
1. In Data Catalog, go to the target Volume and view the written files, such as output_20260101.txt and output_20260101.ok. 2. Download or preview the file content to confirm that the field order, delimiter, and encoding are consistent with the configuration.
FAQs
Q: What Parameters Are Supported for the Target Volume Path or Filename?
A: System time parameters (such as ${yyyyMMdd}) and workflow scheduling parameters are supported. During manual debugging runs, if no parameter values are set, the system uses default values. Define them in Run Parameters in advance to ensure that debugging is repeatable.
Q: Will Existing Files Be Deleted from the Target Volume Path When the Write Mode Is Set to Overwrite?
A: Overwrite only overwrites files with the same name and does not delete other files in the path. Append appends content to the end of a file with the same name, depending on the file type.
Q: How Do I Configure Null Value Conversion for JSON Files?
A: When this feature is enabled, all NULL fields are converted to a specified string, such as an empty string "" or the literal NULL. When this feature is disabled, the literal null is output according to the JSON standard.
Q: Are Columnar Formats Such as .parquet and .avro Supported?
A: Currently, only TEXT / CSV / JSON formats are supported.
References