Overview
File-to-Volume sync writes semi-structured files (FTP / SFTP / COS / S3 / Azure Blob / HDFS, and so on) to a DataBuddy Volume, supporting two usage modes:
Original file transfer: preserves the original file format and writes files directly to a Volume, commonly used for archiving and backup.
Parse and convert format: reads the source file and re-outputs it according to the target file type, commonly used for format conversion (such as TXT to JSON).
Applicable Scenarios
File archiving: writes raw files uploaded by business partners to SFTP into a Volume by date.
Format conversion: reads the upstream TXT file and writes it back to a Volume in JSON format for downstream consumption.
Multi-file merge: reads files from multiple source paths and outputs them centrally in a unified format.
Prerequisites
Before you start this operation, make sure the following prerequisites are met:
You have activated the DataBuddy service and completed workspace initialization.
Compute resources for data ingestion (data ingestion type) have been created.
A Schema of the Volume type already exists in the target Catalog, or the current account has the permission to create one.
Use Limits
Limit | Description |
Source Type | Semi-structured data sources (FTP / SFTP / COS / S3 / Azure Blob / HDFS, and so on) |
Target Type | Volume type Catalog under DataBuddy Catalog |
Read Method | Original file (Write to Volume as is) / Parse data (Parse and output again based on the target file type) |
Field mapping | Original file mode: Neither source / target can be configured; parse mode: Source can be configured (manually add fields), target cannot be configured. |
Output File Type | txt / csv / dat / json only |
Operation Steps
Step 1: Entering the Task Creation Page and Filling in Basic Information
Step 2: Configuring the Source
Parameter | Description | Required |
Data Source Type | Select the source data source type, such as SFTP. | Yes |
Data Source Connection | Select a registered data source connection. | Yes |
File path | File or folder path. Click the folder icon to open the path selector. Parameters and regular expressions are supported. For S3, include the bucket name. For Azure Blob, include the container name. | Yes |
Read Method | Original file or Parsed data (Determines whether to parse) | Yes |
File format. | Required for data synchronization mode: .csv / .avro / .json / .parquet / .txt. | Depends on the mode |
Parsing Configuration | Required for data synchronization mode. Configure by file type. | Depends on the mode |
Synchronization Scope (S3 Only) | Full / Incremental | Depends on the scenario |
Advanced Settings | Extended parameters specific to the data source | No |
For specific source field differences, see File-to-Table Sync. The source configuration of this pipeline is exactly the same as that of File-to-Table. Step 3: Configuring the Target
When the target is selected as a Volume-type Catalog:
Parameter | Description | Required |
Target Catalog | Select a Volume type Catalog for which you have permission. If none exists, click Create New to create one. | Yes |
Target Schema | Schema under Catalog | Yes |
Volume | Select an existing Volume. | Yes |
Volume File Directory | Subpath within the Volume; if left empty, files are written to the Volume root directory. | No (root directory by default) |
File name. | Output file name (required only in data synchronization mode) | Depends on the mode |
File format. | Required for data synchronization mode: txt / csv / dat / json. | Depends on the mode |
Format Settings | Required for data synchronization mode. Configure by target type. | Depends on the mode |
Write Mode | overwrite / append / upsert | Yes |
Output Format Configuration in Data Sync Mode
File Type | Configuration Item |
TXT / CSV / DAT | Compression format, column delimiter, row delimiter, encoding, control conversion, Quote Character, and header included |
JSON | Compression format, encoding, null value conversion, and header included |
Marker File Support
File Type | .ok file | ctl File |
txt | Yes | Yes |
csv | Yes | Yes |
dat | Yes | None |
json | Yes | None |
Step 4: Field Mapping
Mode | Source field mapping | Target field mapping |
File sync (raw) | ❌ Not configurable | ❌ Not configurable |
Data synchronization (parsed) | ✅ Configurable (Add Fields Manually) | ❌ Not configurable |
Step 5: Running Settings, Debugging, Saving, and Publishing
Verification Result
After the task is saved and debugged, you can confirm it in the following ways:
1. Go to the target Volume in Data Catalog and view the written files. 2. In file sync mode, the file name is the same as the original file name. In data sync mode, the file name is determined by the target configuration.
3. Download or preview the file to confirm its content. If there is an .ok file, confirm that the content marker complies with downstream conventions.
FAQs
Q: Can I Change the Read Mode If I Selected the Wrong One?
A: Yes. On the task editing page, switch the Read Mode from Raw File to Parsed Data, fill in the required configuration items, and save. Note: Switching resets the existing parsing configuration and field mappings.
Q: Will Multiple Source Files Be Merged into One When the Read Mode Is Set to "Original File"?
A: No. In file sync mode, source files are written to the Volume in a 1:1 mapping with the original structure, retaining the original file names without merging.
Q: Will Files with the Same Name Be Overwritten in Data Sync Mode?
A: It depends on the write mode:
overwrite: Overwrites files with the same name.
append: Appends content to the end of a file with the same name.
upsert: The behavior is similar to overwrite, as the Volume does not perceive primary keys.
Q: What Happens If Files with the Same Name Are Selected from Different Paths on the Source?
A: An error will be reported. Logically, during reading, all files are collected into a list, and an error is reported if duplicate file names exist.
Q: Can a Task Retain Original Files and Generate Parsed Files at the Same Time?
A: No. A task can only use one sync mode. To archive and transform data at the same time, create two tasks and orchestrate them through Workflow. References