Offline data ingestion tasks are used to schedule and run offline sync tasks of the data ingestion module in Workflow. Tasks establish linkage by referencing ingestion task IDs, and all ingestion configurations are still maintained in the Data Ingestion module. Overview
Characteristics of offline data ingestion tasks:
No longer embeds ingestion configurations: it serves only as a reference execution entry, while detailed sync configurations (source, destination, field mapping, and sync mode) are completed in the Data Ingestion module.
Use data ingestion compute resources: different from the resource types of Notebook and SQL tasks.
Run Details Show Ingestion Results: The task run details page directly displays the progress, the number of rows, and failure reason of the data ingestion run.
For general configurations (task name, upstream dependencies, run conditions, parameters, monitoring metrics, alarms, and failure and timeout policies), see Task General Configuration. Prerequisites
The Data Ingestion module has created and saved the target offline sync task. For details, see the Offline Data Sync documentation series. Operation Steps
Step 1: Creating a Job
1. On the workflow canvas, click + Add Task.
2. Select Offline Data Ingestion as the task type.
Step 2: Configuring Basic Information
Parameter | Description | Required |
Task Name | Unique within the same workflow. | Yes |
Task type | Offline data ingestion (non-editable). | Yes |
Task Sources | Click the drop-down list to select an offline synchronization task in the data ingestion module. | Yes |
Step 3: Selecting an Ingestion Task
1. Click the Task Source dropdown list to display the offline ingestion tasks in the current workspace.
2. Supports both fuzzy search and exact search.
3. After selection, the task name is displayed in the configuration panel. Click it to go to the execution page of the ingestion task for details.
Step 4: Configuring Compute Resources
Parameter | Description |
Computing Resource | Only data ingestion compute resources are displayed. Select from the drop-down list. |
Step 5: Upstream Dependencies, Parameters, Monitoring Metrics, and Alarms
Special Notes on Parameter Handling
Parameter handling is the same as for SQL tasks: configure Key-Value parameters on the task, and they are replaced into the ingestion task execution command at runtime. For details, see Task Parameter Configuration. Monitoring Metrics
Three types of metrics are supported: runtime, wait time, and completion time. For details, see Workflow and Task Alarms. Step 6: Failure and Timeout Policies
Step 7: Saving
Click Create Task or Save Task.
Job Run Details
Run details page of an offline data ingestion task:
Tab | Description |
Execution result | The running progress and execution status of data ingestion tasks are consistent with those viewed in the data ingestion module. |
Running logs | Same as above. |
Use Limits
Only offline data ingestion tasks can be referenced. Real-time data ingestion tasks are not supported.
The compute resource type must be data ingestion.
After an ingestion task is deleted, workflow task runs fail directly.
After the task source is selected, detailed sync configurations (source, destination, mapping, and so on) can only be edited in the Data Ingestion module.
FAQs
Q1: Can real-time data ingestion also be orchestrated in Workflow?
No. Real-time data ingestion tasks have built-in resident scheduling and are not triggered through Workflow. We recommend that you maintain real-time tasks in the Data Ingestion module. For more information, see the Real-Time Data Sync documentation series. Q2: Can an upstream Notebook task pass parameters to an offline data ingestion task?
Yes. The parameter passing rules are the same as those for other tasks:
target_dt = {{tasks.<upstream Notebook task name>.values.target_dt}}
Q3: Can an ingestion task be rerun in Workflow after it fails?
Yes. On the workflow run details page, select the task node and click Rerun. Rerunning triggers a new ingestion task execution. For details, see Ops Operations for Workflows and Tasks. References