The For Each loop task is used to execute internal tasks iteratively over a dataset (an array or an array of objects), and is commonly used in scenarios such as parallel processing by partition, multi-database and multi-table synchronization, and large-scale batch processing.
Overview
A For Each node consists of an outer loop node and an inner loop node:
Outer node: defines loop parameters and concurrency, determining the number of iterations and the degree of parallelism.
Inner node: the task that is actually executed, running once per loop with different input parameters.
Typical use cases:
Scenario | Description |
Parallel processing by partition | The upstream queries multiple date partitions and starts multiple data backfill tasks in parallel. |
Multi-database multi-table synchronization | Read the table list from the metadata database and dynamically generate multiple synchronization tasks. |
Large-scale model inference | Split massive data into multiple batches and start multiple inference nodes in parallel. |
For general configurations (task name, upstream dependencies, run conditions, parameters, monitoring metrics, alarms, and failure and timeout policies), see Task General Configuration. Prerequisites
You are already the workflow owner or have Manage permission.
Loop parameters (an array or an array of objects) are ready, or they can be referenced from the output of an upstream task.
Operation Steps
Step 1: Creating a For Each Node
1. On the workflow canvas, click + Add Task.
2. Select For Each as the task type.
Step 2: Configuring the External For Each Node
Parameter | Description | Required |
Task Name | Unique within the same workflow. | Yes |
Task type | For Each (not modifiable). | Yes |
Loop Parameter | Array or object array that the loop depends on. Up to 5,000 characters. | Yes |
Maximum concurrency | Number of loops running concurrently. A positive integer from 1 to 100. | No (default: 1) |
Supported Sources for Loop Parameters
Source | Syntax |
Static array | ["Beijing", "Shanghai", "Guangzhou"]
|
Static object array | [{"db": "inventory", "region": "shanghai"}, {"db": "orders", "region": "beijing"}]
|
Upstream Notebook | {{tasks.<name>.values.<key>}} (set by the upstream task through dlcutils.jobs.taskValues.set)
|
Upstream SQL | {{tasks.<name>.output.rows}}
|
Upstream Dependencies and Run Conditions
Monitoring Metrics and Alarms
Step 3: Adding an Inner Loop Node
1. After configuring the outer node, click Add Loop Node.
2. Go to the configuration panel of the inner loop node.
3. Select the inner task type: Notebook / Python File / SQL / Offline Data Ingestion, Nested Workflow, Data Quality Task.
4. The configuration items of an inner node are the same as those of the corresponding task type, and loop variables can be used in parameters.
Loop Variables (Used Only in Inner Nodes)
Variables | Description | Example |
{{loopDataArray}}
| The entire loop parameter array. Complete data is returned in each loop. | [{...}, {...}]
|
{{loopDataCurrent}}
| Element corresponding to the current loop. | {"db": "inventory", "region": "shanghai"}
|
{{loopDataCurrent.<key>}}
| Value of the specified key in the current loop element. | shanghai
|
{{loopTimes}}
| Current loop number, starting from 1. | 1
|
If no loop variable is referenced when parameters for an inner node are configured, a message appears below the parameter:
You can obtain the input data of each iteration by setting the parameter value to {{loopDataCurrent}}.
Step 4: Saving
Click Create Task or Save Task. After saving, the For Each node on the canvas appears as a double-layer frame:
Outer frame: the For Each node itself.
Inner frame: the inner loop node.
Canvas Interactions
Operation | Description |
Click the outer frame. | Select the For Each node and open the external configuration panel. |
Click the inner frame. | Select the inner loop node and open its configuration panel. |
Drag the outer frame. | Move the entire For Each node, including its internal nodes. |
Connect to other nodes. | Only the outer frame can be connected. Internal loop nodes cannot be directly connected to external nodes. |
Runtime Behavior
Overall Status
Scenario | Status |
All loops succeed | For Each node status: Succeeded |
Partial loop failure | For Each node status: Failed (N loops in total, X failures) |
At least one loop is running. | Status: Running (N loops in total, X failures) |
Loop parameter is empty. | Status: Succeeded; run details indicate "No loop execution". |
Incorrect loop parameter format. | Status: Failed; run details indicate "Incorrect input field format. The loop parameter must be a JSON array of basic types or an array of objects." |
Viewing Loop Execution Details
After the run ends, click the For Each node to go to the run details page. You can:
1. View Loop Execution List: Each loop occupies a separate row, displaying information such as the number of loops, running status, running duration, and parameter values.
2. Filter Failed Loops: Select "Show Failed Loops Only" to quickly locate failed items.
3. Go to Single Loop Details: Click the run ID of a loop to go to the inner node run details page and view specific execution logs and parameters.
4. View Loop Configuration: The right information panel displays the original values of loop parameters and the maximum concurrency.
Note:
The rerun policy is configured on the inner loop node. When a rerun is triggered, only the inner nodes that failed in that loop are rerun.
Use Limits
For Each supports configuring only one inner loop node. If multiple steps are required, encapsulate them into a nested workflow and reference it.
The inner loop node cannot be connected to other external tasks.
Loop parameters (constants) can be up to 5,000 characters long. Reference values (variables) passed from upstream can be up to 48 KB. If the limit is exceeded, the task run fails.
The maximum number of loops is determined by the array length of the loop parameter, and no independent upper limit can be configured.
The maximum concurrency for a single For Each is 100.
If any loop iteration fails, the overall status of the For Each node is Failed.
FAQs
Q1: If loop parameters are referenced from the output.rows of an SQL task, what is the loop behavior?
Each row in the SQL output is used as one loop. For example:
{{tasks.test_sql_up.output.rows}}
If [{"id": 1, "name": "alice"}, {"id": 2, "name": "bob"}] is returned, the loop runs twice. {{loopDataCurrent.name}} is alice and bob respectively.
Q2: How do I rerun a failed loop iteration?
The rerun policy is configured on the inner loop node. When a rerun is triggered, only the inner nodes that failed in that loop are rerun, and successful loops are not rerun.
Q3: What happens when the loop parameter is an empty array []?
The For Each node is treated as having no loops to run, and its status is set to Successful. Downstream tasks continue to run based on their run conditions.
Q4: When is the loop parameter format validated?
Validation is performed at runtime. Validation is not mandatory during configuration. It is recommended to verify the format validity by running a debug after saving.
References