Parameter Passing allows downstream tasks to reference the output of upstream tasks, enabling data flow between tasks. Parameter passing between Notebook and SQL tasks is currently supported.
Overview
There are three typical scenarios for parameter passing in DataBuddy:
|
Notebook → Notebook | Notebook | Notebook | dlcutils.jobs.taskValues.set/get
|
Notebook → SQL | Notebook | SQL | {{tasks.<name>.values.<key>}} referenced in task parameters
|
SQL → Notebook | SQL | Notebook | {{tasks.<name>.output.rows / first_row / first_row.<col>}} referenced in task parameters
|
Prerequisites
The upstream task has been created in the workflow and configured with an output.
The downstream task must have upstream dependencies configured. Otherwise, the error "Unable to resolve reference" will be reported at runtime.
Scenarios
Scenario 1: Notebook Task to Notebook Task
Setting Parameters in the Upstream Notebook
my_value = "example_value"
dlcutils.jobs.taskValues.set(key='my_value', value=my_value)
Obtaining Parameters in the Downstream Notebook
The downstream Notebook task must have the upstream task configured as a dependency in the workflow. Then:
value = dlcutils.jobs.taskValues.get(
taskKey='upstream_task_name',
key='my_value',
debugValue='debugValue'
)
print(value)
# Output debugValue when debugging in Studio
# Output example_value when the Workflow is scheduled to run
Tip
dlcutils.jobs.taskValues.get can retrieve the value set by the upstream task only during Workflow scheduling. During Studio debugging, debugValue is returned instead.
Scenario 2: Notebook to SQL Task
Setting Parameters in the Upstream Notebook
dlcutils.jobs.taskValues.set(key='favorite_food', value='apple')
Parameter Configuration for the Downstream SQL Task
In the parameter panel of the downstream SQL task, configure the following:
task_param = {{tasks.<upstream Notebook task name>.values.favorite_food}}
Using Parameters in the Downstream SQL Task
SELECT '${task_param}' AS food;
-- Result: apple
Scenario 3: SQL to Notebook Task
Upstream SQL Task
SELECT 1 AS col_a, 2 AS col_b, 3 AS col_c
UNION
SELECT 4, 5, 6
UNION
SELECT 7, 8, 9;
Parameter Configuration for the Downstream Task
The parameter panel of a downstream task can reference three levels of granularity:
|
{{tasks.<name>.output.rows}}
| All rows (JSON array) | [{"col_a":1,"col_b":2,"col_c":3}, ...]
|
{{tasks.<name>.output.first_row}}
| First row (JSON object) | {"col_a":1,"col_b":2,"col_c":3}
|
{{tasks.<name>.output.first_row.<col>}}
| Specified column in the first row | 1
|
Tip
SQL output limits: up to 1,000 rows and a maximum of 48 KB. If an SQL file contains multiple SELECT statements, only the result of the first SELECT is used as the output.
Obtaining Parameters in the Downstream Notebook
param_value = dlcutils.widgets.get('task_param')
print(param_value)
# [{"col_a":1,"col_b":2,"col_c":3}, ...]
Scenario 4: Passing Parameters in a For Each Loop
The loop parameters of a For Each node support referencing the output of upstream tasks:
{{tasks.<upstream SQL>.output.rows}}
{{tasks.<upstream Notebook>.values.<key>}}
Runtime Validation
When the Upstream Task Is Not Running
If you rerun or run only the downstream task without selecting the upstream task at the same time, the following will occur:
|
Scheduling Decision | Fails directly. |
Error Message | Execution failed. Failure cause: Unable to resolve the reference: The output of task "upstream_name" cannot be accessed because the task is not in the running scope.
|
Error Code | Failed to obtain parameters passed from upstream.
|
Rerun Scenarios
|
Downstream Only | Use the parameters from the last successful run. |
Include Upstream | After the upstream is rerun, the downstream uses the latest results. |
Use Limits
dlcutils.jobs.taskValues is used only in Notebook tasks. SQL tasks output data through output.rows.
The output.rows of an SQL task can contain up to 1,000 rows / 48 KB. Data beyond this limit is truncated.
The upstream task referenced by a parameter must be a direct or indirect upstream. You cannot reference tasks at the same level or downstream tasks across the dependency chain.
Cross-workflow parameter passing is not supported. Parameters can only be passed between tasks within the same workflow.
A single parameter value can be up to 2,048 characters long. Reference values passed from upstream can be up to 48 KB for SQL output or 10 KB for job parameters.
FAQs
Q1: Can parameters be passed across workflows?
No. Parameters can only be passed between upstream and downstream tasks within the same workflow. Cross-workflow parameter passing, including nested workflows and Run Job scenarios, is not currently supported.
Q2: The upstream Notebook sets a parameter using dlcutils.jobs.taskValues.set, but the downstream task reports an error and cannot obtain it?
Check whether the upstream task runs in a Workflow rather than in Studio. taskValues are not written during Studio debugging.
Check whether the downstream task has the upstream task configured as a dependency.
Check that the key in dlcutils.jobs.taskValues.set matches the case of the key referenced by the downstream task.
Q3: How can multiple rows of upstream SQL output be used by a downstream For Each loop?
Configure the loop parameter of the For Each node as {{tasks.<upstream SQL>.output.rows}}. Each row output by the SQL task is used as one iteration. For more information, see For Each loop task. References