A Notebook task is used to schedule and run .ipynb files developed in Studio within a Workflow. A Notebook task can reference Notebooks from a Studio workspace or Git folder and run them on user-selected compute resources.
Overview
Notebook tasks are the most commonly used task type in DataBuddy, covering scenarios such as ETL, machine learning training, and feature engineering. This document focuses on the differentiated configuration of Notebook tasks. For general configuration (task name, upstream dependencies, run conditions, parameters, monitoring metrics, alarms, failure and timeout policies), see General Task Configuration. Prerequisites
The workspace is bound to available compute resources that support CPU / GPU compute types, as well as distributed and single-node resource modes. For more information, see Compute Resource Overview. The target Notebook file already exists in Studio, and the current user (or Run As account) has Run permission or higher on the file. For more information, see File Permissions. If you use a Git reference, the workspace must have a Git connection configured. For details, see Set up a Git connection. Operation Steps
Step 1: Creating a Job
1. On the workflow editor canvas, click + Add Task.
2. Select Notebook as the task type.
3. Go to the task configuration panel.
Step 2: Configuring Basic Information
Parameter | Description | Required |
Task Name | Unique within the same workflow. | Yes |
Task type | Notebook. You can switch to other task types from the dropdown list (Python File / SQL / Ray / offline data ingestion, and more). After switching, the parameter area will be cleared. | Yes |
Task Sources | Workspace / Git. | Yes |
Path | Path of the referenced Notebook file. | Yes |
Step 3: Selecting a Notebook File
Job Source: Workspace
1. Click the Path input box to open the file selector:
Workspace: Browse by directory. Only folders and .ipynb files are displayed.
Recently Opened: Lists the Notebooks that the current user has browsed in the last 90 days, in reverse chronological order.
1. After selection, the path is filled into the input box, for example, /Workspace/your_username/data_etl.ipynb.
2. Operations on the right side of the path input box:
Jump icon: opens a new page to open the file in Studio.
Copy icon: copies the full path.
Job Source: Git
1. After Git is selected, a dropdown list displays the Git branches configured for the current workspace.
2. In the Path input box, enter a relative path based on the root directory of the remote repository, for example, notebooks/etl/daily.ipynb.
3. The system does not validate whether the path exists at configuration time. At runtime, if the file does not exist, the task fails with error code Invalid Git path.
Note:
When Git is not configured for the workspace, the Git option is grayed out. Click Configure to go to Set Git Connection and complete the configuration. Step 4: Configuring Compute Resources
Parameter | Description | Required | Default Value |
Computing Resource | Displays the compute resources bound to and available in the current workspace in a drop-down list. | Yes | None (User-Selected) |
Resource mode | Distributed / Single-node. | Yes | Resource mode of the file in Studio when the file is first configured; defaults to "Distributed" when the file is referenced from Git. |
Resource Configuration | Default configuration / Custom configuration. | Yes | Default Configuration |
Resource Mode: Distributed (CPU/GPU)
Configuration Item | Description | Default Value |
Executor Resources | small (1CU) / medium (2CU) / large (4CU) / xlarge (8CU) | Follows the compute resource default value |
Number of Executors | Dynamic allocation (min–max) / fixed allocation (specific quantity) | Compute resource default value |
Driver Resources | Same as Executor specification options | Compute resource default value |
Number of Executor GPUs | Displayed only for GPU computing resources. Decimal values supported. | Compute resource default value |
Number of Driver GPUs | Displayed only for GPU computing resources. Decimal values supported. | Compute resource default value |
Note:
The total amount of custom-configured resources (Executor resources × number of executors + Driver resources) cannot exceed the maximum available compute resource quota. If the quota is exceeded, the Confirm button is grayed out, and a message is displayed indicating that the current maximum available compute resources have been exceeded ({maximum compute resource CU/GPU}).
Resource Mode: Single Node
Configuration Item | Description |
Resource Specification | Select from the drop-down list. The default value is large(4CU). Only options that do not exceed the maximum compute resource specification are displayed. |
GPU configuration | Displayed only for GPU computing resources. Decimal values supported. |
Custom Configuration - JSON Mode
You can switch between UI mode and JSON mode in both directions. The JSON content follows the TencentCloud API specification: you can only modify the Value field, and adding or modifying a Key will fail validation.
Step 5: Upstream Dependencies and Runtime Conditions
Refer to the Upstream Dependencies and Run Conditions sections in General Task Configuration. Run conditions support a simple mode with 12 trigger options covering scenarios such as all successful, all failed, all completed, at least one successful, at least one failed, and no failures with at least one success. They also support an advanced mode where you configure a set of states for each upstream task and connect them with AND / OR. Tasks that do not meet the run conditions end in the Excluded from Run state with error code ExecutionConditionsNotMet. Step 6: Parameters
value = dlcutils.widgets.get('your_param')
Step 7: Monitoring Metrics, Alarms, and Failure and Timeout Policies
Centrally maintained and directly used by this task:
Capability | Central document |
Monitoring Metrics / Alarms | |
Failure and Timeout Policy | |
Step 8: Save
Click Create Task (new scenario) or Save Task (edit scenario).
Parameter Description
Path Example
Source | Example |
Workspace | /Workspace/your_username/data_etl.ipynb
|
Workspace (Git Folder) | /GitFolder/your_username/etl.ipynb
|
Git remote repository | notebooks/etl/daily.ipynb
|
Task Execution Description
For tasks that use a Serverless resource group, the environment configuration in the Notebook file, including the base environment + dependencies installed via pip install, is automatically installed before the task runs. You do not need to install it separately at runtime.
Scheduled runs use the latest file code and the latest task configuration. If a file is modified during a run, the next run uses the new code.
Use Limits
The compute resource is selected by the user during task configuration and must have the required permissions to run.
If a referenced file is moved or deleted, the task run fails with error code xxx File Does Not Exist at Path.
The Git path is not validated at configuration time, which poses a risk of path errors. We recommend that you save the task and then immediately run a debug test to verify it.
The custom resource configuration cannot exceed the maximum quota of the selected compute resource.
FAQs
Q1: Which compute resources can a Notebook task use?
When configuring a Notebook task, you can select from a dropdown list the compute resources bound to the current workspace. The dropdown list displays resource names, quotas, and available quotas. For details on supported resource types and quota management, see Compute Resource Overview. Q2: When a Notebook is referenced from Git, is the latest code pulled for every run?
Yes. Each time a Notebook task runs, including scheduled runs and debug runs, the latest code is obtained from the remote Git repository.
Debugging run: Use the Git configuration of the current runner.
Scheduling run: Use the Git configuration of the task's Run As account.
Q3: Can a parameter value reference the output rows of an upstream SQL task?
Yes. Write it in the task parameters:
parameter = {{tasks.<upstream SQL task name>.output.first_row.<column name>}}
References