Notebook file execution depends on Remote kernels: each Notebook file corresponds to one Kernel, which is started on Serverless compute resources. This section describes how to connect, create, switch, and release Kernels, as well as environment management and execution result handling.
Overview
The running model of Notebook in DataBuddy is as follows:
Each Notebook file corresponds to one Kernel.
The Kernel runs on the job cluster compute resource in the workspace, with pay-as-you-go billing and dynamic scaling.
Each Kernel has an independent runtime environment, including the Python interpreter, installed dependency libraries + Spark configurations.
When you create a new Kernel for the same Notebook file, the previous Kernel is automatically released, and the variables and cache in the original session are cleared.
Run results are displayed below each cell on a per-cell basis, with support for tabular interactions such as sorting, filtering, and downloading.
Prerequisites
The current workspace is bound to available job cluster compute resources. For more information, see Compute Resource Overview. You have Run, Edit, or Manage permission on the current Notebook.
To access tables in a Catalog, you must have the corresponding data access permissions.
Operation Steps
Step 1: Connecting to the Kernel
1. Open the Notebook file.
2. In the upper-right corner of the editor, click Not Connected, select a compute resource, and connect using the default environment configuration.
3. To modify environment configuration parameters, click Modify Configuration, fill in the parameters in the Environment drawer (see New Kernel Parameters below), and then connect.
4. After you click Connect, the current Kernel connection progress is displayed.
5. After a successful connection, the status in the upper-right corner shows "Compute Resource Name (Connected)".
Step 2: Running Cells and Running All
1. To run a cell: click Run on the left side of the cell, or use the shortcut ⌘ + Enter.
2. Run All: Click Run All in the toolbar to execute all cells in order.
3. Run Partial: After selecting a code snippet, use Ctrl + Shift + Enter to run only the selected part.
4. Run all cells above or below: use ⌥ + Shift + ↑ / ⌥ + Shift + ↓.
5. Stop: Click Stop in the toolbar to immediately terminate the currently running cell.
Step 3: Processing Execution Results
3.1 Interacting with Table Results
When you call display(df) or run pandas.DataFrame / pyspark.sql.DataFrame, the data is displayed in a table below, supporting:
Operation | Description |
Copying | Select the cell range, right-click to copy, or use ⌘ + A / ⌘ + C to select all and copy. |
Download | Click Download in the upper-right corner. CSV / Excel / TXT formats are supported. |
Sort | Click a column name to sort in ascending / descending order. |
Field Configuration | Select to show / hide columns, freeze columns, and adjust column order. |
Data Search | Search for specified text in the result area, highlight matches, and support switching between previous and next matches. |
Result Filtering | Add filter conditions (connect multiple conditions with AND / OR). |
3.2 Hiding and Viewing Execution Information
After a cell finishes running, hover over the output area to view the user who ran it, the run time, and the duration.
Click the cell collapse button to hide the run results.
Step 4: Managing Environments
Click the Environment Management drawer on the left side of the Notebook file to view and adjust the runtime environment:
Area | Description |
Resource mode | Supports selecting distributed or single-node. If you select distributed mode, the system starts one Driver and multiple Executor nodes, which is suitable for distributed processing of big data. If you select single-node mode, the system starts only one Driver as the single-node computing resource, which is suitable for running pure Python code. Note: If you select single-node mode but use SQL or Spark in your code, the system automatically switches to distributed mode and starts one Driver and one Executor node of the configured specifications. |
Basic environment | Supports selecting default or custom. If you select Default, the corresponding resource group has a built-in image. If you select Custom, you can select a .yml/.yaml file that declares a list of dependencies. When the Kernel starts, the system automatically installs the dependencies from the file on top of the built-in image. |
Advanced Parameters | Supports customizing specification parameters of the runtime environment, such as the number of Spark executor CUs (spark.executor.memory) and the number of Spark executors (spark.executor.number). |
Dependency library | After connecting to compute resources, you can view the list of dependency libraries installed in the current environment, including both default and custom installations. |
Application | After modification, click Apply to rebuild the kernel with the new configuration. This clears dependencies installed through code and defined variables. |
Reset environment. | Click More > Reset Environment to restore the runtime environment to the base environment state and clear all dependencies and variables installed through pip install. |
Reset environment. | Click More > Export Environment to export the current runtime environment as a .yaml file and save it to the Studio development directory, making it easy to reuse the environment configuration across different files. |
Note:
After you click Apply or Reset Environment, the kernel is rebuilt. Dependencies installed via pip install and variables other than dlcutils.widgets.text(...) are cleared.
Adding Dependencies via pip install
%pip install pandas==2.2.0 jieba
Dependencies take effect in the current session immediately after installation. They are cleared after you create a new session or reset the environment, and you need to reinstall them.
Step 5: Restarting or Releasing the Kernel
Restart Kernel: Click Restart Kernel above the file to restart only the Kernel. Existing Spark apps are not recreated.
Disconnect: Click the resource group name in the upper-right corner and then Disconnect. After you confirm in the dialog box, the current Spark APP is released and the status returns to "Disconnected".
Automatic Release Mechanism: The Kernel is automatically released after its inactive time reaches the configured value (10 minutes by default), and you need to reconnect when you enter the file again. The Spark APP is automatically released after its inactive time reaches the configured value (2 hours by default). If the Spark APP has not been released after the Kernel is released, the existing Spark APP is directly connected upon the next connection.
Use Limits
The Kernel supports connecting only to job cluster - Spark compute resources. Other types are not currently supported.
Dependencies installed via the %pip install command are lost after you create a new kernel or reset the environment. To persist them, use the custom base environment feature.
The system supports displaying only the first 10,000 rows of the run results, with a size not exceeding 2 MB. The result set download follows the same limit.
FAQs
Q1: What should I do if "Kernel Not Connected" is reported at runtime?
Choose Not Connected > Select Compute Resource in the upper-right corner to create a Kernel. If no compute resources are available, contact the workspace administrator to create one in Create Compute Resource. Q2: How do I force a stop if the run is stuck?
Click Stop in the toolbar to terminate the current cell. If it is still not released, click Disconnect in the resource group drop-down in the upper-right corner, and then recreate the Kernel.
Q3: What should I do if an out-of-memory (OOM) error occurs?
Click Environment Management > Advanced Parameters and increase spark.executor.memory / spark.driver.memory.
References