Overview
Data Lineage records the upstream and downstream relationships between tables, views, models, and the tasks that produce them, answering two core questions:
Impact Analysis (Downstream Tracing): If I modify this table, which downstream tasks, reports, and models will be affected?
Data Lineage (Upstream Tracing): Which source tables does the data in this table or model come from, and which task generates it?
DataBuddy provides a Lineage Tab on the table and model details pages, automatically collects lineage information reported by various execution platforms such as SQL development, Notebook, Workflow tasks, Data Integration, and Dashboard, and offers two viewing modes: Graph and List.
This document describes how to view data lineage, node information, collection scope, and typical use cases.
Prerequisites
The current account has permission to view the target asset in the target workspace.
Upstream and downstream relationships depend on reporting from each execution platform, and only tasks that have been run generate lineage records.
Use Limits
Limit | Description |
Lineage dimension | Table-level lineage is currently supported; field-level lineage is planned for subsequent iterations. |
Asset type | Tables (including views, inference tables, and feature tables) and models |
Task type | SQL development, Notebook, Workflow tasks, Data Integration (offline/real-time), BI (Dashboard) |
Collection trigger | Lineage is automatically reported when tasks run, including test runs. |
Deletion linkage | When a task or file is deleted, the platform automatically cleans up the corresponding lineage to avoid leaving invalid nodes. |
Viewing Lineage
Entries
Asset type | Entries |
Table / view | Table details page Lineage Tab |
Model | Model version details page Lineage Tab |
After you go to the Lineage Tab, the List mode is displayed by default. You can click the icon in the upper-right corner to switch to the Graph mode.
List Mode
List mode displays the first-level upstream and downstream relationships of the current asset in a list format, making it suitable for quick viewing and searching by name.
Table List Mode
Column | Description |
Name | Upstream and downstream asset name (table / view / inference table / feature table / model / SQL / Notebook / Task). Click to go to the details page. |
Direction | Whether the item is an upstream or downstream of the current table. |
Type | Table / view / inference table / feature table / model / SQL / Notebook / Task |
Fuzzy search by asset name and paginated browsing are supported.
Model List Mode
The list mode of a model displays only one level of upstream and downstream relationships:
Column | Description |
Name | Upstream and downstream asset name |
Direction | Upstream: data table / feature table; downstream: data table / inference table |
Process Type | Upstream process: Notebook / Studio; downstream process: model service |
Last updated | Last update time of the asset |
Graph Mode
Graph mode displays lineage relationships in a topology diagram, making it easier to understand the overall structure. To switch, click the Graph mode tab in the upper-right corner of List mode.
Node Type
Nodes in the lineage graph are divided into asset nodes and task nodes.
Node Type | Sub-type | Icon |
Asset | Tables (including views, inference tables, and feature tables) | Icons for tables / views / inference tables / feature tables |
Asset | Model. | Model icon (identifiable in the legend at the lower right corner of the canvas) |
Task | TASK (Workflow task) | TASK icon |
Task | SQL (SQL file in Studio) | SQL icon |
Task | NOTEBOOK | NOTEBOOK icon |
Task | Data Integration (offline/real-time) | Icons for offline / real-time integration |
Task | BI(Dashboard) | BI icon |
Node information
Click a node in the graph to view its details. The information displayed for each node type is as follows:
Asset node (table, view)
Business information: description, Tag, creation time, and owner.
Technical information: data source type, Catalog, Schema, storage path, last DDL change time, and last data change time.
Across data sources: the Kafka node name is displayed as the topic name, and the ES node name is displayed as the index name.
TASK node (Workflow task)
Basic information: task name (clickable), task type, description, creation time, update time, creator, and associated workflow (clickable).
Business code: The task code is displayed on the right.
SQL node (SQL file in Studio)
Basic information: run record, type, last run time, and source (clickable).
Business code: The SQL content is displayed on the right.
Notebook node
Basic information: file name (clickable), creation time, and creator.
Business code: The Notebook content is displayed on the right.
Model node
Unique identifier: model name (clickable) and version.
Action: Click View Details to go to the model details page.
Data Integration node
Task name (clickable to go to the task details page), task type (real-time or offline), creator, and creation time.
BI node
BI name (clickable to go to the Dashboard), creator, and creation time.
Canvas Operations
Operation | Description |
Search | Search box in the upper right corner of the canvas, fuzzy search and locate by node name (including machine learning model name). |
Time Filtering | Filter lineage nodes by reporting time range. |
Zoom / Pan | Zoom with the mouse wheel and pan by dragging to view complex lineage networks. |
Legend | A legend of node types is displayed in the lower right corner of the canvas, allowing quick identification among various asset/task icons. |
Lineage Collection and Update
Automatic Collection
DataBuddy automatically collects lineage information from the following execution platforms:
Source | Reporting Content |
Studio SQL files / Workflow SQL tasks | The compute engine dynamically parses SQL input and output tables during execution, and reports them upon each submission (including test runs). |
Spark Jar tasks | Reports upon task running. |
Notebook | Tables referenced by SQL / Python code executed in Notebook |
Workflow tasks | Reported with the task during scheduled runs. |
Data Integration (offline/real-time) | The Data Integration module periodically reports the relationship between source and target tables. |
BI(Dashboard) | The BI module periodically reports the relationship between Dashboard and datasets and source tables. |
Based on the dynamic lineage parsing capability of DLC SQL in the Spark kernel, the system uniformly collects all SQL and jobs submitted to the engine.
Cleanup upon Deletion
When a task or file is deleted, the platform reports the deletion event to the lineage system, automatically cleans up the corresponding lineage relationships, and prevents invalid nodes from remaining in the lineage graph:
Object | Trigger Action |
Task (Workflow Task) | Task Deleted |
File (SQL file in Studio) | File Deleted |
Time Metadata
Lineage nodes carry time metadata to help distinguish active nodes from historical nodes:
Field | Description |
Task creation time. | Time when the task was created, used to distinguish new tasks from old ones. |
Last run time of task | Last run time |
Last run time of SQL | Last run time of the SQL file |
Model Lineage
Model lineage is an extension of table lineage and corresponds to the asset relationships throughout the entire machine learning lifecycle:
Phase | Node Type | Relationship Description |
Raw Data → Feature Table | Upstream: data table; downstream: feature table | Follow table lineage logic (data processing task) |
Feature Table → Training Data | Upstream: feature table; downstream: training data table | Data Processing / Feature Extraction Task |
Training Data → Model | Upstream: table; downstream: model node | Model training task (sourced from Notebook or Studio task) |
Model → Inference Table | Upstream: model; downstream: inference table | Model Service Task |
Key behaviors of model lineage:
The unique identifier of a model uses the model name and version.
There is no concept of "model update". A new version is treated as a new lineage node.
When a task is deleted, the associated lineage nodes are synchronously removed along with their connections or nodes.
Typical Use Cases
Impact Analysis: Assessing Downstream Effects Before Modification
Before you prepare to modify a fact table:
1. Go to the Lineage Tab on the table details page.
2. In list mode, view the downstream asset list: which SQL files, Workflow tasks, dashboards, and models directly depend on this table.
3. Switch to graph mode to further view the second-level and third-level downstream dependencies.
4. Notify the relevant owners in advance and arrange regression testing.
Data Tracing: Locating the Source of Anomalies
When you detect abnormal data in a table:
1. Go to the Lineage Tab on the table details page.
2. In graph mode, expand upstream to find the task node that produces this table (Notebook / Workflow Task / SQL).
3. Click a task node to view its task code and last run time.
4. Go to the task details page to inspect the specific logic or run logs.
Model Governance: Version Comparison and Downstream Assessment
Before switching the model version:
1. Go to the Lineage Tab on the model details page.
2. In list mode, view the downstream inference tables and model services.
3. Evaluate the impact of switching versions on online services.
Relationships with Other Modules
Module | Relationship |
Workflow | Workflow tasks automatically report lineage during runtime, and task nodes appear in the lineage graph. |
Studio | SQL development and Notebook automatically report lineage when executing SQL. |
Model Catalog | Metadata source of model nodes |
Data Integration | Offline / real-time synchronization tasks participate in the graph as lineage nodes. |
Dashboard | Dashboard participates in the graph as a downstream lineage node. |
FAQs
Q: Why hasn't the SQL I just ran appeared in the lineage graph yet?
A: There is a short delay in lineage collection and graph mode rendering. Wait for 1-2 minutes and then refresh the Lineage Tab. If it still does not appear after a long time, check whether the task was executed successfully, whether the SQL uses the three-part catalog.schema.table path, or whether the current SQL context is set correctly.
Q: Does SQL run during testing also generate lineage?
A: Yes. The engine performs lineage parsing for all submitted SQL and jobs, including test runs. If you do not want test runs to be reported, use temporary tables or a dedicated test Schema.
Q: Will lineage nodes disappear immediately after a task is deleted?
A: When a task is deleted, the platform reports a cleanup event, and lineage nodes are typically cleaned up synchronously within a short period of time.
Q: Can I view field-level lineage?
A: The current version supports only table-level lineage. Field-level lineage is already planned for future iterations.
Q: Can cross-Catalog lineage be displayed correctly?
A: Yes. Lineage nodes are uniquely identified by the three-part catalog.schema.object path, and lineage across Catalogs and workspaces can be displayed in the graph. Whether it is visible to you is still controlled by the access permissions of the current account.
References
Dashboard Overview