Quality anomaly diagnosis and lineage analysis enable you to quickly locate the root cause of data quality issues through natural language or alarm notifications. Buddy automatically traces the lineage, evaluates the impact scope, and provides repair suggestions, waiting for your confirmation at key points before executing repair operations.
Overview
Troubleshooting data quality issues often requires analysis across multiple systems and links, which is time-consuming and labor-intensive. Quality anomaly diagnosis and lineage analysis cover the following scenarios:
Quality Rule Alarm Diagnosis: When rule detection fails in a quality task, the system automatically analyzes the failure causes (data-side issues, rule configuration issues, and upstream delays), evaluates the impact scope, and provides repair suggestions.
Anomaly Detection Result Diagnosis: When anomaly detection identifies freshness or integrity anomalies, the system automatically traces the upstream lineage, locates the root cause, and evaluates the impact scope.
Lineage Tracing: Trace the upstream and downstream links of abnormal data based on table-level lineage to quickly locate the source and impact scope of issues.
Impact Scope Assessment: When a quality issue occurs in a table, the system automatically analyzes the affected downstream tables and tasks along the lineage to help assess the impact scope of the anomaly.
Quality Trend Analysis: Analyze the historical quality inspection trends of a table, identify periodic abnormal patterns, and suggest adjusting rule thresholds or sensitivity.
Repair Suggestions and Implementation: Provide repair suggestions based on diagnosis results (rerun upstream tasks, adjust thresholds, or contact the table Owner), and support one-click execution of repair operations.
Tip
Lineage analysis supports table-level lineage tracing, but does not currently support field-level lineage.
Prerequisites
Buddy has been enabled for the workspace.
The current account has Buddy development mode permissions.
You have view permissions on the tables to be diagnosed.
Lineage tracing requires that lineage data already exists in Catalog.
Information You Must Provide
Before performing a diagnosis, Buddy will ask you to provide the following minimum information set, while the rest is automatically analyzed by AI:
Information Item | Description |
Abnormal Table | Name of the table with quality issues |
Issue Description | Alarm type and abnormal phenomena (which can be implicitly expressed through natural language or directly triggered from alarm notifications) |
If a diagnosis is triggered by clicking an alarm notification, Buddy automatically obtains the table name and alarm details without requiring manual input.
AI-Generated Analyses
Analysis Item | AI generation method |
Root cause analysis | Determine the cause of the exception based on alarm details, data changes, and upstream task status. |
Lineage graph | Display upstream and downstream lineages based on table-level lineage relationships. |
Impact Scope | Trace downstream along the lineage and list affected downstream tables and tasks. |
Trend analysis | Analyze historical detection data and identify periodic abnormal patterns. |
Fixing recommendations | Provide a repair solution based on the root cause (rerun the task / adjust the threshold / contact the Owner). |
Operation Steps
Step 1: Triggering Diagnosis
There are two ways to trigger a diagnosis:
Method 1: Trigger from an alarm notification
When a quality alarm is triggered, click the alarm notification to automatically open a diagnostic conversation in the Buddy sidebar. Buddy automatically obtains the alarm context information.
Method 2: Describe in natural language in Buddy
Example Input:
An alarm was triggered for the null rate of dwd_order_detail. Please help me find the cause.
Anomaly detection for the ods_user table shows a freshness anomaly. Please help me troubleshoot it.
The completeness of this table has dropped sharply by 60%. Please help me check which downstream items are affected.
dws_daily_gmv has triggered frequent alarms over the past week. Should the threshold be adjusted?
Please show me the upstream lineage of dwd_order_detail.
Step 2: Viewing Root Cause Analysis
Buddy automatically analyzes the cause of the anomaly and displays the diagnostic results:
Anomaly Summary: alarm rule, trigger time, anomaly metric, and actual value
Root Cause Determination: data-side issue (upstream data anomaly) / rule configuration issue (overly strict threshold) / upstream task delay
Evidence Chain: key data points that support the root cause determination
Step 3: Viewing Lineage Tracing and Impact Scope
Buddy displays the table-level lineage analysis:
Upstream Tracing: Trace upward from the abnormal table to display the data source lineage and mark abnormal nodes.
Downstream Impact: Trace downward from the abnormal table to list all affected downstream tables and associated tasks.
Impact Assessment: number of affected downstream tables, number of associated tasks, and business impact level
Tip
Lineage tracing is based on table-level lineage relationships and does not currently support field-level lineage analysis.
Step 4: Viewing Trend Analysis (Optional)
If you need to understand historical trends, Buddy analyzes historical quality inspection data:
Recent trend of detection metrics
Periodic anomaly pattern recognition
Threshold adjustment suggestions (for example, if the current threshold is too strict, causing frequent false positives)
Step 5: Confirming and Executing Remediation
Buddy provides repair suggestions based on the diagnostic results, and you can execute them with one click after confirmation:
Fix action | Description |
Rerun upstream tasks. | When upstream data is abnormal, trigger the upstream task to rerun. |
Adjust alarm thresholds. | When the threshold is unreasonable, adjust it as recommended. |
Contact table Owner. | When manual intervention is required, automatically send a notification to the table Owner. |
Tip
All repair operations are executed only after your confirmation, and Buddy does not automatically execute any repair operations.
Key Milestone Confirmation
Confirmation Item | Confirmation Timing | Description |
Diagnostic Scope | Before Diagnosis | Confirm the tables and alarm types to diagnose. |
Fixing solution | Before Fix Execution | Review the fixing recommendations and execute after confirmation. |
Threshold Adjustment | Before Adjustment | Confirm the new threshold settings. |
Rerun Task. | Before Rerun | Upstream tasks to be rerun for confirmation. |
FAQs
Q: Does Diagnosis Support Field-Level Lineage?
A: Currently, only table-level lineage tracing is supported, and field-level lineage analysis is not supported. This capability may be extended in future versions.
Q: How Do Alarm Notifications Trigger Buddy Diagnosis?
A: When a quality alarm is triggered, click the alarm notification to automatically open a diagnostic conversation in the Buddy sidebar. Buddy automatically obtains the alarm context information, such as the table name, rules, and metrics, and starts diagnostic analysis directly.
Q: Does Rerunning Upstream Tasks with One Click Affect Other Tasks?
A: A rerun operation is executed according to the task's normal scheduling logic and does not affect other independent tasks. However, if the rerun task has downstream dependencies, the downstream tasks may be triggered to rerun, and Buddy will prompt you with the impact scope upon confirmation.
Q: How Much Historical Data Is Required for Trend Analysis?
A: It is recommended to have at least 7 days of historical quality inspection data to effectively identify periodic abnormal patterns. The more data you have, the more accurate the trend analysis will be.
References
Data Quality Overview