Quality monitoring tasks are used to schedule and run quality tasks defined in the data quality module in Workflow. They perform rule checks on monitored objects (tables and fields), and the results are stored in run records.
Overview
Key features of quality monitoring tasks:
Do not edit rules in Workflow: All quality rules are maintained in the Data Quality module.
Orchestrated as a workflow node: It can be mixed with other task types and run according to dependencies in the DAG.
Run Results Reflect Rule Checks: The check result of each rule (normal, abnormal, or failed) is displayed on the task run details page.
For general configurations (task name, upstream dependencies, run conditions, parameters, monitoring metrics, alarms, and failure and timeout policies), see Task General Configuration.
Prerequisites
The target quality task has been created and published in the Data Quality module. For details, see Create Quality Rules.
The workspace is bound to a data computing or data analysis Serverless resource group.
Operation Steps
Step 1: Creating a Job
1. In the workflow list, click the workflow name to go to the workflow configuration. Select Task Orchestration, and then click + Create Task on the workflow canvas.
2. Select Data Quality as the task type.
Step 2: Configuring Basic Information
Parameter | Description | Required |
Task Name | Unique within the same workflow. | Yes |
Task type | Data quality. You can switch to other task types from the dropdown list. After switching, the parameter area will be cleared. | Yes |
Task Sources | Reference the published rule sets in the data quality module. | Yes |
Step 3: Selecting a Quality Job
1. Click Task Source, and then select the quality task to run from the drop-down list.
2. Only published rule sets are displayed.
3. Fuzzy search is supported.
4. After selection, the task name is displayed in the configuration panel. Click it to go to the details page of the corresponding quality task.
Step 4: Configuring Compute Resources
Parameter | Description | Required |
Computing Resource | Data computing or data analysis Serverless resource group. | Yes |
Tip
Orchestration scenario vs. trial run scenario: In workflow orchestration (the configuration described in this section), you can select data computing or data analysis resources. In the trial run entry of the Data Quality module, the drop-down list displays only data analysis resources.
Step 5: Upstream Dependencies, Parameters, Monitoring Metrics, Alarms, and Failure and Timeout Policies
See Task General Configuration.
The two ways to integrate with workflows are as follows:
Method | Configuration Location |
Create a quality task node in a workflow (this section). | Workflow Orchestration Canvas |
Associate a quality task with a workflow (scheduled at the same frequency as a published task). | Create Quality Rule. |
In the second method, the quality task appears on the canvas of the associated workflow (read-only) and serves as an online check.
Step 6: Saving the Job
Click Create Task or Save Task.
Job Run Details
Quality monitoring task run details page:
Area | Display |
Overall execution status | Pending run / Running / Succeeded / Failed / Terminated due to timeout |
Detection status | Normal / Abnormal / Detecting / Not detected |
Normal / total rules | For example, 8 / 10 |
Rule execution details | One row per rule: rule name / template type / monitored object / detection result / actual value / operation (log, execution history) |
For detailed field descriptions, see Quality Task Ops (to be generated).
Use Limits
Only published quality tasks can be referenced. Quality tasks in draft state are not displayed in the drop-down list.
A quality task can have only one published version at a time. After editing, a new draft version is generated. The original published version remains effective, and workflow scheduling always uses the latest published version, so the existence of a draft does not cause failures.
When a quality task is scheduled in Workflow, the Run As account of the workflow task is used. This account must have access permissions to the monitored objects (tables and fields).
Alarms and monitoring metrics for quality monitoring tasks apply only to task execution (run duration, wait duration, and so on). Business alarms triggered by rules are handled through Quality Rule Alarm Configuration (to be generated).
FAQs
Q1: What is the difference between a quality task run failure and a quality rule detection exception?
Status | Description |
Task Execution Failed | The task cannot run normally (for example, due to code errors or insufficient resources). |
Detecting Exceptions | The task ran successfully, but the rule detection result does not meet the business constraint. |
You can handle the two scenarios separately in the alarm configuration: failure triggers a task failure alarm, while detection exceptions are sent through the rule alarm configuration.
Q2: Can quality rules be edited in Workflow?
No. Quality rules are maintained in the Data Quality module, and Workflow only serves as the scheduling and execution entry. To modify rules, go to the Data Quality module.
Q3: Can a downstream quality task run detection immediately after an upstream task outputs a table?
Yes. In Workflow, set the quality task as a downstream task of the table write task:
[Notebook task: Write table] → [Quality task: Check table]
After upstream dependencies are configured, the quality task waits for the table write task to succeed before starting detection.
References
Task General Configuration
Create a quality rule
Quality Task Ops