Overview
This document describes how to create a quality task and configure quality rules in it. A quality task is a container for multiple quality rules and can apply to tables or fields. It is edited in the YAML editor. After the task is created, you can proceed to the publishing, scheduling, and running phases.
Prerequisites
Before you start this operation, make sure the following prerequisites are met:
You have activated the DataBuddy service and completed workspace initialization. You must have the Owner role on the target table or the Manager permission on the target table. For details about role permissions, see Members and Permission Management. When a user without the required permissions hovers over the create button, a message appears: "Only the table Owner and users with the table Manager permission can create." A data analysis compute resource is available in the target Workspace for trial runs and scheduled execution.
Use Limits
|
Creation Permission | Only table Owner and table Manager roles can create tables. |
Monitored Object of a Single Task | Only one table can be selected at a time (rules can be configured for multiple fields of the table). |
Rule Editing Mode | Only the YAML editor is supported. The UI form and YAML cannot coexist. |
Rule Type | System built-in templates (table-level and field-level) and custom SQL rules |
Operation Steps
Step 1: Entering the Creation Page from the Catalog or Standalone Menu
DataBuddy provides four creation entries. These entries only affect the default value of monitored objects, and the remaining steps are the same.
|
Quality Tab on the table details page | Data Catalog > Select Table > Quality Tab > Create Quality Task | Skip Select Object. The monitored object defaults to the current table. |
Data quality standalone menu - Catalog level | Left Navigation Data Quality > Select Catalog > Create Quality Task | Go to Select Object. The current Catalog is expanded. |
Data quality standalone menu - Schema level | Left Navigation Data Quality > Select Schema > Create Quality Task | Go to Select Object. The current Schema is expanded. |
Data quality standalone menu - Table level | Left Navigation Data Quality > Select Table > Create Quality Task | Skip Select Object. The monitored object defaults to the current table. |
Step 2: Selecting the Monitoring Object
In the Select Object panel, select the table to be monitored. A quality task supports only one table at a time. After selection, go to Step 3 to configure rules.
Step 3: Creating a Quality Rule (YAML)
1. Supports AI-assisted rule writing.
2. Click Expand Template List, select the desired template name, and then click the insert icon. The editor inserts a complete rule YAML snippet, including the rule name, rule type, system template identifier, detection expression, and alarm settings.
The key fields of a YAML rule are as follows:
|
rule_name
| Rule name, used as the display name in the alarm and Ops list | Yes | Order ID null value detection
|
template
| System template identifier; fill in custom_sql for custom SQL. | Yes | field_null_rate
|
dimension
| Quality dimension (used with template): completeness / uniqueness / timeliness / validity / consistency / accuracy | Yes | completeness
|
target.column
| Target field for field-level rules; can be omitted for table-level rules. | Depends on the template | order_id
|
expression
| Detection expression or custom SQL | Yes | value <= 0.01
|
filter
| Data filtering condition, for example, specifying a partition | No | dt = '${yyyyMMdd}'
|
alert
| Alarm configuration of the rule. "enable" indicates that an alarm notification will be sent after the rule is triggered. The default value is "enable". | No | - |
3. After a template is inserted, the positions in the YAML that require your input are marked with a # suffix, indicating that you need to provide the content there.
4. Click the Help Documentation link above the template list to view the YAML rule configuration instructions.
Step 4: Configuring Basic Settings
In the Basic Configuration area on the right side of the Objects & Rules panel, fill in the following fields:
1. Task name: required. The default value is "Quality Task _ Date _ Timestamp", which can be manually modified.
2. Owner: required. The default value is the current creator, who must be a user with operation permissions on the engine data accessed by this task.
3. Task description: optional. It is recommended to specify the monitoring purpose and covered business scope for easier identification during subsequent Ops.
Step 5: Alarm Configuration
You can configure alarm notifications for quality rules. After the configuration is complete, the system sends an alarm message when a rule is triggered.
1. Select the rule for which you want to enable alarms from the dropdown list. The content in the YAML editor is displayed in sync with the rule selected from the dropdown list.
2. Configure notification channels for the selected alarm rules. If no notification channel has been configured, click Add Channel to go to the Set Notification Channel page and create a channel. Step 6: Saving or Saving and Publishing
Three action buttons are provided at the bottom of the page:
Save and Publish: saves the current YAML and basic configuration, closes the drawer to return to the task list, and immediately opens the publish dialog.
Save Only: saves the current configuration as a draft version and closes the drawer to return to the task list. Draft tasks are not scheduled. You can click Publish in the task list later.
Cancel: opens a confirmation dialog. After confirmation, unsaved YAML and basic configuration changes are discarded.
Note:
Draft tasks cannot be scheduled in workflows. When you select a quality task in a workflow, only the latest published version of the task is displayed.
Parameter Description
Basic Task Configuration
|
Task Name | Display name of the task in the list and alarm | Yes | Quality task_date_timestamp |
Owner | The user to whom the task belongs must have operation permissions on the monitored table. | Yes | Current creator |
Task Description | Business description of the task | No | - |
YAML Rule Fields
See Creating Quality Rules (YAML)
Verification Result
After the task is created, you can go to the standalone Data Quality menu or the task list in the Quality Tab on the table details page > Quality Tab to view the created task information. Click the task name to update the quality rules.
FAQs
Q: Why Can't I See the "Create Quality Task" Button?
A: This button is available only to table Owners and users with the table Manager permission. You can view your role on the table in the permission information on the table details page. If necessary, contact the Owner for authorization.
Q: Can I Create a Task to Monitor Multiple Tables at Once?
A: This feature is not supported in the current release. A quality task can monitor only one table. To cover multiple tables in batches, create a task for each table, or enable Anomaly Detection at the Schema level to obtain zero-configuration fallback monitoring.