Quality task creation and scheduling enables you to complete intelligent recommendation, configuration, trial runs, and workflow scheduling of quality rules through natural language descriptions. Buddy automatically analyzes table structures and historical data characteristics, recommends suitable rule types and thresholds, and publishes the rules as formal quality tasks after waiting for your confirmation at key checkpoints.
Overview
Data quality rules are a key means of ensuring data trustworthiness. Traditional approaches require table-by-table analysis and rule-by-rule authoring, which are inefficient and prone to omissions. Quality task creation and scheduling covers the following scenarios:
Intelligent Rule Recommendation: Analyzes table structures, field types, and historical data distributions to automatically recommend suitable quality rules (such as null rate, uniqueness, value range, data volume, and freshness).
Automatic Rule Generation: AI automatically generates quality rule configurations, which can be published as formal quality tasks after user confirmation, ensuring rules are accurate and controllable.
Intelligent Threshold Setting: Automatically sets reasonable alarm thresholds based on statistical characteristics of historical data (mean, standard deviation, and quantiles), avoiding thresholds that are too loose or too strict.
Rule Template Reuse: Supports batch application of rule templates to tables with similar structures under the same Schema, allowing one-time configuration and reuse across multiple tables.
Trial Run: After a quality rule is created, you can run a trial immediately to verify whether the rule effect and alarm thresholds are reasonable, avoiding false alarms.
Workflow Scheduling: After a trial run passes, users can choose to add quality tasks to an existing workflow or create a new workflow for scheduled execution, enabling automated periodic quality checks.
Prerequisites
Buddy has been enabled for the workspace.
The current account has Buddy development mode permissions.
You have view permissions on the tables for which quality rules need to be configured.
Have the permission to create rules in the quality module.
To run quality rules, available compute resources are required.
Information You Must Provide
Before generating quality rules, Buddy will ask you to provide the following minimum information set, while the rest is automatically recommended by AI:
|
Target Table | Table name for which a quality rule needs to be configured (single table or batch supported) |
Rule Requirement | Recommended rule / Specify rule type / Template reuse (can be implicitly expressed through natural language). |
If your input is incomplete, Buddy will ask for the missing information through an "Ask User card" before continuing.
Configurations That AI Can Intelligently Generate
Configuration Item | AI generation method |
Rule Type | Analyze field types and business semantics, and recommend null rate / uniqueness / value range / data volume / freshness, and other rule types. |
Detection Field | Automatically select fields to detect based on field names and types (for example, primary key → uniqueness, time field → freshness). |
Alarm threshold | Automatically set based on statistical characteristics of historical data (mean +/- N times standard deviation, quantiles). |
Rule Name | Default: {table_name}_{rule_type}_{field_name} |
Rule Description | Automatically generate a readable description based on the rule configuration. |
Operation Steps
Step 1: Expressing Quality Rule Requirements
Describe your requirements in natural language in the Buddy input box, and Buddy will automatically identify the rule type and target table.
Example Input:
Recommend quality rules for the dwd_order_detail table.
Create a uniqueness rule for the order_id field with a null rate not exceeding 1%.
Add freshness detection rules to all tables in the ods Schema.
Referring to the rules of dwd_order_detail, configure a set for dwd_payment_detail as well.
Step 2: Completing Missing Information
If the input is incomplete, Buddy will ask you for the missing information through an Ask User card:
If no target table is specified, the tables in the current Schema are listed for you to choose from.
If no reference table is specified during template reuse, you will be asked to confirm.
After you answer, Buddy will continue the execution.
Step 3: Reviewing AI-Recommended Rules
Buddy displays the recommended rule set in structured cards, with each rule containing:
Rule name, rule type, and detected field.
Alarm thresholds and determination logic.
Recommendation reason.
Tip
You can modify any rule configuration on the card before confirming, and Buddy will apply the modified rules.
Step 4: Confirming and Creating Rules
After you confirm the rule configuration, Buddy calls the DataBuddy API to create a quality rule task and displays:
Number of rules successfully created.
Task ID and status of each rule.
Failed items and reasons.
Step 5: Test Run
After a quality rule is created, you can use natural language to request an immediate trial run to verify the rule's effectiveness:
Example Input:
Run a trial on the newly created rule to see how it works.
Buddy will perform a trial run and display the results:
Detection result of each rule (passed or failed).
Alarm trigger status and specific values.
Whether there are false positives and suggestions for threshold adjustment.
A trial run helps you verify whether the rule configuration is reasonable before going live, avoiding unnecessary alarm noise.
Step 6: Adding to Workflow Scheduling
After the trial run passes, you can add quality tasks to a workflow for automated periodic execution:
Example Input:
If the trial run succeeds, add it to the workflow that runs at 2:00 AM every day.
Create a workflow for these rules and run it once a day.
Buddy will:
List existing workflows for selection, or create a new workflow.
Confirm the scheduling time and frequency.
Add quality tasks to the workflow and display the configuration results.
Key Milestone Confirmation
Confirmation Item | Confirmation Timing | Description |
Target Table | Before rule recommendation | Confirm the table for which rules need to be configured. |
Rule Configuration | Before creation | Review rule types, thresholds, and detection fields. |
Batch Application | When reusing a template | Confirm the application scope of template rules. |
Trial Run | Before formal scheduling | Verify rule effectiveness and threshold reasonableness. |
Workflow Scheduling | Before adding to a workflow | Confirm scheduling time, frequency, and workflow selection. |
FAQs
Q: How Are Thresholds Recommended by Buddy Calculated?
A: Buddy reads the statistical characteristics of the target table data from the last N days, such as the mean null rate, mean data volume, and standard deviation, and automatically calculates thresholds based on the mean +/- N times the standard deviation or the quantile method. If historical data is insufficient, industry-standard default values are used.
Q: What Should I Do If Field Names in the Target Table Are Not Exactly the Same When a Rule Template Is Reused?
A: When reusing a template, Buddy performs field name matching. Exact matches take priority, followed by semantic similarity matching (for example, create_time and created_at). For fields that cannot be matched, Buddy skips the rule and marks it in the results.
Q: What Is the Difference Between Automatically Created Quality Rules and Manually Created Ones in the Quality Module?
A: There is no difference at all. Rules created by Buddy and rules created manually are in the same management system, and you can view and manage them centrally in Data Quality Overview.
Q: What Is the Difference Between a Test Run and a Production Schedule?
A: A trial run is a one-time immediate execution used to verify whether the rule effect and thresholds are reasonable, and it does not generate persistent alarms. After the rule is formally added to workflow scheduling, it runs automatically at the configured interval, and triggered alarms go to the alarm notification process.
References
Data Quality Overview