tencent cloud

Tencent Cloud DataBuddy

ドキュメントTencent Cloud DataBuddyData GovernanceData Quality OverviewQuality Rule Configuration Guide

Quality Rule Configuration Guide

ダウンロード
フォーカスモード
フォントサイズ
最終更新日: 2026-10-09 16:16:13
AI翻訳

Overview

Data quality rules are used to perform automated quality checks on data tables and fields. By configuring rules, the system can automatically monitor data completeness, accuracy, uniqueness, consistency, validity, and timeliness, and trigger an alarm when the detected value meets the trigger condition.
The configuration file supports the following capabilities:
Two rule types are supported: system templates (system_template) and custom SQL (custom_sql).
Two rule levels are supported: table-level rules (table_rules) and field-level rules (field_rules).
You can define multiple rules in a single configuration file and combine them flexibly.

Core Concepts

Rule Type
Rule Type
Label
Description
System Template
system_template
Use the system built-in detection template and only specify template_code. The system template has built-in quality dimensions, so users do not need to specify dimension.
Custom SQL
custom_sql
Users write custom SQL query statements to flexibly implement various detection logic. Users need to specify dimension (quality dimension) and custom_sql (SQL statement).
Rule Level
Rule Level
Configuration Node
Description
Table-level rule
table_rules
Detect the entire table, such as row quantity check and data output timeliness check.
Field-level rule
field_rules
Detect a specific field in the table, such as null rate, duplicate values, and average values. You need to additionally specify field_name.
Quality Dimension
The quality dimension is used to classify rules and needs to be specified only for the custom_sql type, as it is built into system templates.
Dimension Identifier
Chinese Name
Description
completeness
Completeness
Detect whether data has missing values.
accuracy
Accuracy
Detect whether data values are within a reasonable range.
validity
Validity
Detect whether data formats and types are compliant.
uniqueness
Unique
Detect whether data has duplicate values.
consistency
Consistency
Detect whether data is consistent across different data sources or tables.
timeliness
Timeliness
Detect whether data is produced on time.

System Template Reference

Table-Level Template
Template Identifier
Term
Description
trigger_condition
table_row_count
Number of Rows
Check Whether the Total Number of Rows in the Table Is Within a Reasonable Range.
User-customized
data_timeliness
Data Timeliness
Detect whether data is produced on time.
Fixed by the system and no manual input is required. Configure business filter conditions through filter.
Field-Level Template
Template Identifier
Term
Description
trigger_condition
null_count
Field Null Quantity
Count the number of null values in a field.
User-customized
null_rate
Field Null Rate
Calculate the proportion of null values in a field.
User-customized
not_null_count
Field Non-null Count
Count the number of non-null values in a field.
User-customized
duplicate_count
Field Duplicate Quantity
Count the number of duplicate values in a field.
User-customized
distinct_count
Field Distinct Count
Count the number of distinct values in a field.
User-customized
min_value
Field Minimum Value
Obtain the minimum value of a field.
User-customized
max_value
Field Maximum Value
Obtain the maximum value of a field.
User-customized
sum_value
Field Sum
Calculate the sum of a field.
User-customized
avg_value
Field Average Value
Calculate the average value of a field.
User-customized
median_value
Field Median
Calculate the median of a field.
User-customized
std_dev
Field Standard Deviation
Calculate the standard deviation of a field.
User-customized
enum_range_consistency
Enumeration Range Consistency
Check whether the field value is within the valid enumeration range.
Fixed by the system and does not need to be filled in. Configure the list of valid enumeration values through params.enum_values.

Trigger Condition Expression Syntax

The trigger_condition describes the condition under which the detected value triggers an exception.
Core semantics: an exception is triggered when the trigger_condition is met.

Simple Comparison

Use comparison operators to compare directly with the target value.
Supported operators:
Operator
Description
==
Equal to
!=
Not equal to
>
Greater than
<
Less than
=
Greater than or equal to
<=
Less than or equal to
Example:
trigger_condition: "> 100"  # Triggered when the detected value is greater than 100.
trigger_condition: "<= 0"   # Triggered when the detected value is less than or equal to 0.
trigger_condition: "== 24"  # Triggered when the detected value equals 24.
trigger_condition: "!= 0"   # Triggered when the detected value is not equal to 0.
trigger_condition: "> 0"    # Triggered when the detected value is greater than 0.


Range Expression

Used to check whether the detected value is within or outside a specified numeric range.
Syntax: between or not_between followed by left parenthesis, lower bound, comma, upper bound, and right parenthesis.
Parentheses description:
Bracket
Description
[ or ]
Includes boundary values (closed interval)
( or )
Excludes boundary values (open interval)
Expression description:
Expression
Description
Trigger Timing
between [0,150]
The value falls within the range of 0 to 150.
Triggered When Value Is Within the Range.
between (0,100)
The value falls within the range of 0 to 100.
Triggered When Value Is Within the Range (Boundaries Excluded).
between [0,100)
The value falls within the range of 0 to 100.
Triggered When Value Is Within the Range (Left Boundary Included, Right Boundary Excluded).
between (0,100]
The value falls within the range of 0 to 100.
Triggered When Value Is Within the Range (Left Boundary Excluded, Right Boundary Included).
not_between [0,100]
The value is not within the range of 0 to 100.
Triggered When Value Is Outside the Range.
not_between (0,100)
The value is not within the range of 0 to 100.
Triggered When Value Is Outside the Range.
In real-world scenarios, not_between is often used to indicate that an exception is triggered when the detected value falls outside the normal range.
Example:

trigger_condition: "not_between [1000,1000000]"   
# Triggered when the table row count is outside this range.
trigger_condition: "not_between [10, 10000]"      
# Triggered when the field average is outside this range.
trigger_condition: "not_between [1, 50]"          
# Triggered when the distinct count is outside this range.


Combining Conditions

You can use AND (all conditions must be met) and OR (any condition can be met) to combine multiple conditions.
Connector
Description
AND
Triggers when all conditions are met simultaneously.
OR
Triggers when any condition is met.
Example:
Option 1

trigger_condition: "< 100 OR > 1000000"  # Triggered when the row count is too low or too high.
trigger_condition: "> 100 AND < 200"     # Triggered when the value is greater than 100 and less than 200.
Note:
The range in between and not_between expressions will not be mistakenly split into multiple conditions.

Custom SQL Field Name Prefix

When the rule type is custom_sql, each condition in trigger_condition must include a field name prefix, and the field name must match the column name returned by the SQL query.
Example:
Option 1

trigger_condition: "check_value > 0"                
# The column name returned by SQL is check_value.
trigger_condition: "null_count == 0"                         
# The column name returned by SQL is null_count.
trigger_condition: "check_value > 0 AND total_count >= 100"  
# Multi-Column Combined Evaluation
trigger_condition: "check_value not_between [0, 100]"        
# Ranges are also supported.

Template and Trigger Condition Compatibility Constraints

Different templates have different configuration requirements for trigger_condition. Strictly follow the constraints below.
Template or Type
trigger_condition Configuration Method
Additional Notes
Standard system templates (such as table_row_count, null_rate, and avg_value)
User-customized. Supports simple comparison, range, and combined conditions.
None
data_timeliness template
No manual input is required. The trigger condition is fixed by the system.
Configure business filter logic through the filter field.
enum_range_consistency template
No manual input is required. The trigger condition is fixed by the system.
Configure the list of valid enumeration values through params.enum_values.
Custom SQL
User-customized. Supports all expression formats with field name prefixes.
Field names in trigger_condition must be consistent with the column names returned by SQL.

Configuration Field Description

Common Fields

Field
Type
Required
Description
rule_name
string
Yes
Rule name. A descriptive name is recommended.
rule_type
string
Yes
Rule type: system_template or custom_sql
trigger_condition
string
Depends on the template
Trigger condition expression (fixed by the system for some templates and does not need to be entered)
filter
string
No
Optional. Adds a WHERE filter condition.

System Template-Specific Fields

Field
Type
Required
Description
template_code
string
Yes
System template identifier, queried through the DescribeRuleTemplates API
params
object
No
Additional parameters required by some templates

Custom SQL-Specific Fields

Field
Type
Required
Description
custom_sql
string
Yes
Custom SQL query statement
dimension
string
Yes
Quality dimension: completeness, accuracy, validity, uniqueness, consistency, timeliness

Field-Level Rule-Specific Fields

Field
Type
Required
Description
field_name
string
Yes
Name of the Target Detection Field

Configuration Example

Table-Level Rule Example

Example 1: Table row quantity range check (system template)
Option 1
table_rules:
  - rule_name: Table row count range check
    rule_type: system_template
    template_code: table_row_count
    trigger_condition: not_between [1000,1000000]
    filter: status != DELETED
Note:
Use the table_row_count template to check the table row count. Trigger a medium alarm when the row count is not between 1000 and 1000000, and use filter to exclude deleted data.
Example 2: Table row quantity combined condition check (system template)
Option 1
table_rules:
  - rule_name: Table row count anomaly check
    rule_type: system_template
    template_code: table_row_count
    trigger_condition: < 100 OR > 1000000

Example 3: Data output timeliness check (system template - fixed trigger)
Option 1
table_rules:
  - rule_name: Data output timeliness check
    rule_type: system_template
    template_code: data_timeliness
    filter: update_time >= DATE_SUB(NOW(), INTERVAL 2 HOUR)
Example 4: Cross-table data consistency check (custom SQL)
Option 1
table_rules:
  - rule_name: Order amount consistency check
    rule_type: custom_sql
    dimension: consistency
    custom_sql: >
      SELECT ABS(
        (SELECT SUM(order_amount) FROM order_table WHERE dt = 20260101)
        - (SELECT SUM(detail_amount) FROM order_detail_table WHERE dt = 20260101)
      ) as check_value FROM dual
    trigger_condition: check_value != 0

Note:
Use custom SQL to calculate the amount difference between the two tables. A non-zero difference indicates an amount inconsistency and triggers an alarm. Note that check_value in trigger_condition must match the column name returned by the SQL.

Field-Level Rule Example

Example 1: Field null rate check.
Check the null rate of the user_id field. As a required field, it must not contain any null values.
Option 1
field_rules:
  - field_name: user_id
    rule_name: User ID null rate check
    rule_type: system_template
    template_code: null_rate
    trigger_condition: "> 0"
Example 2: Field duplicate value check
Perform a uniqueness check on the order_id field. Duplicate order IDs are not allowed.
Option 1
field_rules:
  - field_name: order_id
    rule_name: Order ID duplicate value check
    rule_type: system_template
    template_code: duplicate_count
    trigger_condition: "> 0"
Example 3: Field average value range check
Check whether the average value of the order_amount field falls within the reasonable range of 10 to 10000.
Option 1
field_rules:
  - field_name: order_amount
    rule_name: Order amount average value check
    rule_type: system_template
    template_code: avg_value
    trigger_condition: not_between [10, 10000]

Example 4: Custom SQL - Field format validation
Option 1

field_rules:
  - field_name: phone_number
    rule_name: Phone number format validation
    rule_type: custom_sql
    dimension: validity
    custom_sql: 
      SELECT COUNT(*) as check_value
      FROM user_info_table WHERE dt = 20260101
      AND phone_number IS NOT NULL
      AND phone_number NOT REGEXP 1[3-9][0-9]{9}
    trigger_condition: check_value > 0

Example 5: Enum value range consistency check (system template - fixed trigger)
Option 1
field_rules:
  - field_name: gender
    rule_name: Gender enum value range consistency check
    rule_type: system_template
    template_code: enum_range_consistency
    params:
     enum_values: M,F,Unknown

FAQ

Q1: How do I choose between system templates and custom SQL?
It is recommended to use system templates first because they are simple, efficient, and do not require specifying quality dimensions. Use custom SQL only when templates cannot meet your requirements.
Scenario
Recommended Type
Standard field statistics check (null rate, duplicate values, mean, and so on)
system_template
Cross-table Data Consistency Check
custom_sql
Complex Business Logic Validation (Format, Range, and Association Checks)
custom_sql
Simple Table Row Quantity or Timeliness Check
system_template
Q2: When is trigger_condition not required?
The trigger conditions for the following two templates are fixed by the system, so you do not need to fill in trigger_condition:
data_timeliness: Configure business filter conditions through filter.
enum_range_consistency: Configure valid enum values through params.enum_values.
Q3: What are the requirements for field names in trigger_condition in custom SQL?
Field names referenced in trigger_condition must exactly match the column names in the result set returned by the custom SQL query.
For example, if SQL returns the check_value column, the condition should be written as check_value > 0, not > 0.
Q4: What is the difference between between and not_between?
between [a, b]: An exception is triggered when the value falls within the range (rarely used).
not_between [a, b]: An exception is triggered when the value falls outside the range (commonly used to indicate that the value is out of the normal range).
In real-world scenarios, not_between is used in most cases to trigger an alarm when the detected value falls outside the normal range.
Q5: What is the purpose of the filter field?
filter is an optional field used to append a WHERE clause during detection, limiting the scope of data involved in the detection.
Common use cases:
Exclude deleted or invalid data.
Limit data to specific partitions or regions.
Use the data_timeliness template to limit the time range.
Q6: How do I obtain a list of available system templates?
Query all template information supported by the current system by calling the DescribeRuleTemplates API, including the template identifier (template_code), applicable level (table-level or field-level), and built-in quality dimensions.
Q7: How many rules can be defined in a configuration file?
There is no limit on the number of rules. You can freely combine table-level rules and field-level rules in the same configuration file, and the system will execute the checks one by one.
Q8: When is the dimension field required?
System template (system_template): Not required, as the quality dimension is built into the system.
Custom SQL (custom_sql): Required. Select from completeness, accuracy, validity, uniqueness, consistency, and timeliness.

前の記事へ: Creating a Quality Task次の記事へ: Data Profiling

ヘルプとサポート

この記事はお役に立ちましたか？

フィードバック

キーワードで検索
    Enter keywords to search documentation