Overview
Anomaly detection is a zero-configuration data quality fallback capability provided by DataBuddy. The system automatically learns the update frequency and data volume history patterns of each table in the background, establishes a baseline, and then continuously monitors. When the actual value deviates from the baseline beyond the threshold, an alarm is triggered. Users only need to enable the switch at the Schema level without defining any rules.
Anomaly detection covers the following two types of core risks:
Type | Detection target | Core issue |
Freshness | Whether the table is updated in a timely manner | Is this table supposed to receive data but did not? |
Completeness | Whether the data volume of the table is normal | Is the data volume received by this table this time incorrect? |
Prerequisites
Before you start this operation, make sure the following prerequisites are met:
You have activated the DataBuddy service and completed workspace initialization.
Configuration permission: Only users with the Schema Manager permission can enable or edit anomaly detection. For details about role permissions, see Members and Permission Management. View permission: Users with the Schema read permission can view the detection results.
Compute resources for running detection tasks are ready.
Anomaly detection alarms are not supported in this release.
Use Limits
Limit | Description |
Configuration granularity | Batch enabling is supported only at the Schema level. Table level automatically inherits the setting and does not support separate enabling or disabling. |
Detection dimension | Freshness and completeness are supported in this release. Other dimensions are not supported yet. |
Behavior during learning | No alarm is generated during the period of insufficient samples (freshness < 7 updates, completeness < 14 times), and the status is displayed as "Learning". |
Operation Steps
Step 1: Enabling Anomaly Detection at the Schema Level
2. Go to Data Catalog > select the target Catalog > select the target Schema > Details.
3. Click the enable entry in Data Quality Anomaly Detection to complete the configuration.
Step 2: Viewing Detection Results
1. In the Data Quality Anomaly Detection area on the Schema details page, click View Results.
2. You can filter and view results by Catalog/Schema, and click the refresh button to refresh the list.
3. The list page displays the proportion of healthy tables, along with a list of healthy, unhealthy, and learning tables.
4. The list fields are as follows:
Column Name | Description |
Table Name | Name of the table. |
Schema | Schema of the table |
Status | Table status |
Last detection time | Last detection time of the table |
Detection frequency | Detection frequency of the table |
Operation | View details. Click to go to the table details page. |
Column Name | Description |
Table Name | Table with exceptions |
Schema | Schema of the table |
Status | Table health status |
Exception Type | Completeness exception / Freshness exception |
Upstream Root Cause | Obtain the upstream root cause of the table exception. |
Downstream Impact | Impact scope on downstream after a table exception. |
Last detection time | Last detection time of the table |
Operation | View table details. Click to go to the table details page. |
Column Name | Description |
Table Name | Name of the table. |
Schema | Schema of the table |
Status | Table status |
Last detection time | Last detection time of the table |
Detection frequency | Detection frequency of the table |
Operation | View details. Click to go to the table details page. |
Step 3: Viewing Detection Results on the Table Details Page
1. Go to Data Catalog > select the target table > Quality Tab.
2. You can view the detection results of the table in the Anomaly Detection area.
Note:
This entry only displays the detection results of the current table. You cannot enable or disable anomaly detection for the table separately.
When you click the switch, you will be redirected to the enable entry of the associated Schema. When you click "Disable for Schema", you will be redirected to the disable entry on the Schema details page.
Step 4: Editing or Disabling Anomaly Detection
Operation | Entries | Description |
Editing an alarm policy | Schema Details > Data Quality Monitoring > Edit | After the drawer opens, you can modify the compute resources and alarm notifications. |
Disabled. | Schema Details > Data Quality Monitoring > Disable Switch | A second confirmation dialog appears: "After disabling, all tables will stop intelligent quality checks. Are you sure you want to disable it?". |
Note:
After anomaly detection is disabled, all tables in the Schema stop being detected, and alarm sending also stops.
Recovery Logic for Disabling and Re-enabling
To avoid losing learned samples when repeatedly enabling it, the system follows these rules:
Table status before disabling | Behavior after re-enabling |
Learning (insufficient samples) | Resume learning from the breakpoint: Historical data during the disabled period is retained without clearing. After re-enabling, collection continues, and the status changes to baseline ready after accumulated samples meet the requirement. |
Baseline ready (normal detection in progress) | Resume detection immediately: Continue to use the baseline established before the feature is disabled for judgment. If the feature is disabled for more than 30 days, the system automatically triggers baseline rebuilding, and the status temporarily returns to "Learning" during the rebuilding period. |
Tables added while disabled | Automatically include in detection, start learning from scratch, and the status is "Learning". |
Data processing principles during the disablement period:
During the disablement period, no data is collected, no detection is performed, and no alarms are sent.
The collected historical data and baseline parameters are retained in storage and are not cleared.
After re-enabling, prioritize using existing data to avoid starting from scratch.
Parameter Description
Schema-Level Configuration
Parameter | Description | Required | Default Value | Value Range |
Detection Object | Schema scope | Yes | Current Schema | Not Modifiable |
Detection dimension | Currently supported detection types | Yes | Freshness + Completeness | Not Modifiable |
Collection frequency | Data collection frequency | Yes | Automatically Learn from Historical Data | Not Modifiable |
Alarm Channels | Alarm channels and recipients | Yes | - | Channels enabled in notification management |
FAQs
Q: How soon can alarms be triggered after anomaly detection is enabled for a table?
A: The system immediately starts collecting historical data. If the table metadata already contains sufficient update records (at least 7), a baseline can be established immediately and detection can start. Otherwise, the system waits until enough samples are accumulated, during which the status remains "Learning" and no alarms are sent.
Q: What are the differences between anomaly detection and manual quality rules, and which should I use?
A: The two complement each other. Anomaly detection targets "unknown unknowns" by discovering freshness and integrity deviations with zero configuration. Manual quality rules target "known risks" by precisely safeguarding field-level business rules. We recommend that you first enable anomaly detection as a fallback, and then create manual rules for the high-frequency issues identified to implement refined management. For details about manual rules, see Creating Quality Rules.
Q: Can anomaly detection be disabled for a specific table?
A: This is not supported yet. The detection policy is enabled or disabled uniformly at the Schema level, and the table level inherits it automatically. To exclude a table from detection, disable anomaly detection at the Schema level and then selectively enable it.
References
Create a quality rule