Release Notes
Product Introduction
Purchase Guide
Getting Started
Buddy
- Buddy - Data Analytics Scenarios
- Data Engineering Agent
- Buddy Overview
- Data Governance Agent
- Buddy Data Security
- Overview of Buddy Agent Basic Capabilities
Platform Management
- Workspace Management
- Compute Resource Management
- Data Source Management Overview
Data Integration
- Supported Data Ingestion Solutions
- Supported Data Sources and Read Capabilities
- Data Source Management
- List of Data Sources
- Offline Data Synchronization
- Real-Time Data Synchronization
- File Upload
Data Engineering
- Studio
- Workflow
- CI/CD
Data Science
- Data Science Overview
- Model Experiments
- Feature Management
- Model Management
- Model Service
Data Analysis
- Data Exploration Overview
Data Governance
- Data Catalog Overview
- Data Discovery Overview
- Data Security Overview
- Data Quality Overview
- Semantic Model Overview
- Governance Tag Overview
Data Applications
- Agent Development
- MCP
- Model List
API Documentation
- History
- Introduction
- API Category
- Making API Requests
- Console APIs
- Workspace APIs
- Computational Resource APIs
- Development Studio APIs
- Workflow APIs
- Catalog APIs
- Data Types
- Error Codes
FAQs
Agreements
Contact Us
Glossary

Data Profiling

ダウンロード

フォーカスモード

フォントサイズ

最終更新日: 2026-10-09 16:16:16

AI翻訳

Overview
Data Profiling automatically generates a complete field-level statistical report and drift analysis dashboard for a table, helping you quickly understand what the table looks like before you use the data. DataBuddy's data profiling supports three types of profiling-snapshot, time-series, and inference-covering statistical metrics for numeric, string, time, and boolean fields, and supports calculating distribution drift based on a baseline table.
Prerequisites
Before you start this operation, make sure the following prerequisites are met:
You have activated the DataBuddy service and completed workspace initialization.
Permission configuration: Only the table Owner and users with the table Manager permission can enable or edit data profiling. For details about role permissions, see Members and Permission Management.
View permission: Users with table read permission can view dashboards and run records.
Available compute resources are ready for running profiling jobs.
If you need to calculate drift metrics, determine the baseline table in advance.
Use Limits
Limit
Description
Entries
Data profiling entry in the Quality Tab on the Catalog table details page
Drift Output Conditions
Drift metrics are generated only when a baseline table is configured; otherwise, drift metrics are empty.
Required Columns for Inference Profiling
Configure at least the timestamp, prediction, and model ID columns. For fairness metrics, additionally specify the tag column, control group, and protected group.
Deleting a dashboard
Currently, deleting a dashboard also deletes intermediate tables. This will be optimized later to delete only the configuration and retain the intermediate tables.
Notification
Alarm notifications for profiling tasks are not supported in this release.
Operation Steps
Step 1: Enable Data Profiling on the Table Details Page
1. Log in to the DataBuddy console.
2. Go to Data Catalog > select the target table > Quality.
3. Turn on the switch in the Data Profiling section. After the switch is enabled, a configuration drawer appears.
Step 2: Select a Profiling Type
The profiling type determines the configuration items to be filled in and the final set of metrics to be produced.
Type
Scenario
Required configuration
Snapshot
Overall distribution and quality summary of static tables
Computing resources, resource mode, resource configuration
TimeSeries
Continuously observe distribution trends by time dimension.
Computing resources, resource mode, resource configuration, timestamp, metric granularity
Inference
Monitor feature drift and model quality in ML model inference logs.
Computing resources, resource mode, resource configuration, timestamp, metric granularity, tag column, prediction column, model ID
Step 3: Configure Profiling Parameters
1. Detection object: The current table is displayed by default and cannot be modified.
2. Compute Resources: Select a resource group from the dropdown list.
3. Scheduling Method: Switch to scheduled scheduling in Advanced Options, or keep the default manual scheduling.
Manual: Executes once upon initial creation, and can be triggered manually afterward by clicking Refresh Dashboard.
Scheduled scheduling: Executes once upon initial creation, and then executes periodically according to the schedule plan. The scheduling logic reuses the workflow.
1. Baseline Table: Select it in Advanced Options. It is used to calculate drift metrics. If it is not configured, drift metrics will be empty.
2. Metric Granularity (required for time series and inference types): Select from hour, day, week, or month.
3. Additional Parameters for Inference Type:
Parameter
Required
Description
Problem type
Yes
Classification / Regression
Timestamp column
Yes
Timestamp field of inference requests
Prediction column (prediction_column)
Yes
Field for prediction values output by the model
Model ID column (model_id_column)
Yes
Model identifier field, supporting multi-version comparison
Label column (label_column)
Yes
After configuration, model quality metrics such as accuracy and precision can be calculated.
1. Fairness and Bias Parameters (classification models only, optional): After selecting the prediction column and label column, additionally specify the comparison column, protected group, and control group.
Step 4: Confirm and Start
Confirm after completing the configuration. The system will start the profiling job based on the current configuration and generate the corresponding Profiling dashboard.
Tip
Upon initial execution, the system generates a complete statistical metric snapshot for the full table data, which serves as the basis for subsequent incremental updates.
Step 5: Manage Dashboards and Profiling Jobs
After data profiling is enabled, the following operations appear in the profiling area of the Quality Tab:
Operation
Description
Viewing a Dashboard
Go to the dashboard module and locate the profiling dashboard for this table.
Refreshing a Dashboard
Re-run the queries on the dashboard to pull the latest data from the intermediate table. This does not recalculate metrics or change the chart structure.
Deleting a dashboard
Remove all charts and profiling data from the current dashboard. A confirmation dialog appears after clicking.
Editing Configuration Information
Open the configuration drawer from the right side. All fields can be re-edited. After saving, the dashboard is updated based on the new configuration.
Run Records
A dialog displays the profiling execution history, including execution time, trigger method, policy, status, and duration. You can click to view logs.
Note:
Deleting a dashboard will overwrite or clear the current profiling results. Confirm that there are no downstream dependencies before proceeding.
Parameter Description
Basic Configuration
Parameter
Description
Required
Default Value
Value Range
Profiling Type
Determines the metric set and required items
Yes
Snapshot
Snapshot / Time Series / Inference
Computing Resource
Computing resources for running profiling jobs
Yes
-
Available Computing Resources in Workspace
Scheduling Method
Controls the trigger method of profiling
Yes
Manual
Manual / Scheduled (in Advanced Options)
Baseline Table
Reference table for drift comparison
No
-
Tables in the Same Workspace
Required Configuration for Time Series and Inference Types
Parameter
Time series
Inference
Description
Timestamp column
Yes
Yes
Used to aggregate data by time buckets
Metric granularity
Yes
Yes
Hour / Day / Week / Month
Tag column
-
Yes
Optional for inference. After configuration, model quality metrics can be calculated.
Prediction column
-
Yes
Required for inference
Model ID column
-
Yes
Required for inference
Metric Reference
Drift Metrics
Drift metrics are calculated based on a comparison between the current profiling results and the user-specified baseline table.
Numeric Fields:
Metric Value
English Name
Description
KS Test
Kolmogorov-Smirnov Test
Test whether two distributions come from the same population.
PSI
Population Stability Index
Measures the degree of distribution shift and is the most commonly used metric in the industry.
Wasserstein Distance
Wasserstein Distance
Earth mover's distance between two distributions
Categorical Fields:
Metric Value
English Name
Description
Chi-Square Test
Chi-Square Test
Test whether categorical distribution changes significantly.
Jensen-Shannon Divergence
Jensen-Shannon Divergence
Measure the symmetric distance between two distributions.
Total Variation Distance
TV Distance
Core metric for measuring the difference between two probability distributions
Chebyshev Distance
L∞ Distance
Measure the maximum difference between two distributions in a single dimension.
Inference Metrics (Inference Type)
After the predict and label columns are configured, the platform additionally produces the following model quality metrics:
Model Type
Metric Value
Description
Category
Accuracy (accuracy_score)
Proportion of correct predictions
Category
Confusion Matrix
Displayed only on the dashboard, not fixed in rule templates.
Category
Precision (precision)
TP / (TP + FP)
Category
Recall (recall)
TP / (TP + FN)
Category
F1 Score (f1_score)
Harmonic mean of precision and recall
Regression
MSE / RMSE / MAE / MAPE
Error metrics, lower values indicate better performance.
Regression
Coefficient of Determination R²
Ranges from 0 to 1, with values closer to 1 indicating a better fit.
Classification (Fairness)
Predictive Parity, Predictive Equality, Equal Opportunity, Statistical Parity
Additionally specify the protected group and control group.
Verification Result
After enabling and initial execution are completed, you can verify in the following ways:
1. You can see View Dashboard in the profiling area of the Quality Tab. After you click to enter the dashboard module, charts can be loaded normally.
2. In the Run Records dialog, you can view the instance records and status of the initial execution.
3. If a baseline table is configured, the drift metrics area in the dashboard displays specific values. If it is not configured, drift metrics are empty.
FAQs
Q: What Are the Differences in Profiling Behavior Between Partitioned and Non-Partitioned Tables?
A: Profiling of partitioned tables scans only new partitions and merges historical statistics. Global metrics, such as median, are recalculated by combining historical values with new partitions. Non-partitioned tables are fully rescanned each time, and previous results are overwritten.
Q: If Drift Detection Is Disabled Midway, Will Historical Drift Data Be Retained?
A: It will be retained. Historical drift data remains unchanged, but subsequent profiling will no longer append drift calculations.
Q: The Dashboard Shows Intermediate Table Data. How Do I Recalculate Metrics?
A: "Refresh Dashboard" only re-pulls data from existing intermediate tables and does not re-execute Profiling. To recalculate metrics, use "Regenerate Dashboard".
References
Create a quality rule.
Anomaly Detection
Create dashboards and layouts.

ヘルプとサポート

この記事はお役に立ちましたか？

営業担当者にお問い合わせいただくかチケットを提出してサポートを求めることができます。

フィードバック

tencent cloud

Tencent Cloud DataBuddy

Data Profiling

Overview

Prerequisites

Use Limits

Operation Steps

Step 1: Enable Data Profiling on the Table Details Page

Step 2: Select a Profiling Type

Step 3: Configure Profiling Parameters

Step 4: Confirm and Start

Step 5: Manage Dashboards and Profiling Jobs

Parameter Description

Basic Configuration

Required Configuration for Time Series and Inference Types

Metric Reference

Drift Metrics

Inference Metrics (Inference Type)

Verification Result

FAQs

Q: What Are the Differences in Profiling Behavior Between Partitioned and Non-Partitioned Tables?

Q: If Drift Detection Is Disabled Midway, Will Historical Drift Data Be Retained?

Q: The Dashboard Shows Intermediate Table Data. How Do I Recalculate Metrics?

References

ヘルプとサポート

Limit	Description
Entries	Data profiling entry in the Quality Tab on the Catalog table details page
Drift Output Conditions	Drift metrics are generated only when a baseline table is configured; otherwise, drift metrics are empty.
Required Columns for Inference Profiling	Configure at least the timestamp, prediction, and model ID columns. For fairness metrics, additionally specify the tag column, control group, and protected group.
Deleting a dashboard	Currently, deleting a dashboard also deletes intermediate tables. This will be optimized later to delete only the configuration and retain the intermediate tables.
Notification	Alarm notifications for profiling tasks are not supported in this release.

Type	Scenario	Required configuration
Snapshot	Overall distribution and quality summary of static tables	Computing resources, resource mode, resource configuration
TimeSeries	Continuously observe distribution trends by time dimension.	Computing resources, resource mode, resource configuration, timestamp, metric granularity
Inference	Monitor feature drift and model quality in ML model inference logs.	Computing resources, resource mode, resource configuration, timestamp, metric granularity, tag column, prediction column, model ID

Parameter	Required	Description
Problem type	Yes	Classification / Regression
Timestamp column	Yes	Timestamp field of inference requests
Prediction column (prediction_column)	Yes	Field for prediction values output by the model
Model ID column (model_id_column)	Yes	Model identifier field, supporting multi-version comparison
Label column (label_column)	Yes	After configuration, model quality metrics such as accuracy and precision can be calculated.

Operation	Description
Viewing a Dashboard	Go to the dashboard module and locate the profiling dashboard for this table.
Refreshing a Dashboard	Re-run the queries on the dashboard to pull the latest data from the intermediate table. This does not recalculate metrics or change the chart structure.
Deleting a dashboard	Remove all charts and profiling data from the current dashboard. A confirmation dialog appears after clicking.
Editing Configuration Information	Open the configuration drawer from the right side. All fields can be re-edited. After saving, the dashboard is updated based on the new configuration.
Run Records	A dialog displays the profiling execution history, including execution time, trigger method, policy, status, and duration. You can click to view logs.

Parameter	Time series	Inference	Description
Timestamp column	Yes	Yes	Used to aggregate data by time buckets
Metric granularity	Yes	Yes	Hour / Day / Week / Month
Tag column	-	Yes	Optional for inference. After configuration, model quality metrics can be calculated.
Prediction column	-	Yes	Required for inference
Model ID column	-	Yes	Required for inference

Metric Value	English Name	Description
KS Test	Kolmogorov-Smirnov Test	Test whether two distributions come from the same population.
PSI	Population Stability Index	Measures the degree of distribution shift and is the most commonly used metric in the industry.
Wasserstein Distance	Wasserstein Distance	Earth mover's distance between two distributions

Model Type	Metric Value	Description
Category	Accuracy (accuracy_score)	Proportion of correct predictions
Category	Confusion Matrix	Displayed only on the dashboard, not fixed in rule templates.
Category	Precision (precision)	TP / (TP + FP)
Category	Recall (recall)	TP / (TP + FN)
Category	F1 Score (f1_score)	Harmonic mean of precision and recall
Regression	MSE / RMSE / MAE / MAPE	Error metrics, lower values indicate better performance.
Regression	Coefficient of Determination R²	Ranges from 0 to 1, with values closer to 1 indicating a better fit.
Classification (Fairness)	Predictive Parity, Predictive Equality, Equal Opportunity, Statistical Parity	Additionally specify the protected group and control group.