Tip
Estimated duration: 90 minutes. Learning objective: Using an e-commerce order scenario as an example, run through the complete data governance pipeline end to end, from creating a Catalog → layered schemas and table creation → governance Tags → data quality monitoring → column-level masking and row-level permissions → data discovery and lineage.
Scenario Introduction
You are a data governance engineer who has just joined the team. The business wants to consolidate e-commerce order data scattered across various business systems into a unified data catalog, and establish classification and grading, sensitive data protection, quality monitoring, and discoverability at the very first line to avoid future remediation.
This tutorial will walk you through:
1. Create a Table Catalog for the e-commerce business domain, and understand the boundaries of the three Catalog types (Table / Volume / Model).
2. Create layered Schemas (ODS / DWD / DWS) under the Catalog, and create tables with SQL to load demo data.
3. Create a set of business Tags and sensitive Tags in the governance tag system, and apply them to tables and fields.
4. Create a data quality task: configure a null rate rule for the uid field of the core table, then run a trial, publish it, and associate an alarm.
5. Configure column-level masking (based on sensitive Tags) and row-level permissions so that the same table returns different views to different roles.
6. Verify that assets are searchable through data discovery, and view upstream and downstream impact through data lineage.
After completing this tutorial, you will master the full set of core operations for the data governance role in daily governance work.
Prerequisites
Before starting this tutorial, make sure the following prerequisites are met:
You have activated the DataBuddy service and completed workspace initialization.
The current account has the Data Governance Administrator role or a higher role in the target workspace, and has the create catalog permission. If you do not have this permission, contact the workspace administrator or metadata lake admin to grant it.
The workspace is bound to a data analysis Serverless resource group, which is used for SQL table creation and quality task execution. For more information, see Compute Resource Overview. At least one user account (for example, analyst_alice and ops_bob) must exist to verify the masking and row-level permission effects.
You are familiar with the basic operations of the SQL editor in Data Engineering or Data Analysis.
Tip
This tutorial assumes that the workspace already has a data lakehouse foundation by default, such as the internally used DataLakeCatalog. The goal of this tutorial is to create a governance Catalog for the e-commerce business domain, decoupled from the default Catalog to facilitate the demonstration of governance processes.
Tutorial Data Model
All SQL statements in this tutorial are based on a single order fact table, orders_raw:
Field | Type | Description | Sensitivity |
order_id
| BIGINT | Order number | Business-critical field |
customer_id
| BIGINT | Customer ID | - |
phone
| STRING | Customer phone number | Sensitive (PII) |
product_name
| STRING | Product name | - |
quantity
| INT | Quantity | - |
unit_price
| DECIMAL(10,2) | Unit Price | - |
region
| STRING | Sales region: North / South / East / West | Row-level Permission Dimension |
order_status
| STRING | Order status: paid / shipped / completed / cancelled | - |
order_time
| TIMESTAMP | Order time | - |
uid
| STRING | Business primary key (associated with the user center) | Quality Rule Monitoring Field |
Tip
To avoid naming conflicts when multiple users run this tutorial at the same time, replace <your_name> with your own name prefix, such as alice_ecommerce or alice_ods_raw.
Step 1: Creating an E-Commerce Business Domain Catalog
Catalog is the top-level namespace in the DataBuddy data governance system. The boundaries between the three Catalog types are hard constraints:
Type | Objects contained |
Catalog | Tables, views, and functions |
Volume | Volumes - unstructured files such as images, PDFs, and CSVs |
Model | Models - ML Model registration and version management |
The type cannot be changed after creation. E-commerce order data is a structured asset, so create a Table Catalog.
1. Log in to the DataBuddy console and switch to the target region and workspace at the top. 2. Select Data Catalog in the left sidebar.
3. Click Create Data Catalog at the top of the left directory tree.
Catalog name: <your_name>_ecommerce (for example, alice_ecommerce).
Catalog Type: Structured Data Catalog.
Description: A structured data governance catalog for the e-commerce order business domain.
1. Click Confirm.
After successful creation, you will be automatically directed to the Catalog details page, where you can see a system-generated Schema: default (General Schema). As the creator, you automatically become the Owner of the Catalog and have full administrative permissions on it.
Note:
If the + Create Data Catalog button is grayed out and a tooltip on hover indicates "You do not have permission", contact the workspace administrator or metadata lake admin to grant you the create catalog permission. For details, see Structured Data Catalog. Tip
If you also need to manage unstructured assets (such as product images) or ML models, create a file volume catalog (for example, <your_name>_ecommerce_files) and a model file catalog (for example, <your_name>_ml_models) by following the same process. For details about the object boundaries of the three Catalog types, see Unstructured Data Catalog and Model Catalog. Step 2: Creating a Layered Schema and Creating Tables to Load Data
Create three Schemas based on the data warehouse layering standards: ODS (Operational Data Store) / DWD (Data Warehouse Detail) / DWS (Data Warehouse Summary).
2.1 Creating a Schema
In the upper-right corner of the Catalog details page, click Create Schema and create the following in sequence:
Schema name | Description |
ods_raw
| Raw data ingested into the data lake from upstream business systems |
dwd_detail
| Fact detail table after cleansing and processing |
dws_summary
| Metric summary table aggregated by dimension |
Each time you enter the name and description, click OK.
2.2 Creating Tables and Loading Data in SQL Analytics
1. In the left navigation pane, choose SQL Analysis > Create SQL and set the file name to governance_setup.sql.
2. In the upper-right corner of the editor, select Data Analysis Serverless Resource Group.
3. Paste the following SQL statements in full, then select them one by one and press ⌘ + Enter (Ctrl + Enter for Windows users) to execute them. Make sure to replace your_name with your own name prefix.
-- 1. ODS Layer Order Raw Table
CREATE TABLE IF NOT EXISTS your_name_ecommerce.ods_raw.orders_raw (
order_id BIGINT COMMENT 'Order ID',
customer_id BIGINT COMMENT 'Customer ID',
phone STRING COMMENT 'Customer phone number (sensitive field)',
product_name STRING COMMENT 'Product name',
quantity INT COMMENT 'Quantity',
unit_price DECIMAL(10,2) COMMENT 'Unit price',
region STRING COMMENT 'Sales region',
order_status STRING COMMENT 'Order status',
order_time TIMESTAMP COMMENT 'Order time',
uid STRING COMMENT 'Business primary key (associated with the user center)'
) COMMENT 'Raw e-commerce order table for data lake ingestion';
-- 2. Load demo data (intentionally leave one row with a NULL uid to trigger the subsequent quality rule).
INSERT INTO your_name_ecommerce.ods_raw.orders_raw VALUES
(1001, 501, '13800138000', 'Wireless Mouse', 2, 79.00, 'North', 'completed', TIMESTAMP '2025-01-15 10:30:00', 'u_001'),
(1002, 502, '13900139000', 'Mechanical Keyboard', 1, 399.00, 'South', 'completed', TIMESTAMP '2025-01-15 11:15:00', 'u_002'),
(1003, 503, '13700137000', 'USB-C Hub', 3, 128.00, 'East', 'paid', TIMESTAMP '2025-01-15 14:20:00', 'u_003'),
(1004, 501, '13800138000', 'Monitor Stand', 1, 259.00, 'North', 'completed', TIMESTAMP '2025-01-16 09:00:00', 'u_001'),
(1005, 504, '13600136000', 'Wireless Mouse', 1, 79.00, 'West', 'shipped', TIMESTAMP '2025-01-16 16:45:00', NULL);
-- 3. DWD layer detail table (for demonstration purposes, with only one simple layer of cleansing)
CREATE TABLE IF NOT EXISTS your_name_ecommerce.dwd_detail.orders_detail AS
SELECT
order_id,
customer_id,
phone,
product_name,
quantity,
unit_price,
quantity * unit_price AS total_amount,
region,
order_status,
order_time,
uid
FROM your_name_ecommerce.ods_raw.orders_raw
WHERE order_status != 'cancelled';
-- 4. DWS layer summary table (aggregated by region)
CREATE TABLE IF NOT EXISTS your_name_ecommerce.dws_summary.daily_revenue AS
SELECT
region,
DATE(order_time) AS order_date,
SUM(quantity * unit_price) AS total_revenue,
COUNT(DISTINCT customer_id) AS unique_customers
FROM your_name_ecommerce.ods_raw.orders_raw
WHERE order_status != 'cancelled'
GROUP BY region, DATE(order_time);
2.3 Verifying Table Registration in the Data Catalog
1. Return to **Data Catalog** and click <your_name>_ecommerce.
2. Switch to the ods_raw Schema, and you should see the orders_raw table. Click the table to go to its details page, where you can see the metadata of 10 fields (name, type, and description) in the Overview Tab.
3. In the same way, confirm that dwd_detail.orders_detail and dws_summary.daily_revenue have been registered.
Tip
After a table is created by SQL, it automatically appears in the Catalog. If you do not see it, refresh the page. If it is still not visible, check whether the three-part name in the SQL is correct and whether the current account has the CREATE TABLE permission on the Schema. For details, see Structured Data Catalog. Step 3: Establishing a Governance Tag System and Tagging Assets
Governed tags serve as the metadata foundation for subsequent asset discovery, sensitive data protection, and compliance auditing. In this step, create two types of tags:
Business tags: used to filter assets by business dimensions in data discovery (such as domain=ecommerce and tier=gold).
Sensitive tags: identify PII fields (such as customer mobile numbers) and associate with masking policies in Step 5 to implement "masking upon tagging". 3.1 Creating Business Tags
1. In the upper-left corner of the Data Catalog page, click Tag Management (the entry point for governed tags is located here).
2. Click Create Tag:
Tag Key: domain.
Tag Type: Business Tag.
Tag Value: add candidate values ecommerce, finance, and logistics.
Description: the business domain to which the data asset belongs.
1. Click Confirm.
Create a second business tag in the same way:
Tag Key: tier, Tag Value: gold / silver / bronze, Description: asset tiering (gold/silver/bronze).
3.2 Reusing Built-in Sensitive Tags
DataBuddy has 36 built-in system-level sensitive tags (such as class.cn_id_card and class.financial_account), which are ready to use out of the box and cannot be renamed or deleted.
In this tutorial, we directly reuse the built-in mobile phone number sensitive tag instead of creating a new one. In the tag governance list, filter Tag Source = System and Tag Type = Sensitive Tag to find the built-in sensitive tag that corresponds to the semantics of "mobile phone number, Chinese mainland mobile phone number" (the tag Key looks like class.cn_mobile_phone, and the actual naming depends on the platform). Note down its Key for use in subsequent steps.
Tip
In this tutorial, the Key of this sensitive tag is uniformly recorded as class.cn_mobile_phone. If the built-in Key in your environment is different, replace the tag references in subsequent steps with the actual name. For details, see Creating and Managing a Tag System. 3.3 Tagging Tables with Business Tags
1. In Data Catalog, go to the details page of the <your_name>_ecommerce.dwd_detail.orders_detail table.
2. In the Business Information area on the right, click Add Tag.
3. In the pop-up window:
Select the tag Key domain and the tag Value ecommerce.
Click Add to continue adding: Tag Key tier, Tag Value gold.
1. Click Confirm.
Return to the Overview Tab. You can see that two tags are displayed in the table header.
3.4 Tagging Fields with Sensitive Tags
1. In the field list of the Overview Tab on the orders_detail table details page, find the phone field.
2. Click Add Tag at the end of the phone field row.
3. In the pop-up window, select the sensitive tag Key class.cn_mobile_phone and click Confirm.
In the same way, you can add the same sensitive tag to the phone field in ods_raw.orders_raw.
Tip
Tagging only attaches a semantic identifier to a field. The actual masking behavior is implemented in Step 5 by associating a sensitive tag with a masking policy. The separation of tagging from policy allows the governance team to complete classification first and then iterate on policies gradually. For details, see Tagging Data Assets. Step 4: Creating a Data Quality Task
The goal of data quality monitoring: configure a "uid validity check" rule on the uid field of the orders_detail table to perform regular checks and trigger an alarm when an exception occurs.
4.1 Going to the Quality Task Creation Page
1. In Data Catalog, go to the details page of the dwd_detail.orders_detail table.
2. Switch to the Quality Tab.
3. Click **Go to Create**.
Entry and process: If you enter from the table details page, the monitoring object is automatically locked to the current table, and no further selection is required. If you enter from the Data Quality entry in the left navigation, you need to manually select <your_name>_ecommerce.dwd_detail.orders_detail in the "Select Object" step.
Note:
If the button is grayed out and a message appears saying "Only the table Owner and users with the table Manager permission can create rules", confirm or request the required permission in the Permissions Tab of the table first. For details, see Create Quality Rule.
4.2 Writing Rules in the Rule Configuration Box
After entering the "Object & Rule" panel, you can directly use **AI to generate rules**, or expand the rule templates on the right to reuse them directly (table-level templates and field-level templates).
field_rules:
- field_name: uid
rule_name: uid validity check
rule_type: custom_sql
dimension: validity
custom_sql: SELECT COUNT(*) as check_value FROM ecommerce.dwd_detail.orders_detail WHERE uid IS NOT NULL
trigger_condition: check_value > 0
alert:
enable: true
Rule description:
field_rule: field-level rule
field_name: applies to the uid field.
rule_type: Template type: Custom SQL.
dimension: Dimension.
trigger_condition: check value > 0: The value is considered abnormal if it is greater than 0.
alert enable: whether to enable alarms for the rule. true: enabled; false: disabled.
Tip
For advanced usage such as template field descriptions, custom SQL syntax, and filter (partition filtering), see Create Quality Rule. You can also click Sample Document above the template list on the right side of the editor to view the complete field descriptions.
4.3 Filling in Basic Configuration
In the Basic Configuration area on the right side of the YAML:
Task Name: orders_detail_uid_quality.
Owner: Defaults to the current account.
Task Description: Monitor the number of null values in the uid field of the order detail table, and trigger an alarm if the number exceeds 1.
4.4 Adding Alarm Configuration
Select the rules and alarm channels for which alarms are required. If none are selected, alarms are not enabled.
4.5 Saving and Publishing
1. Click Save and Publish at the bottom.
2. In the publish dialog:
Scheduling Method: Select Independent Workflow (for tutorial demonstration purposes, a workflow containing only this quality task is automatically generated in Workflow and scheduled on a daily basis).
Scheduling Time: Every day at 08:00.
Alarm Channel: Select a channel configured in your workspace (such as a WeCom group bot or email), and add yourself to Recipients.
1. Click OK to complete the publishing.
After publishing is complete, you can view the task run status in the following two places:
Data Governance > Data Quality > Task List: A unified view of tasks and their historical instances. For details, see Quality Task Ops.
The Quality Tab on the table details page: all quality tasks associated with the current table and the latest check status.
Tip
To add a quality check gate downstream of orders_detail in the ETL workflow, you can add a Quality Monitoring Task Node to the workflow and configure it by using the dynamic parameter capability.
4.6 Verifying Rules in a Test Run
1. In the quality task list, click Test Run.
2. Wait for 10–30 seconds, and then click View Run Result.
3. Expected result (in the tutorial data, 1 of the 5 rows has a NULL uid):
Field | Expected Value |
Execution Status | Successful |
Detection status | Abnormal |
Detected value | 1
|
Threshold | value <= 0
|
Tip
A trial run only evaluates rules against the current data and does not generate alarms or block any downstream tasks. It is the most important verification method before publishing.
Step 5: Configuring Column-Level Masking and Row-Level Permissions
At this point, the phone field has been tagged with a sensitive tag. In this step, associate the tag with a specific masking policy so that queries return different views based on user groups. Also configure row-level filtering for the region field.
5.1 Creating a Column-Level Masking Policy
1. Choose Data Governance > Data Security in the left sidebar.
2. Go to the Masking Policy list and click Create Policy:
Policy Name: policy_mobile_phone_mask.
Associate Sensitive Tag: Select class.cn_mobile_phone. After association, all fields with this tag automatically inherit this policy, so you do not need to bind the policy to each field individually.
Masking Method:
For the user group data_admins, no masking policy is required (administrators can view all data).
For the user group customer_service, use the masking method Keep the first 4 or last 4 characters (customer service sees 1387******** or *******8012).
For the user group analysts, use the hashing masking method (analysts see hashed values).
1. Click Confirm.
Tip
The relationship between masking policies and sensitive tags is many-to-many: a policy can be associated with multiple tags to process multiple semantically similar fields in a unified way, while a field can have only one policy in effect at a time.
5.2 Configuring Row-Level Permissions
Add a row-level filter to the orders_detail table so that Ops personnel in different regions can only see orders from their own regions.
1. Go back to the details page of the dwd_detail.orders_detail table.
2. Switch to the Permissions Tab, locate the Row-Level Permissions section, and click Add Row Filter.
3. In the pop-up window:
Authorized Principal Type: User group.
Authorized Object: ops_north (North China Operations Group).
Row filter expression: region = 'North'.
1. Click Confirm.
2. Repeat the above steps to add rules for Ops personnel in other regions, for example:
ops_south → region = 'South'.
ops_east_west → region IN ('East', 'West').
User groups that do not appear in any row filter are allowed to access all data. For more information, see Row-Level Permission Control.
5.3 Verifying Policy Effects
Switch to a test account that belongs to different user groups (for example, analyst_alice belongs to analysts, and ops_bob belongs to ops_north), and then query in SQL analysis:
SELECT order_id, phone, region, total_amount
FROM your_name_ecommerce.dwd_detail.orders_detail
LIMIT 10;
Expected result:
Test account | phone field return
| Row visibility scope |
Data governance administrator | Plaintext (for example, 13800138000) | All rows |
analyst_alice (belongs to analysts)
| ***
| All rows |
ops_bob (belongs to ops_north)
| If the analysts policy is not matched, plaintext is returned (depending on the actual user group). | Rows where region = 'North' only |
Note:
The returned value of the phone field for ops_bob depends on the user group to which the user belongs in the masking policy. To mask phone numbers for the customer service group as well, go back to 5.1 and explicitly add that user group to the policy.
Tip
Masking and row filtering take effect consistently across all read paths, including SQL queries, Notebooks, and dashboards, eliminating the need for callers to implement them repeatedly in business code.
Step 6: Verifying Governance Results via Data Discovery and Data Lineage
After the above steps are completed, the metadata you create should be searchable and traceable.
6.1 Data Discovery: Finding Assets by Business Tag
1. In the global search box at the top of the DataBuddy console, enter Order Revenue or orders_detail and press Enter to go to the full-screen Data Discovery page. 2. In the asset type filter on the left, switch to Table.
3. In the top filter bar, add the governance tags domain:ecommerce and tier:gold. The result list should then contain only the assets you created this time.
4. Click the result card orders_detail to go to the table details page.
Tip
In Data Discovery, AI semantic search and keyword search run in parallel, and governance tags can be used as filter dimensions for precise narrowing. For more information, see Global Data Search and Filter by Tag and Category. 6.2 Data Lineage: Tracing from DWS Back to ODS
The DWS table daily_revenue is derived from orders_raw through orders_detail, and the lineage provides an intuitive trace of the entire path.
1. On the dws_summary.daily_revenue table details page, switch to the Lineage Tab.
2. By default, the List mode is displayed, and you should be able to see the upstream orders_raw and orders_detail tables, along with the SQL task records that produce this table.
3. Click the icon in the upper-right corner to switch to Graph mode, and you can see the topology diagram:
4. Click any node to view its details. A table node displays business information (description, tags, and owner) and technical information (data source, storage path, and last DDL change time). A task node displays the task code and the associated workflow (which can be redirected to).
Tip
Lineage is automatically reported as tasks run (including trial runs), requiring no manual maintenance. When a task or file is deleted, the lineage is automatically cleaned up. Table-level lineage is currently supported, and field-level lineage will be available in future iterations. For more information, see Data Lineage. Step 7: Cleaning Up Resources (Optional)
If you no longer need the resources created in this tutorial, we recommend that you clean them up in the following order to avoid quota usage:
1. Delete Quality Task: In Data Governance > Data Quality > Task List, click Delete for orders_detail_uid_quality.
2. Delete Row-Level Permissions and Masking Policies: On the orders_detail table details page, delete the row filter in the Permissions Tab. In Data Security > Masking Policies, disassociate policy_mobile_phone_mask.
3. Remove Governance Tags: Remove business tags and sensitive tags from orders_detail / orders_raw / daily_revenue. System built-in tags cannot be deleted and can only be unbound.
4. Delete Table and Schema: Execute the following in SQL analysis:
DROP TABLE IF EXISTS your_name_ecommerce.dws_summary.daily_revenue;
DROP TABLE IF EXISTS your_name_ecommerce.dwd_detail.orders_detail;
DROP TABLE IF EXISTS your_name_ecommerce.ods_raw.orders_raw;
Then delete the three Schemas separately in the data catalog.
1. Delete Catalog: In the upper-right corner of the <your_name>_ecommerce Catalog details page, click ⋯ > Delete Data Catalog.
Warning
Deleting a Schema and a Catalog also deletes all tables and objects under them, and this operation is irreversible. Before deletion, confirm that no business dependencies exist.
Verification Result
After completing all steps, you should be able to confirm:
1. Catalog and Schema: <your_name>_ecommerce (Table type) is visible in the data catalog and contains three Schemas: ods_raw / dwd_detail / dws_summary, plus the system information_schema and default.
2. Asset Metadata: The three tables orders_raw / orders_detail / daily_revenue are all registered in the Catalog, and their field metadata is consistent with the SQL.
3. Governance Tags: Business tags domain and tier have been created, and the sensitive tag class.cn_mobile_phone has been referenced. The orders_detail table has been tagged with business tags, and the phone field has been tagged with the sensitive tag.
4. Data Quality: The orders_detail_uid_quality task is in the published state. The trial run result shows that the detection status is abnormal (null rate 20%, exceeding the 1% threshold). The task has been added to a workflow and is scheduled on a daily basis.
5. Data Security: The masking policy policy_mobile_phone_mask has been created and associated with class.cn_mobile_phone. The orders_detail table has at least one row filter rule.
6. Data Discovery and Lineage: You can find all assets produced in this tutorial by filtering with the domain:ecommerce tag. The lineage graph of daily_revenue can be traced upward to orders_raw.
Summary
In this tutorial, you have completed:
Create an e-commerce business domain Catalog in Data Governance > Data Catalog, and understand the hard constraints of the three Catalog types: Table / Volume / Model.
Create Schemas according to the ODS / DWD / DWS layering specification, and create tables with SQL to load demo data. All assets are automatically registered in the Catalog.
Create a business Tag system in Tag Governance, reuse the built-in sensitive Tags, and apply classification and grading semantic metadata to tables and fields.
In Data Quality, create a null rate monitoring rule via YAML, run a trial to verify it, publish it, and connect it to an alarm channel.
Create a masking policy in Data Security and associate it with sensitive Tags to achieve "masking upon tagging". Configure row-level filtering on the table details page to achieve region-based data isolation.
Find assets by business Tag through data discovery, and trace upstream and downstream impact through data lineage to verify governance outcomes in a closed loop.
This end-to-end pipeline covers more than 80% of the high-frequency operations in the daily work of data governance roles. You can further explore advanced capabilities such as anomaly detection (zero-configuration baselines), data profiling (field distribution and drift), semantic models and metric governance, and automatic tagging with the sensitive data identification AI Skill.