Overview
The DataBuddy data catalog adopts a unified metadata management architecture, centrally registering all consumable asset entities in the data lake-tables, views, volumes, models, and metadata from external data sources-into a single catalog service. Data engineers can discover all data assets through a single entry point and view the upstream and downstream lineage relationships between them.
This document describes the overall architecture of the data catalog, its key components, and how it differs from traditional data source management approaches.
Architecture Overview
The data catalog consists of three layers:
Layer | Role | Description |
MetaLake (metadata lake) | Metadata storage | A region-level metadata service instance. Each tenant has one MetaLake in each region by default, and all catalogs are registered under the MetaLake. |
Catalog | Top-level namespace | Multiple catalogs can be created under a MetaLake. Each Catalog has an independent storage path and type, and is visible across workspaces. |
Schema / Asset | Namespace sublevel and objects | Databases are organized by Schema under a Catalog. Tables, views, volumes, models, and other asset objects are mounted under a Schema. |
By default, storage paths follow the rules below and are stored in Tencent Cloud COS:
cos://<bucket>/root/workspaces/<catalogN>/...
Key Components
MetaLake (metadata lake): a region-level and tenant-level metadata hub. It maintains registration information for all catalogs, metadata for schemas and assets, permission policies, and lineage relationships. A single tenant has only one MetaLake in each region.
Catalog: a metadata namespace that can be created. A Catalog contains:
Type: determines which assets can be created under a Catalog. Currently, three types are supported: Table (including views), Volume, and Model.
Storage path: a COS storage path prefix owned by a Catalog, decoupled from compute resources.
Connection information: A Catalog can be connected to external data sources, for example, by creating a Foreign Catalog based on a Connection.
Schema: a namespace under a Catalog, equivalent to a Database in a traditional database. After each Catalog is created, two schemas are automatically included: information_schema and default.
Asset object: a specific resource mounted under a Schema, including tables, views, volumes, models, functions, and more. The types of objects that each asset can host are determined by the Catalog type.
Built-in Catalog
When a user enters the data catalog for the first time, DataBuddy automatically creates a built-in Catalog:
Built-in Catalog | Purpose | Rename or Delete Supported |
default
| Default directory for storing test and sample data during initial use | Cannot Be Renamed or Deleted |
Design Principles
The DataBuddy data catalog follows these basic design principles:
1. One MetaLake with multiple Catalogs: Multiple Catalogs can be created under a single MetaLake instance, and Catalogs are visible across workspaces.
2. One MetaLake per region: Each tenant has only one MetaLake in each region by default, serving as the single source of truth for metadata. MetaLake instances are fully isolated across regions, and Catalogs with the same name in different regions are treated as different objects.
3. Catalog division by storage: MetaLake divides Catalogs by COS storage paths, and each Catalog has an independent storage prefix.
4. Each Catalog has a specific type: The type must be specified when a Catalog is created, determining the types of assets it can host, including Table, Volume, and Model.
5. Catalog built-in connection information: A Catalog can host connection information for connecting to external resources, such as Connections and External Locations.
6. Catalog decoupled from compute resources: A Catalog only describes metadata and storage, independent of compute resources. The same Catalog can be accessed by any compute resource with the required permissions.
7. Unified storage path rule: Storage paths follow the format cos://<bucket>/root/workspaces/<catalogN>/... and are automatically maintained by the system.
Relationship with Compute Engine
A Catalog only describes metadata and storage, without binding to specific compute resources. When you execute SQL or run Notebooks in Studio, or read and write data in Workflow tasks, compute resources such as DLC SQL, Spark, and Ray directly access the Catalog through the unified metadata interface:
Metadata operations such as table creation, alteration, and deletion are delivered through a unified interface, ensuring that all engines see consistent metadata.
Data reads and writes follow the permission policies on the Catalog. Unauthorized access is blocked at the engine layer.
The same Catalog can be accessed simultaneously by multiple compute resources and multiple workspaces, subject to permission and workspace policy controls.
Relationship Between Workspaces and Catalogs
A Catalog is registered on MetaLake and is visible across workspaces: A Catalog created in Workspace A can also be seen in other workspaces in the same region.
A workspace uses permission policies to determine which Catalogs and assets can be accessed by which users in the current workspace.
Cross-workspace isolation is primarily achieved through access control based on the workspace-user-resource object triad. The accessible scope of the same Catalog in different workspaces can be configured independently.
Use Limits
Limit | Description |
Storage Backend | Currently, only Tencent Cloud COS is supported as the underlying storage. |
Custom COS Bucket | When a Catalog is created, the platform's default storage space is used. User-defined COS buckets are not supported currently. The system allocates storage space based on the default storage space. |
Number of MetaLakes | Each tenant has one fixed MetaLake in each region, and creating multiple MetaLakes is not supported. |
Maximum Number of Catalogs | The maximum number of Catalogs under each MetaLake is 100. You can apply for a quota increase. |
Asset Type Scope | Tables, views, volumes, and models are currently supported. Functions, APIs, and application types are planned for support in later versions. |
FAQs
Q: Why are Catalogs visible across workspaces, while table access is still restricted by workspace?
A: A Catalog is a globally visible namespace on MetaLake, facilitating cross-workspace metadata sharing. However, actual table reads and writes are subject to dual control by permission policies at the Catalog/Schema/asset layers, as well as workspace policies. Metadata visibility does not mean data readability.
Q: Are Catalog and data source the same concept?
A: No. In DataBuddy:
Catalog: A directory that carries metadata and underlying storage, storing data managed internally by DataBuddy, including tables, volumes, and models.
Data source (Connection): Used to connect to external databases such as MySQL and PostgreSQL for data ingestion or mirroring. An external Catalog created based on a Connection is called a Foreign Catalog. For details, see External Data Management. Q: What can be seen in the built-in system Catalog?
A: The system Catalog stores system-level metadata that is automatically maintained by the platform and managed centrally by the platform. Regular users have read-only access to it by default.
References