Overview
In addition to managing internally hosted assets (tables, volumes, and models), the DataBuddy data catalog supports bringing external data into the governance system. External data management includes two types of objects:
External Location: Registers a Tencent Cloud COS path as a governance object, serving as the storage entry for creating external volumes or external tables and centrally managing storage authorization across workspaces.
External Catalog (Foreign Catalog): A mirror-type catalog created from a configured data source Connection. It does not store data but only synchronizes metadata from external data sources, allowing users to query external databases as if they were accessing internal catalogs.
This document describes how to create and manage these two types of objects.
Prerequisites
You have activated the DataBuddy service and completed workspace initialization.
Creating an External Location requires the Console Administrator role.
To create an External Catalog, you must first create a Connection in the data source and have the owner permission of the Connection or the corresponding granted permission. The TC Catalog service role (ccatalog_QCSLinkedRoleInMetadataManagement) has been configured in CAM, and COS access permissions have been assigned to it (cos:* action / cos:* resource).
Use Limits
Limit | Description |
Storage Type | External Location currently supports only Tencent Cloud COS. |
Permission to Create External Location | Only console administrators can create; the corresponding entry for other users is grayed out. |
Path Uniqueness | The same COS path cannot be registered repeatedly. |
Data Source Scope of External Catalog | Currently, only TencentDB PostgreSQL is supported; data sources such as MySQL, Hive, and TCHouse-D will be supported gradually in later versions. |
External Catalog Operations | Foreign Catalog is a read-only mirror. Creating / modifying schemas and tables within the Catalog is not supported. DDL and data writes for tables must be executed in the source database. |
Table Details of External Catalog | The "Table Change History" and "Lineage" tabs are not displayed; the Overview, Details, Data Preview, Permissions, and Workspace tabs are retained. |
External Location Management
Creating an External Location
Step 1: Going to the External Storage Page
2. Select Data Catalog in the left sidebar.
3. Click External Data at the top to go to the external data management page.
4. Switch to the External Storage Tab.
Step 2: Creating External Storage
1. Click Create in the upper-right corner of the External Storage page.
2. Fill in the parameters on the creation page:
Service Role: Extracted from the configured TC Catalog service role, currently fixed as ccatalog_QCSLinkedRoleInMetadataManagement.
Storage Type: Currently, only Tencent Cloud COS is supported.
Storage Address: Select the target COS Bucket path from the drop-down list.
Description: Optional, for team collaboration.
1. Click Create to submit.
The platform calls the COS API (ListBucket) to perform the following checks:
Whether the Bucket exists.
Whether the path duplicates a registered External Location.
Whether the service role is not denied access to the path by a cloud policy.
If validation fails, a specific error is displayed (for example, "The platform service role does not have permission to access this path. Contact the cloud administrator to check the CAM policy."). After validation succeeds, a "Created successfully" message appears at the top of the page.
Note:
Default Policy: When an External Location is created, the browse permission is granted to all users and access is granted to all workspaces by default. To limit the scope, adjust the permissions on the Permissions page after creation.
External Location List Page
On the Data Catalog > External Data > External Storage page, you can:
Use the name search box to perform a fuzzy search on names.
View the name, URL, creator, creation time, and description in the list.
Click Create External Storage in the upper-right corner to go to the creation page. Only console administrators can click this button. For other users, the button is grayed out, and hovering over it displays a message stating that only console administrators can create external storage.
External Location Details Page
Click the name of an External Location in the list to go to its details page:
Area | Content. |
Top breadcrumb | External data management > External storage name |
Operation | Rename, Delete |
Basic Information | Description (editable), storage type, URL, creation time, creator |
Permissions Tab | Authorization status of the External Location |
Renaming and Deleting
In the upper-right corner of the details page:
Rename: Modify the name in the pop-up window and save it.
Delete: Delete the registration information after a second confirmation.
Warning
Deleting an External Location only removes its registration with DataBuddy and does not delete the actual files in COS. However, if external volumes or external tables have been created based on this Location, remove the dependencies before deletion. Otherwise, the related assets will become invalid.
External Location Permission Management
Permission Point
Permission Point | Description |
CREATE EXTERNAL TABLE
| Create or modify an external table based on the Location. |
CREATE EXTERNAL VOLUME
| Create or modify an external Volume based on the Location. |
ALL PRIVILEGES
| All permissions listed above |
MANAGE
| Can edit and delete Locations and grant permissions to others. |
Configuring Permissions
1. Go to the Permissions Tab on the External Location details page.
2. Click Add Authorization and select the authorized principal (user or user group).
3. Select the permission points.
4. Click Confirm to complete the authorization.
Only console administrators, Location creators, and users granted the MANAGE permission can initiate authorization.
Creating a Volume Based on an External Location
When you need to reuse a registered COS path as an unstructured asset container, you can create an external Volume based on an External Location. The entire process does not copy data but only establishes a metadata registration relationship in DataBuddy.
Operation Steps
1. Select Data Catalog in the left sidebar to go to the Schema details page of the target Volume-type Catalog.
2. Choose Add Data Object > Volume in the upper-right corner.
3. In the dialog, select External volume as the Volume type.
4. Select a registered target Location from the External Location drop-down list. You can optionally specify a subpath under the Location.
5. Enter the Volume name and description, and click OK to complete the creation.
Permission Requirements
The current user must have the CREATE EXTERNAL VOLUME permission (or ALL PRIVILEGES) on the selected External Location.
The current user must have the create volumes permission on the target Schema.
Behavior Description
Multiple external Volumes can point to different subpaths under the same External Location, and they are isolated from each other.
Deleting an external Volume only removes its metadata registration and does not delete the actual files in the External Location.
If files in the External Location are modified by external processes, the content of the external Volume changes accordingly. DataBuddy does not perform secondary caching.
External Catalog (Creating an External Data Catalog from a Data Source)
External Catalog allows you to query external relational databases (such as TencentDB PostgreSQL) as if they were internal catalogs, without data ingestion or ETL. It is suitable for ad-hoc queries and data exploration scenarios.
Creating an External Catalog
DataBuddy provides two entry points for creating an External Catalog:
Entry A: Creating from an Existing Data Source
This is suitable for scenarios where a data source has already been created but a Catalog has not yet been created.
1. Select Data Catalog in the left sidebar.
2. Click External Data Management at the top to go to the data source list.
3. Click the name of the target data source to go to the details page.
4. Click Create External Data Catalog in the upper-right corner of the details page. A configuration page appears.
Entry B: Guided Creation After a Data Source Is Created
This is suitable for scenarios where a data source is created for the first time.
2. After creation is complete, select Create External Data Catalog in the dialog box.
3. Follow the process for creating an external data catalog.
Differences between creating an external data catalog and creating a data ingestion task:
Option | Scenario |
Create external data catalog. | Ad hoc queries and data exploration; no data copy |
Create data ingestion task. | High-performance queries and data transformation; synchronize external data to DataBuddy internal storage. |
Guidance rules:
MySQL data sources: By default, users are guided to create an external data catalog.
Other data sources: By default, users are guided to perform data ingestion.
Step 2: Configure the Catalog
1. Fill in the basic information on the Create External Catalog configuration page:
Parameter | Description | Required | Default Value |
Catalog name | Manually enter | Yes | - |
Catalog Source | Direct connection to data source by default. | Yes | Direct Data Source Connection |
Data Source | Select data source name | Yes | - |
Description | - | No | - |
Tip
Currently, you can create external data catalogs only based on Doris, Starrocks, GaussDB, PostgreSQL, and MySQL. Other data source types will be supported gradually in later versions.
1. Click Confirm to submit. After successful creation, you will be automatically redirected to the Catalog details page, with a message indicating "Catalog created successfully."
External Catalog Details Page Features
The External Catalog details page is similar in structure to the internal Catalog details page, but with the following differences:
Difference | Internal Catalog | External Catalog |
Connection field | None | Displays the associated Connection name and can be redirected to the Connection details page. |
Create Schema/Table | Supported | Not supported (Foreign Catalog is a mirror only) |
Modify/Delete | Supported | Currently read-only, and operations such as description, deletion, and modification are not supported yet. |
Table details Tab | Overview, data preview, permissions, lineage, table change history | Overview, details, data preview, permissions |
Note:
An External Catalog is a mirror of the source database metadata. To change schemas and table structures, execute DDL statements in the source database. The External Catalog will reflect the metadata changes synchronously.
External Catalog Permission Management
The permission model of an External Catalog is the same as that of a regular Catalog:
By default, the browse permission is granted to all account users.
You can grant Catalog-level permissions to authorization subjects.
The permissions required to create an External Catalog are the same as those required to create an internal Catalog.
Specific permission determination rules:
Operation | Allowed roles |
Create External Catalog | Connection owner / Console administrator / Users or user groups granted the Create Catalog permission and with use connection permission |
Modify Connection Description | Connection owner / Console administrator / Users or user groups granted use connection |
Transfer Connection owner | Connection owner / Console administrator / Users or user groups granted use connection |
Metadata Synchronization Behavior
An External Catalog does not store data. However, it pulls metadata from the source database at the following times to ensure that the Schema / table structures visible in DataBuddy remain consistent with the source database:
Trigger Scenario | Synchronization method | User visibility |
Catalog Created. | Synchronize full metadata once immediately. | After successful creation, users can see schemas and tables in the directory tree. |
User expands Catalog / Schema in the directory tree. | Pull on demand (lazy loading) | Reads from the source database in real time when a node is expanded. |
Go to Table Details. | Pull metadata such as fields and attributes of the table in real time. | Refreshes when the page is opened. |
Re-access After Source Database DDL Change | Automatic synchronization upon next access | Users can see the latest structure by reopening the node / details page. |
Tip
External Catalog does not maintain scheduled background synchronization tasks. When the source database metadata changes, DataBuddy refreshes the corresponding assets on demand the next time a user accesses them. If the synchronization fails (for example, due to an unreachable source database, an invalid Connection, or a network exception), the page displays a synchronization error and prompts the user to check the network connectivity between the Connection and the compute resources.
Cross-Workspace Sharing
Both External Locations and External Catalogs support cross-workspace sharing:
External Location: All workspaces with the required permissions can create external Volumes / external tables on this Location.
External Catalog: By default, it is visible to all workspaces in read-only mode. To restrict access, you can clear the default selection during creation and specify a list of workspaces that can access it.
FAQs
Q: Are External Location and External Catalog the same thing?
A: No.
External Location is a COS storage path registered as a governance object, primarily used as the storage location for creating external Volumes / external tables.
External Catalog is a mirror of external database metadata created based on a Connection, used to query external relational databases in DataBuddy.
They address different problems: External Location focuses on external storage governance, while External Catalog focuses on external database metadata governance.
Q: What is the difference between a Volume created from an External Location and a managed Volume?
A:
For a managed Volume, the platform allocates the storage path and cleans up the underlying files when the Volume is deleted.
An external Volume reuses the COS path on an External Location, and multiple external Volumes can point to different subpaths under the same Location. Deleting an external Volume only removes its metadata registration and does not delete the files.
Q: Can I use INSERT INTO to write data in an External Catalog?
A: This is currently not supported. Foreign Catalog is a read-only mirror. To write data to an external database, perform the operation directly in the source database or use an offline/real-time data synchronization task.
Q: How is the query performance of External Catalog?
A: Queries are pushed down by the compute engine to the external database for execution, and performance depends on the query capability of the source database and the network. For high-concurrency or large-data queries, use data ingestion tasks to synchronize data to the DataBuddy internal Catalog.
Q: What happens to the external Volumes on an External Location when it is deleted?
A: Before deletion, remove all external Volumes / external tables that depend on this Location. Otherwise, these assets will become invalid.
References