Overview
The Tencent Cloud COS file upload to table feature is used to parse structured or semi-structured files already stored in COS buckets and directly load them into target tables under Catalog, eliminating the need to download files locally before they are uploaded.
The overall interaction is consistent with local upload to table. The main difference lies in the Source step: browse and select files from a COS bucket instead of local browser files. Applicable Scenarios
Data has been accumulated in COS: For example, tracking logs, reconciliation files, and third-party SaaS export files are regularly delivered to COS, and you may want to quickly create tables for one-time analysis.
Cross-account and cross-region data ingestion: Ingest files from other COS buckets (with access permissions) into the lakehouse of the current Workspace.
Avoid local transfer of large files: Avoid downloading multi-GB files before uploading them. Instead, complete parsing and table creation directly in the cloud.
To periodically and continuously pull new date-partitioned files from COS into tables, use File to Table Sync instead. Prerequisites
|
Data Integration Permission | The current account has the operation permission for the Data Integration module in the associated Workspace. |
COS Access Authorization | The current Tencent Cloud account has been granted the COS access permission required for this module (CAM authorization). |
Computing Resource | Created data interactive cluster computing resources for parsing and table creation. |
Operation Steps
Step 1: Entering COS File Upload to Table
In the console, go to the Data Integration module. On the Access Method Tab, click the Tencent Cloud COS File Upload to Table card. The system opens the "Tencent Cloud COS File Upload to Table" drawer on the right.
Step 2: Selecting a COS Bucket
|
Select COS Bucket. | The drop-down list displays the COS buckets that the current account has permission to access. |
The buckets in the dropdown list come from the COS resources of the current Tencent Cloud account. If the list is empty, see COS Authorization Process to complete authorization or create a bucket. Step 3: Selecting a File
After a bucket is selected, the list displays the files and folders in the bucket root directory.
|
Browse | Click a folder name to go to the next level. Empty folders do not display subitems. |
Required or Not | Select the checkboxes before file names. Multiple selection is supported. Select at least one file. |
Limit | Only files can be selected, and folders cannot be selected. File type and quantity constraints are the same as those for uploading local files to a table (up to 10 files, with a total size no greater than 2 GB). |
If the list displays "No bucket address retrieved yet" or a bucket displays "No data available", go to the COS console first to confirm that the bucket and files actually exist.
Step 4: Selecting Compute Resources
Compute resources are required and support single selection only. Only interactive cluster compute resources that the current account has permission to use are displayed. If the selected resource is not started, the system starts it automatically before parsing. For details, see Local File Upload to Table - Step 3: Select Service Resources. Step 5: Data Preview
Click Preview Table to enter data preview. The subsequent behaviors (parsing inference, table creation method, field editing, and advanced settings) are exactly the same as those for local upload to table:
Step 6: Storage Settings and Table Creation
Go to the Storage Settings step, select the target Catalog/Schema and processing method (overwrite with the same name or create a new table), and click Create Table Now to complete table creation. For details, see Local File Upload to Table - Step 5: Storage Settings. COS Authorization Process
DataBuddy accesses COS through the Tencent Cloud CAM system. If the current account has not granted DataBuddy access to COS, an authorization guide is displayed when you go to "Select COS Bucket":
1. The page displays the current Tencent Cloud account information:
Account name: the username of the current Tencent Cloud account.
Account ID: the UIN of the current account.
Required permissions: the COS permission items required by this module.
1. Click the Copy icon to copy the above information.
3. After adding COS-related permission policies to accounts or roles in CAM as needed, return to the DataBuddy page and click Refresh:
After authorization is successful, the "Select COS Bucket" section displays a list of accessible buckets.
If you still have no permissions and the guide page is still displayed, check whether the CAM policy has taken effect.
FAQs
Q: Why does the file list show "No Data" after a COS bucket is selected?
A: The root directory of the bucket may indeed be empty, or the selected bucket may be located in a different region or under a different sub-account. Go to the COS console to check the actual status of the bucket.
Q: Can I select a COS folder to import data into a table in batches?
A: Currently, selecting folders is not supported. Only file selection is supported. To batch import files from an entire directory into a table, use File-to-Table Sync. Q: What should I do if the total size of COS files exceeds 2 GB?
A: As with local uploads, a single upload is still subject to the 2 GB limit. We recommend splitting the upload into multiple parts or using File-to-Table Sync to ingest data through offline sync tasks. Q: Is cross-region COS supported?
A: It depends on the network reachability between the current Workspace and the COS bucket. For specific reachable regions, see the list of supported regions displayed in the DataBuddy console.
References