tencent cloud

Tencent Cloud DataBuddy

DocumentaçãoTencent Cloud DataBuddyData IntegrationList of Data SourcesMongoDB Data Source

MongoDB Data Source

Baixar
Modo Foco
Tamanho da Fonte
Última atualização: 2026-10-09 16:09:26
Traduzido por IA

Overview

MongoDB is a high-performance, open-source NoSQL document database known for its flexible document model and strong horizontal scalability. It is widely used in Web applications, mobile applications, and big data analytics scenarios. DataBuddy data ingestion supports using MongoDB as a source for real-time full-database synchronization and offline synchronization. This document describes its connection configuration, supported capabilities, prerequisites, and common issues.

Supported Versions

Type
Supported Version
Self-Built MongoDB
3.6 / 4.0 / 4.2 / 4.4 / 5.0 / 6.0 / 7.0 / 8.0
TencentDB for MongoDB
3.6 / 4.0 / 4.2 / 4.4 / 5.0 / 6.0 / 7.0 / 8.0
Real-time synchronization (Change Streams) requires MongoDB version 3.6 or later.

Read Capability

Capability
Offline read
Real-time full database read
Supported
✓
✓
For a complete comparison of data source capabilities, see Supported Data Sources and Read Capabilities.

Use Limits

Real-Time Sync Limitations

Limit
Description
Version Requirement
MongoDB version must be 3.6 or later.
Replica set
Must be configured in Replica Set mode to use Change Streams.
oplog Retention
Sufficient oplog retention time is required to support incremental sync.
System Collections
Synchronization of system collections such as system.* is not supported.

Offline Sync Limitations

Limit
Description
Single document size
No more than 16 MB (MongoDB limit)
Nesting depth
Control document nesting depth. Excessive nesting may affect performance.

Supported Field Types

For the complete list of BSON types, see the MongoDB official documentation.
Field Type
Offline read
Real-time read
Int32 / Int64 / Long / Double / Decimal128
✅
✅
Boolean / String / ObjectId / UUID / Symbol / Regex
✅
✅
JavaScript / BinData
✅
✅
Date / Timestamp
✅
✅
Object / Array / Null / Undefined / MinKey / MaxKey / DBPointer
✅
✅
MongoDB is a Schema-free database, and the same field may contain multiple types. It is recommended to unify field types at the source to avoid synchronization exceptions.

Supported DML/DDL for Real-Time Sync

DML

Operation
Whether Supported
Description
INSERT / UPDATE / REPLACE / DELETE
✅
Document-level change operation

DDL

Operation
Whether Supported
DROP COLLECTION / DROP DATABASE / RENAME COLLECTION
❌
MongoDB captures changes based on Change Streams and does not support automatic synchronization of DDL changes. Operations such as collection deletion and renaming must be handled manually at the destination.

Prerequisites

1. Confirming Replica Set Configuration (Real-Time Sync Only)

rs.status()

2. Configuring Account Permissions

Offline synchronization:
use admin
db.createUser({
  user: "wedata_user",
  pwd: "your_password",
  roles: [{ role: "read", db: "your_database" }]
})
Real-time synchronization:
use admin
db.createUser({
  user: "wedata_cdc_user",
  pwd: "your_password",
  roles: [
    { role: "read", db: "your_database" },
    { role: "read", db: "local" }   // Required for reading the oplog.
  ]
})

3. Configuring oplog Size (Real-Time Sync Only)

use local
db.oplog.rs.stats()                                  // View the current oplog configuration.
db.adminCommand({ replSetResizeOplog: 1, size: 10240 })   // Unit: MB
To modify the oplog size, you must perform the operation on each node in the replica set.

Creating a Data Source

Operation Steps

1. Log in to the DataBuddy console.
2. Switch to the target region and Workspace at the top.
3. Go to Data Integration > Task Management > Data Source Management (or Platform Management > Data Source Management).
4. Click Add Data Source and select MongoDB.
5. Enter the connection configuration and authentication information. For details, see Parameter Description.
6. Click Test Connection (see Data Source Connectivity Test).
7. After the test passes, click Save or Save & Create Task.

Parameter Description

Parameter
Description
Required
Data Source Name
Unique identifier of the data source within the workspace
Yes
Connection Address
MongoDB connection address (see the format below)
Yes
Database
Default database to connect
Yes
Username.
Database connection account (if authentication is enabled)
Depends on Authentication
Password
Database connection password (if authentication is enabled; SSM credential hosting supported)
Depends on Authentication
Data Source Version
Yes

Connection Address Format

Deployment Mode
Format example
Single-node
mongodb://host:port
Replica set
mongodb://host1:port1,host2:port2,host3:port3/?replicaSet=rsName
Sharded cluster
mongodb://mongos1:port1,mongos2:port2
Complete example:
mongodb://10.0.0.1:27017,10.0.0.2:27017,10.0.0.3:27017/?replicaSet=rs0
mongodb://10.0.0.1:27017/mydb?authSource=admin

Using in Data Ingestion Tasks

Real-Time Full Database Sync

Configuration Item
Description
Source Table Scope
Specified collections (only the selected collections are synchronized; new collections require a restart) / regex matching (specify database name + collection name regex; new collections that match the rule are automatically added)
Read Mode
Full + incremental / Incremental only
Subscription Type
You can select INSERT / UPDATE / DELETE, and at least one must be selected.

Offline Single Table Sync

Parameter
Description
Database / Collection
Required; supports selection or manual input.
Filtering Condition
MongoDB query condition (JSON format)
Filter condition examples:
{"age": {"$gte": 18}, "status": "active"}
{"create_time": {"$gte": {"$date": "2024-01-01T00:00:00Z"}}}

Best Practices

Account Permission Minimization: Use a dedicated account and avoid root privileges.
oplog Retention Period: Set it to at least 72 hours to prevent data loss after task restarts.
Index Optimization: Create indexes on query fields during offline synchronization to improve read efficiency.
Network Connectivity: Use a private network connection whenever possible.

FAQs

Q: Connection Failed?

A: Check network connectivity, confirm that the username and password are correct, and check the authSource configuration.

Q: Change Streams Unavailable?

A: MongoDB is not configured in replica set mode, or its version is earlier than 3.6. Configure MongoDB as a replica set and upgrade it to version 3.6 or later.

Q: oplog Not Found Error During Incremental Sync?

A: The oplog retention period is too short, and the oplog has been overwritten. Increase the oplog size:
db.adminCommand({ replSetResizeOplog: 1, size: 20480 })

Q: Type Mismatch Error During Sync?

A: MongoDB is Schema-free, and the same field may contain different types. It is recommended to unify data types at the source or configure type conversion at the write end.

Q: Some Documents Failed to Sync?

A: A single document exceeds the 16 MB limit. Split large documents, or use GridFS to store large files.

References

Add Data Source Connection
Data Source Connectivity Test
Supported Data Sources and Read Capabilities
Create a Single-Table Sync Task
Real-Time Whole-Database Multi-Table Task Configuration
Anterior: Kafka Data SourcesPróximo: MySQL Data Source

Ajuda e Suporte

Esta página foi útil?

comentários

Search
    Enter keywords to search documentation