Type | Supported Version |
Self-Built Kafka | 2.x / 3.x |
CKafka | 2.4.1 / 2.8.1 / 3.2.3 |
Capability | Offline read | Real-time full database read |
Supported | ✓ | ✓ |
Scenario | Read Format |
Full database synchronization | canal-JSON / debezium / ogg-json / csv /json /raw |
For whole-database synchronization, DDL changes support only adding columns, dropping columns, and creating tables.
Parameter | Description | Required |
Data Source Name | Unique identifier of the data source within the workspace | Yes |
Kafka Type | Self-built open source Kafka / Tencent Cloud CKafka | Yes |
Bootstrap Servers | host1:port1,host2:port2 (for example, 10.0.0.1:9092,10.0.0.2:9092) | Yes |
Authentication Method | No | |
Username / Password | Required when SASL authentication is selected (passwords support SSM credential hosting). | Depends on Authentication |
Method | Description | Required field |
No Authentication (PLAINTEXT) | Authentication Disabled | - |
SASL_PLAINTEXT | Plaintext SASL Authentication | Username and password |
SASL_GSSAPI(Kerberos) | Kerberos Authentication | keytab file, conf file, principal, and Kafka service name |
SASL_SSL | SASL + SSL Dual Authentication | Truststore certificate file, Keystore password, username, and password |
SSL | Mutual TLS Authentication | Truststore certificate file, Truststore password, Keystore certificate file, Keystore password, and private key password |
listeners / advertised.listeners in the server-side server.properties must be set to externally reachable addresses, which is a common root cause of connection failures.Parameter | Description |
Data Source | Created Kafka data source that supports connectivity tests. |
Topic | Topic to consume, supporting multi-select from the dropdown list. |
Serialization Format | |
Read Position | Earliest (earliest) / Latest (latest) / Specified Time |
Subscription Type | You can select INSERT / UPDATE / DELETE, and at least one must be selected. |
source.nested.json.enabled=true
source.key.nested.json.enabled=false. Otherwise, an error will be reported during key parsing.parent_field->child_field format. Use multiple -> to connect multiple levels of nesting. When referencing them in a function, enclose the reference in backticks:TO_TIMESTAMP(`book->published`, 'yyyyMMdd')
properties. prefix, for example, properties.fetch.max.wait.ms=500.Parameter | Description | Default Value |
source.nested.json.enabled | Enable nested JSON (applies to both keys and values). | false |
source.key.nested.json.enabled | Whether to enable nested JSON for keys | Follows source.nested.json.enabled |
source.ignore-parse-errors | Skip error records when parsing fails. | false |
scan.topic-partition-discovery.interval | Interval for periodically discovering new partitions / topics; set to 0 to disable. | 5min |
timestamp.format | Time field parsing format | ISO8601 |
{"data": [{ "id": "1", "name": "Zhang San", "age": "25" }],"database": "test_db","table": "user_info","type": "INSERT","es": 1589373560000,"ts": 1589373560798,"isDdl": false,"mysqlType": { "id": "int", "name": "varchar(50)", "age": "int" },"sqlType": { "id": 4, "name": 12, "age": 4 },"pkNames": ["id"],"old": null,"sql": ""}
{"before": null,"after": { "id": 1, "name": "Zhang San", "age": 25 },"source": {"version": "1.5.0.Final","connector": "mysql","name": "mysql_server","ts_ms": 1589373560000,"db": "test_db","table": "user_info"},"op": "c","ts_ms": 1589373560798}
advertised.listeners is reachable from the network where the resource group resides.Possible cause | How to Confirm |
The read position is set to latest, but the Topic has no new messages. | Change to earliest or a specified time point and restart. |
The Topic actually has no data. | consume once directly from the Kafka console. |
The Consumer group already has a persisted offset externally. | Change to a different group ID and restart. |
Data is completely filtered out by the filter operation or database/table scope. | Check the corresponding configuration of the sync task. |
source.nested.json.enabled=true.source.key.nested.json.enabled=false.canal-json / debezium, messages must carry these metadata fields. Confirm that the upstream Canal / Debezium output includes them. You can use a tool to pull a message and verify.- and > are not valid SQL identifiers. When referencing nested fields in a function, enclose them in backticks and confirm that the fields have been declared through "Add Field":TO_TIMESTAMP(`book->published`, 'yyyyMMdd')
フィードバック