kafka
|
-Instance
+ | Instance
|
-[Format]
+ | [Format]
DMS:*:*: kafka:instance ID
[Notes]
For instance resources, IAM automatically generates the prefix (DMS:*:*:kafka:) of the resource path.
@@ -78,26 +76,26 @@
DMS for Kafka Request ConditionsRequest conditions are useful in determining when a custom policy is in effect. A request condition consists of condition keys and operators. Condition keys are either global or service-level and are used in the Condition element of a policy statement. Global condition keys (starting with g:) are available for operations of all services, while service-specific condition keys (starting with a service name such as dms:) are available only for operations of specific services. An operator must be used together with a condition key to form a complete condition statement.
DMS for Kafka has a group of predefined condition keys that can be used in IAM. For example, to define an "Allow" permission, use the condition dms:ssl to filter instances by SASL configurations. The following table lists the DMS for Kafka predefined condition keys.
- Table 2 Predefined condition keys of DMS for KafkaCondition Key
+Table 2 Predefined condition keys of DMS for KafkaCondition Key
|
-Operator
+ | Operator
|
-Description
+ | Description
|
|---|
-dms:publicIP
+ | dms:publicIP
|
-Bool
+ | Bool
|
-Whether public access is enabled
+ | Whether public access is enabled
|
-dms:ssl
+ | dms:ssl
|
-Bool
+ | Bool
|
-Whether SSL is enabled
+ | Whether SSL is enabled
|
@@ -107,7 +105,7 @@
diff --git a/docs/dms/umn/Kafka-client-best-practice.html b/docs/dms/umn/Kafka-client-best-practice.html
index 8b886d9f2..ff40f7bbb 100644
--- a/docs/dms/umn/Kafka-client-best-practice.html
+++ b/docs/dms/umn/Kafka-client-best-practice.html
@@ -1,12 +1,14 @@
Suggestions on Using the Kafka Client
-Consumers- Ensure that the owner thread does not exit abnormally. Otherwise, the client may fail to initiate consumption requests and the consumption will be blocked.
- Commit messages only after they have been processed. Otherwise, the messages may fail to be processed and cannot be polled again.
- Generally, do not commit every message. Otherwise, there will be many OFFSET_COMMIT requests, causing high CPU usage. For example, if a consumption request pulls 1000 messages and commits every one of them, TPS of the commit requests is 1000 times that of consumption. The smaller the message size, the larger the ratio. You can commit a specific number of messages in batches or enable enable.auto.commit. However, if the client is faulty, some cached consumption offset may be lost, resulting in repeated consumption. Therefore, you are advised to commit messages in batches based on service requirements.
- A consumer cannot frequently join or leave a group. Otherwise, the consumer will frequently perform rebalancing, which blocks consumption.
- The number of consumers cannot be greater than the number of partitions in the topic. Otherwise, some consumers may fail to poll for messages.
- Ensure that the consumer polls at regular intervals to keep sending heartbeats to the server. If the consumer stops sending heartbeats for long enough, the consumer session will time out and the consumer will be considered to have stopped. This will also block consumption.
- Ensure that there is a limitation on the size of messages buffered locally to avoid an out-of-memory (OOM) situation.
- Set the timeout for the consumer session to 30 seconds: session.timeout.ms=30000.
- Kafka supports exactly-once delivery. Therefore, ensure the idempotency of processing messages for services.
- Always close the consumer before exiting. Otherwise, consumers in the same group may be blocked within the timeout set by session.timeout.ms.
- Do not start a consumer group name with a special character, such as a number sign (#). Otherwise, monitoring data of the consumer group cannot be displayed.
+ Consumers- Ensure that the owner thread does not exit abnormally. Otherwise, the client may fail to initiate consumption requests and the consumption will be blocked.
- Commit messages only after they have been processed. Otherwise, the messages may fail to be processed and cannot be polled again.
- Generally, do not commit every message. Otherwise, there will be many OFFSET_COMMIT requests, causing high CPU usage. For example, if a consumption request pulls 1000 messages and commits every one of them, TPS of the commit requests is 1000 times that of consumption. The smaller the message size, the larger the ratio. You can commit a specific number of messages in batches or enable enable.auto.commit. However, if the client is faulty, some cached consumption offset may be lost, resulting in repeated consumption. Therefore, you are advised to commit messages in batches based on service requirements.
- A consumer cannot frequently join or leave a group. Otherwise, the consumer will frequently perform rebalancing, which blocks consumption.
- The number of consumers in a consumer group must be within the total partitions subscribed by the consumer group. Otherwise, some consumers cannot pull messages.
- Ensure that the consumer polls at regular intervals to keep sending heartbeats to the server. If the consumer stops sending heartbeats for long enough, the consumer session will time out and the consumer will be considered to have stopped. This will also block consumption.
- Ensure that there is a limitation on the size of messages buffered locally to avoid an out-of-memory (OOM) situation.
- Set the timeout for the consumer session to 30 seconds: session.timeout.ms=30000.
- Kafka supports exactly-once delivery. Therefore, ensure the idempotency of processing messages for services.
- Always close the consumer before exiting. Otherwise, consumers in the same group may be blocked within the timeout set by session.timeout.ms.
- Do not start a consumer group name with a special character, such as a number sign (#). Otherwise, monitoring data of the consumer group cannot be displayed.
- Handle AuthorizationException in the consumption logic and set consumption retry. For example, in SpringKafka, add the following configuration.
# Set the authorization exception retry interval to 10 seconds.
+spring.kafka.listener.authorizationExceptionRetryInterval=10000
+
- Producers- Synchronous replication: Set acks to all.
- Retry message sending: Set retries to 3.
- Optimize message sending: For latency-sensitive messages, set linger.ms to 0. For latency-insensitive messages, set linger.ms to a value ranging from 100 to 1000.
- Ensure that the producer has sufficient JVM memory to avoid blockages.
- Set the timestamp to the local time. Messages will fail to age if the timestamp is a future time.
+ Producers- Synchronous replication: Set acks to all.
- Retry message sending: Set retries to 3.
- Optimize message sending: For latency-sensitive messages, set linger.ms to 0. For latency-insensitive messages, set linger.ms to a value ranging from 100 to 1000.
- Ensure that the producer has sufficient JVM memory to avoid blockages.
- Set the timestamp to the local time. Messages will fail to age if the timestamp is a future time.
- Try reusing producers. Do not create producers frequently. When idempotence is enabled (default for producer clients 3.0 and later), producing messages creates producer state objects on the server. Frequent creation results in too many objects to be reclaimed in time, causing server memory surges and performance deterioration. Set enable.idempotence to false if the idempotence is not required.
- Catch exception AuthorizationException. You are advised to retry message production on the service side. Self-healing can be implemented through limited retry and backoff policies.
TopicsRecommended topic configurations: Use 3 replicas, enable synchronous replication, and set the minimum number of in-sync replicas to 2. The number of in-sync replicas cannot be the same as the number of replicas of the topic. Otherwise, if one replica is unavailable, messages cannot be produced.
- You can enable or disable automatic topic creation. If automatic topic creation is enabled, the system automatically creates a topic when a message is produced in or consumed from a topic that does not exist. This topic has the following default settings: 3 partitions, 3 replicas, aging time 72 hours, synchronous replication and flushing disabled, CreateTime message timestamp, and maximum 10,485,760 bytes message size.
+ You can enable or disable automatic topic creation. Enabling this function automatically creates a topic when a message is produced in or consumed from a topic that does not exist.
OthersMaximum number of connections: 3000
Maximum size of a message: 10 MB
diff --git a/docs/dms/umn/Kafka-client-parameter.html b/docs/dms/umn/Kafka-client-parameter.html
index 4a5818312..d00d9ef06 100644
--- a/docs/dms/umn/Kafka-client-parameter.html
+++ b/docs/dms/umn/Kafka-client-parameter.html
@@ -7,9 +7,9 @@
Default Value
|
- Recommended Value
+ | Recommended Value
|
- Description
+ | Description
|
@@ -17,10 +17,10 @@
1
|
- all or –1 (if high reliability mode is selected)
+ | all or –1 (if high reliability mode is selected)
1 (if high throughput mode is selected)
|
- Indicates the number of acknowledgments the producer requires the server to return before considering a request complete. This controls the durability of records that are sent. The value of this parameter can be any of the following:
+ | Number of acknowledgments the producer requires the server to return before considering a request complete. This controls the durability of records that are sent. The value of this parameter can be any of the following:
0: The producer will not wait for any acknowledgment from the server at all. The record will be immediately added to the socket buffer and considered sent. No guarantee can be made that the server has received the record, and the retries configuration will not take effect (as the client generally does not know of any failures). The offset given back for each record will always be set to –1.
1: The leader will write the record to its local log but will respond without waiting until receiving full acknowledgement from all followers. If the leader fails immediately after acknowledging the record but before the followers have replicated it, the record will be lost.
all or -1: The leader needs to wait until all backups in the ISR are written into logs. As long as any backup survives, data will not be lost. min.insync.replicas specifies the minimum number of replicas that must acknowledge a write for the write to be considered successful.
@@ -30,20 +30,20 @@
|
0
|
- /
+ | /
|
- Number of times that the client resends a message. Setting this parameter to a value greater than zero will cause the client to resend any record that failed to be sent.
+ | Number of times that the client resends a message. Setting this parameter to a value greater than zero will cause the client to resend any record that failed to be sent.
Note that this retry is no different than if the client re-sent the record upon receiving the error. Allowing retries will potentially change the ordering of records because if two batches are sent to the same partition, and the first fails and is retried but the second succeeds, then the records in the second batch may appear first.
You are advised to configure producers so that they can be able to retry in case of network disconnections. Set retries to 3 and the retry interval retry.backoff.ms to 1000.
|
request.timeout.ms
|
-30000
+ | 30,000
|
-/
+ | /
|
-Maximum amount of time (in ms) the client will wait for the response of a request. If the response is not received before the timeout elapses, the client will throw a timeout exception.
+ | Maximum amount of time (in ms) the client will wait for the response of a request. If the response is not received before the timeout elapses, the client will throw a timeout exception.
Setting this parameter to a large value, for example, 127000 (127s), can prevent records from failing to be sent in high-concurrency scenarios.
|
@@ -51,33 +51,42 @@
TRUE
|
- TRUE
+ | TRUE
|
- Setting this parameter to TRUE indicates that when buffer memory is exhausted, the producer must stop receiving new message records or throw an exception.
+ | Setting this parameter to TRUE indicates that when buffer memory is exhausted, the producer must stop receiving new message records or throw an exception.
By default, this parameter is set to TRUE. However, in some cases, non-blocking usage is desired and it is better to throw an exception immediately. Setting this parameter to FALSE will cause the producer to instead throw "BufferExhaustedException" when buffer memory is exhausted.
|
batch.size
|
-16384
+ | 16,384
|
-262144
+ | 262,144
|
-Default maximum number of bytes of messages that can be processed at a time. The producer will attempt to batch records together into fewer requests whenever multiple records are being sent to the same partition. This helps improve performance of both the client and the server. No attempt will be made to batch records larger than this size.
+ | Default maximum number of bytes of messages that can be processed at a time. The producer will attempt to batch records together into fewer requests whenever multiple records are being sent to the same partition. This helps improve performance of both the client and the server. No attempt will be made to batch records larger than this size.
Requests sent to brokers will contain multiple batches, one for each partition with data available to be sent.
A smaller batch size will make batching less common and may reduce throughput (a batch size of zero will disable batching entirely). A larger batch size may use more memory as a buffer of the specified batch size will always be allocated in anticipation of additional records.
|
buffer.memory
|
-33554432
+ | 33,554,432
|
-67108864
+ | 67,108,864
|
-Total bytes of memory the producer can use to buffer records waiting to be sent to the server. If records are sent faster than they can be delivered to the broker, the producer will stop sending records or throw a "block.on.buffer.full" exception.
+ | Total bytes of memory the producer can use to buffer records waiting to be sent to the server. If records are sent faster than they can be delivered to the broker, the producer will stop sending records or throw a "block.on.buffer.full" exception.
This setting should correspond roughly to the total memory the producer will use, but is not a rigid bound since not all memory the producer uses is used for buffering. Some additional memory will be used for compression (if compression is enabled) as well as for maintaining in-flight requests.
|
+ enable.idempotence
+ |
+- Earlier than v3.0: false
- v3.0 and later: true
+ |
+If idempotence is not required, you are advised to set this parameter to false.
+ |
+If you have enabled idempotence on the producer client, and produced messages, message offsets are not continuous on the consumer client or on the Message Query page on the Kafka console. This is because enabling idempotence generates some metadata control messages during message production. These control messages are produced to topics, and are invisible to consumers.
+ |
+
|
Default Value
|
-Recommended Value
+ | Recommended Value
|
-Description
+ | Description
|
|---|
@@ -96,9 +105,9 @@
TRUE
|
-FALSE
+ | FALSE
|
-If this parameter is set to TRUE, the offset of messages already fetched by the consumer will be periodically committed to ZooKeeper. This committed offset will be used when the process fails as the position from which the new consumer will begin.
+ | If this parameter is set to TRUE, the offset of messages already fetched by the consumer will be periodically committed to ZooKeeper. This committed offset will be used when the process fails as the position from which the new consumer will begin.
Constraints: If this parameter is set to FALSE, to avoid message loss, an offset must be committed to ZooKeeper after the messages are successfully consumed.
|
@@ -106,21 +115,65 @@
latest
|
-earliest
+ | earliest
|
-Indicates what to do when there is no initial offset in ZooKeeper or if the current offset has been deleted. Options:
-- earliest: Automatically reset to the smallest offset.
- latest: The offset is automatically reset to the largest offset.
- none: The system throws an exception to the consumer if no offset is available.
- anything else: The system throws an exception to the consumer.
- NOTE: If this parameter is set to latest, the producer may start to send messages to new partitions (if any) before the consumer resets to the initial offset. As a result, some messages will be lost.
-
+ | Indicates what to do when there is no initial offset in ZooKeeper or if the current offset has been deleted. Options:
+- earliest: Automatically reset to the smallest offset.
- latest: Automatically reset to the largest offset.
- none: The system throws an exception to the consumer if no offset is available.
- anything else: The system throws an exception to the consumer.
+If this parameter is set to latest, the producer may start to send messages to new partitions (if any) before the consumer resets to the initial offset. As a result, some messages will be lost.
|
connections.max.idle.ms
|
-600000
+ | 600,000
|
-30000
+ | 30,000
|
-Timeout interval (in ms) for an idle connection. The server closes the idle connection after this period of time ends. Setting this parameter to 30000 can reduce the server response failures when the network condition is poor.
+ | Timeout interval (in ms) for an idle connection. The server closes the idle connection after this period of time ends. Setting this parameter to 30000 can reduce the server response failures when the network condition is poor.
+ |
+
+max.poll.records
+ |
+500
+ |
+Must be less than the value of max.poll.interval.ms.
+ |
+The maximum number of messages that a consumer can pull from a broker at a time.
+ |
+
+max.poll.interval.ms
+ |
+300,000
+ |
+Increase this parameter if complex and time-consuming logic exists between two polls.
+ |
+The maximum interval between consumer polls, in milliseconds. If this parameter is exceeded, the consumption fails and the consumer is removed from the consumer group, triggering rebalance.
+ |
+
+heartbeat.interval.ms
+ |
+3,000
+ |
+≥ 3000
+ |
+Heartbeat interval between a consumer and Kafka, in milliseconds.
+ |
+
+session.timeout.ms
+ |
+10,000
+ |
+Set this parameter to at least 3 times the value of heartbeat.interval.ms.
+ |
+The consumer-broker session timeout when the offset is managed by consumer group, in milliseconds.
+ |
+
+fetch.max.bytes
+ |
+1,000,000
+ |
+max.request.size < message.max.bytes < fetch.max.bytes
+ |
+The maximum bytes of a message that a consumer can pull from a broker at a time.
|
diff --git a/docs/dms/umn/Kafka-specification.html b/docs/dms/umn/Kafka-specification.html
index ee729afbd..632a3bbfe 100644
--- a/docs/dms/umn/Kafka-specification.html
+++ b/docs/dms/umn/Kafka-specification.html
@@ -1,189 +1,129 @@
-Specifications
-Kafka Instance SpecificationsKafka instances are compatible with open-source Kafka v1.1.0, v2.3.0, v2.7, and v3.x. Kafka instances are classified into cluster and single-node types. A cluster instance consists of three or more brokers and a single-node one has one broker.
- Kafka instances are classified based on instance ECS flavors as follows:
- - Cluster
- kafka.2u4g.cluster.small
- kafka.2u4g.cluster
- kafka.4u8g.cluster
- kafka.8u16g.cluster
- kafka.12u24g.cluster
- kafka.16u32g.cluster
- - Single-node
- kafka.2u4g.single.small
- kafka.2u4g.single
-
-  For Kafka instances, the number of transactions per second (TPS) is the maximum number of messages that can be written per second. In the following table, transactions per second (TPS) are calculated assuming that the size of a message is 1 KB. The test scenario is private access in plaintext. The disk type is ultra-high I/O.
- Cluster Kafka instances support v2.3.0, v2.7, and v3.x. Single-node Kafka instances support v2.7.
+ Cluster Kafka Instances
+ Instance SpecificationsA cluster Kafka instance has three or more brokers, and is compatible with open-source Kafka 2.3.0, 2.7, and 3.x.
+ In the following table:
+ - For Kafka instances, the number of transactions per second (TPS) is the maximum number of messages that can be written per second.
- TPS performance: number of messages processed per second assuming that the size of a message is 1 KB.
- Test scenario: private access in plaintext. The disk type is ultra-high I/O.
- Table 1 Kafka instance specifications (cluster)Flavor
+Table 1 Cluster Kafka instance specificationsFlavor
|
-Brokers
+ | Brokers
|
-Maximum TPS per Broker
+ | Maximum TPS per Broker
|
-Maximum Partitions per Broker
+ | Maximum Partitions per Broker
|
-Recommended Consumer Groups per Broker
+ | Recommended Consumer Groups per Broker
|
-Maximum Client Connections per Broker
+ | Maximum Client Connections per Broker
|
-Storage Space
+ | Storage Space (GB)
|
-Traffic per Broker (MB/s)
+ | Traffic per Broker (MB/s)
|
|---|
-kafka.2u4g.cluster.small
+ | kafka.2u4g.cluster.small
|
-3–30
+ | 3–30
|
-20,000
+ | 20,000
|
-100
+ | 100
|
-15
+ | 15
|
-2000
+ | 2,000
|
-300 GB–300,000 GB
+ | 300–300,000
|
-40
+ | 40
|
-kafka.2u4g.cluster
+ | kafka.2u4g.cluster
|
-3–30
+ | 3–30
|
-30,000
+ | 30,000
|
-250
+ | 250
|
-20
+ | 20
|
-2000
+ | 2,000
|
-300 GB–300,000 GB
+ | 300–300,000
|
-100
+ | 100
|
-kafka.4u8g.cluster
+ | kafka.4u8g.cluster
|
-3–30
+ | 3–30
|
-100,000
+ | 100,000
|
-500
+ | 500
|
-100
+ | 100
|
-4000
+ | 4,000
|
-300 GB–600,000 GB
+ | 300–600,000
|
-200
+ | 200
|
-kafka.8u16g.cluster
+ | kafka.8u16g.cluster
|
-3–50
+ | 3–50
|
-150,000
+ | 150,000
|
-1000
+ | 1000
|
-150
+ | 150
|
-4000
+ | 4,000
|
-300 GB–1,500,000 GB
+ | 300–1,500,000
|
-375
+ | 375
|
-kafka.12u24g.cluster
+ | kafka.12u24g.cluster
|
-3–50
+ | 3–50
|
-200,000
+ | 200,000
|
-1500
+ | 1,500
|
-200
+ | 200
|
-4000
+ | 4,000
|
-300 GB–1,500,000 GB
+ | 300–1,500,000
|
-625
+ | 625
|
-kafka.16u32g.cluster
+ | kafka.16u32g.cluster
|
-3–50
+ | 3–50
|
-250,000
+ | 250,000
|
-2000
+ | 2,000
|
-200
+ | 200
|
-4000
+ | 4,000
|
-300 GB–1,500,000 GB
+ | 300–1,500,000
|
-750
- |
-
-
-
- Table 2 Kafka instance specifications (single-node)Flavor
- |
-Brokers
- |
-TPS per Broker
- |
-Maximum Partitions per Broker
- |
-Recommended Consumer Groups per Broker
- |
-Maximum Client Connections per Broker
- |
-Storage Space
- |
-Traffic per Broker (MB/s)
- |
-
|---|
-
-kafka.2u4g.single.small
- |
-1
- |
-20,000
- |
-100
- |
-15
- |
-2000
- |
-100 GB–10,000 GB
- |
-40
- |
-
-kafka.2u4g.single
- |
-1
- |
-30,000
- |
-250
- |
-20
- |
-2000
- |
-100 GB–10,000 GB
- |
-100
+ | 750
|
@@ -199,9 +139,9 @@
Assume that the current flavor is kafka.2u4g.cluster, the traffic per broker is 100 MB/s, and the number of brokers is 3. What are the total network traffic, maximum read traffic, and maximum write traffic of the instance?
- Total network traffic = Traffic per broker x Broker quantity = 100 MB/s x 3 = 300 MB/s
- Maximum read traffic = Total instance network traffic/Default number of replicas/2 = 300 MB/s/3/2= 50 MB/s
- Maximum write traffic = Total instance network traffic/Default number of replicas/2 = 300 MB/s/3/2 = 50 MB/s
-Mapping Between Old and New FlavorsTable 3 compares the old and new Kafka instance flavors.
+ Mapping Between Old and New FlavorsTable 2 compares the old and new Kafka instance flavors.
- Table 3 Mapping between old and new Kafka instance flavorsOld Flavor
+Table 2 Mapping between old and new Kafka instance flavorsOld Flavor
|
New Flavor
|
@@ -243,20 +183,20 @@
600 MB/s
|
|---|
-1200 MB/s
+ | 1,200 MB/s
|
-1200 MB/s
+ | 1,200 MB/s
|
kafka.4u8g.cluster * 6
|
-1250 MB/s
+ | 1,250 MB/s
|
Instances with new flavors have the following features:
-- Better performance and cost effectiveness: They use exclusive resources (except for kafka.2u4g.cluster.small). By contrast, old flavors use non-exclusive resources. If the load is heavy, resources conflicts will occur.
- Latest functions, for example, reassigning partitions and changing the SSL setting.
- Flexible flavor changes: For example, you can increase the broker flavor.
- Flexible disk capacity: Only related to the broker quantity, and not to the flavor.
- More specification options: A wider range of combinations of broker flavor (over 10,000 MB/s) and quantity are available.
+- Better performance and cost effectiveness: They use exclusive resources (except for kafka.2u4g.cluster.small). By contrast, old flavors use non-exclusive resources. If the load is heavy, resources conflicts will occur.
- Latest functions, for example, reassigning partitions and changing the SSL setting.
- Flexible flavor changes: For example, you can increase the broker flavor.
- Flexible disk capacity: Only related to the broker quantity, and not to the flavor.
- More specification options: A wider range of combinations of broker flavor (over 10,000 MB/s) and quantity are available.
- More disk type options: General Purpose SSD and Extreme SSD are now available, in addition to the original disk types.
Flavor Selection- kafka.2u4g.cluster.small with 3 brokers
Recommended for up to 6000 client connections, 45 consumer groups, and 60,000 TPS
- kafka.2u4g.cluster with 3 brokers
Recommended for up to 6000 client connections, 60 consumer groups, and 90,000 TPS
@@ -271,15 +211,14 @@
The storage space can be expanded as your service grows.
Topic QuantityThere are limits on the topic quantity and the aggregate number of partitions in the topics. When the partition quantity limit is reached, you can no longer create topics.
- The number of topics is related to the maximum number of partitions allowed (see Table 1 and Table 2) and the specified number of partitions in each topic (see Figure 1).
- Figure 1 Setting the number of partitions
+ The number of topics is related to the maximum number of partitions allowed (see Table 1) and the specified number of partitions in each topic.
The maximum number of partitions allowed for an instance with kafka.2u4g.cluster and 3 brokers is 750.
- If the number of partitions of each topic in the instance is 3, the maximum number of topics is 750/3 = 250.
- If the number of partitions of each topic in the instance is 1, the maximum number of topics is 750/1 = 750.
Permission
+Permissions
If you need to grant your enterprise personnel permission to access your DMS resources, use Identity and Access Management (IAM). IAM provides identity authentication, fine-grained permissions management, and access control. IAM helps you secure access to your cloud resources.
You can create IAM users for your employees, and assign permissions to these users on a principle of least privilege (PoLP) basis to control their access to specific resource types. For example, you can create IAM users for software developers and assign specific permissions to allow them to use Kafka instance resources but prevent them from being able to delete resources or perform any high-risk operations.
If your account does not require individual IAM users for permissions management, skip this section.
diff --git a/docs/dms/umn/TagManagement.html b/docs/dms/umn/TagManagement.html
index 79d70be80..b83d0ce18 100644
--- a/docs/dms/umn/TagManagement.html
+++ b/docs/dms/umn/TagManagement.html
@@ -2,7 +2,7 @@
Configuring Kafka Instance Tags
Tags facilitate Kafka instance identification and management.
- You can add tags to a Kafka instance when creating the instance or add tags on the Tags tab page of the created instance. Up to 20 tags can be added to an instance. Tags can be deleted.
+ You can add tags to a Kafka instance when creating the instance or add tags on the Tags page of the created instance. Tags can be deleted.
A tag consists of a tag key and a tag value. Table 1 lists the tag key and value requirements.
Table 1 Tag key and value requirementsParameter
@@ -13,22 +13,26 @@
|
|---|
Tag key
|
-- Cannot be left blank.
- Must be unique for the same instance.
- Can contain 1 to 128 characters.
- Can contain letters, digits, spaces, and special characters _ . : = + - @
- Cannot start or end with a space.
- Cannot start with _sys_.
+ | - Cannot be left blank.
- Must be unique for the same instance.
- Can contain 1 to 128 characters.
- Can contain letters, digits, spaces, and special characters _.:=+-@
- Cannot start or end with a space.
- Cannot start with _sys_.
|
Tag value
|
-- Can contain 0 to 255 characters.
- Can contain letters, digits, spaces, and special characters _ . : = + - @
- Cannot start or end with a space in instance creation.
+ | - Can contain 0 to 255 characters.
- Can contain letters, digits, spaces, and special characters _.:=+-@
- Cannot start or end with a space in instance creation.
|
- Procedure- Log in to the console.
- Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the name of an instance.
- In the navigation pane on the left, choose Tags.
View the tags of the instance.
- - Perform the following operations as required:
- Add a tag
- Click Create/Delete Tag.
- Enter a tag key and a tag value, and click Add.
If you have predefined tags, select a predefined pair of tag key and value, and click Add.
- - Click OK.
+Notes and ConstraintsUp to 20 tags can be added to a Kafka instance.
+ Procedure- Log in to the console.
- Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired instance to go to the instance details page.
- In the navigation pane, choose Tags.
- Add or delete tags as required.
Permission Management
+Permissions Management
diff --git a/docs/dms/umn/en-us_image_0000001093972624.png b/docs/dms/umn/en-us_image_0000001093972624.png
index 9f3f6ab1d..bfedf06b3 100644
Binary files a/docs/dms/umn/en-us_image_0000001093972624.png and b/docs/dms/umn/en-us_image_0000001093972624.png differ
diff --git a/docs/dms/umn/en-us_image_0000001160616010.png b/docs/dms/umn/en-us_image_0000001160616010.png
deleted file mode 100644
index 478a5525d..000000000
Binary files a/docs/dms/umn/en-us_image_0000001160616010.png and /dev/null differ
diff --git a/docs/dms/umn/en-us_image_0000001206335999.png b/docs/dms/umn/en-us_image_0000001206335999.png
deleted file mode 100644
index 98259cb05..000000000
Binary files a/docs/dms/umn/en-us_image_0000001206335999.png and /dev/null differ
diff --git a/docs/dms/umn/en-us_image_0000001427644729.png b/docs/dms/umn/en-us_image_0000001427644729.png
deleted file mode 100644
index 25d9bb2ab..000000000
Binary files a/docs/dms/umn/en-us_image_0000001427644729.png and /dev/null differ
diff --git a/docs/dms/umn/en-us_image_0000001453201733.png b/docs/dms/umn/en-us_image_0000001453201733.png
index 9f2eb4a1f..1e95e5814 100644
Binary files a/docs/dms/umn/en-us_image_0000001453201733.png and b/docs/dms/umn/en-us_image_0000001453201733.png differ
diff --git a/docs/dms/umn/en-us_image_0000001540501562.png b/docs/dms/umn/en-us_image_0000001540501562.png
index 5a75764f9..9d01bad71 100644
Binary files a/docs/dms/umn/en-us_image_0000001540501562.png and b/docs/dms/umn/en-us_image_0000001540501562.png differ
diff --git a/docs/dms/umn/en-us_image_0000001707049736.png b/docs/dms/umn/en-us_image_0000001707049736.png
deleted file mode 100644
index 5a75764f9..000000000
Binary files a/docs/dms/umn/en-us_image_0000001707049736.png and /dev/null differ
diff --git a/docs/dms/umn/en-us_image_0000001921463342.png b/docs/dms/umn/en-us_image_0000001921463342.png
index 7408cf8ef..61c9a8e66 100644
Binary files a/docs/dms/umn/en-us_image_0000001921463342.png and b/docs/dms/umn/en-us_image_0000001921463342.png differ
diff --git a/docs/dms/umn/en-us_image_0000001998520149.png b/docs/dms/umn/en-us_image_0000001998520149.png
index cab936f3f..140b0d990 100644
Binary files a/docs/dms/umn/en-us_image_0000001998520149.png and b/docs/dms/umn/en-us_image_0000001998520149.png differ
diff --git a/docs/dms/umn/en-us_image_0000002063422361.png b/docs/dms/umn/en-us_image_0000002063422361.png
deleted file mode 100644
index eccb3209b..000000000
Binary files a/docs/dms/umn/en-us_image_0000002063422361.png and /dev/null differ
diff --git a/docs/dms/umn/en-us_image_0000002110644228.png b/docs/dms/umn/en-us_image_0000002110644228.png
new file mode 100644
index 000000000..79bcaf267
Binary files /dev/null and b/docs/dms/umn/en-us_image_0000002110644228.png differ
diff --git a/docs/dms/umn/en-us_image_0000002157871349.png b/docs/dms/umn/en-us_image_0000002157871349.png
new file mode 100644
index 000000000..ffd75cf7e
Binary files /dev/null and b/docs/dms/umn/en-us_image_0000002157871349.png differ
diff --git a/docs/dms/umn/en-us_image_0000001756508438.png b/docs/dms/umn/en-us_image_0000002281749220.png
similarity index 100%
rename from docs/dms/umn/en-us_image_0000001756508438.png
rename to docs/dms/umn/en-us_image_0000002281749220.png
diff --git a/docs/dms/umn/en-us_image_0000001756853218.png b/docs/dms/umn/en-us_image_0000002309000337.png
similarity index 100%
rename from docs/dms/umn/en-us_image_0000001756853218.png
rename to docs/dms/umn/en-us_image_0000002309000337.png
diff --git a/docs/dms/umn/en-us_image_0000002317148937.png b/docs/dms/umn/en-us_image_0000002317148937.png
new file mode 100644
index 000000000..47a039305
Binary files /dev/null and b/docs/dms/umn/en-us_image_0000002317148937.png differ
diff --git a/docs/dms/umn/en-us_image_0000002317222029.png b/docs/dms/umn/en-us_image_0000002317222029.png
new file mode 100644
index 000000000..47a039305
Binary files /dev/null and b/docs/dms/umn/en-us_image_0000002317222029.png differ
diff --git a/docs/dms/umn/kafka-bp-migration.html b/docs/dms/umn/kafka-bp-migration.html
index 339368220..f0e536200 100644
--- a/docs/dms/umn/kafka-bp-migration.html
+++ b/docs/dms/umn/kafka-bp-migration.html
@@ -27,7 +27,7 @@
9094
|
-0.0.0.0/0
+ | IP address or IP address group of the Kafka client
|
Accessing a Kafka instance in a public network (in plaintext)
|
@@ -38,7 +38,7 @@
9095
|
-0.0.0.0/0
+ | IP address or IP address group of the Kafka client
|
Accessing a Kafka instance in a public network (in ciphertext)
|
@@ -56,8 +56,7 @@
Migration Scheme 2: Migrating the Production LaterUse multiple consumers for the consumption service. Some consume messages from the original Kafka instance, and others consume messages from the new Kafka instances. Then, migrate the production service to the new Kafka instance so that all messages can be consumed in time.
For a certain period of time, the consumption service consumes messages from both the original and new Kafka instances. Before the migration, message consumption from the new Kafka instance has already started, so there is no latency. However, early on in the migration, data is consumed from both the original and new Kafka instances, so the messages may not be consumed in the order that they are produced. This scheme is suitable for services that require low latency but do not require strict message sequence.
- - Start new consumer clients, set the Kafka connection addresses to that of the new Kafka instance, and consume data from the new Kafka instance.
Original consumer clients must continue running. Messages are consumed from both the original and new Kafka instances.
-
+- Start new consumer clients, set the Kafka connection addresses to that of the new Kafka instance, and consume data from the new Kafka instance.
Original consumer clients must continue running. Messages are consumed from both the original and new Kafka instances.
- Change the Kafka connection address of the producer to that of the new Kafka instance.
- Restart the producer client to migrate the production service to the new Kafka instance.
- After the production service is migrated, check whether the consumption service connected to the new Kafka instance is normal.
- After all data in the original Kafka is consumed, close the original consumption clients.
- The migration is complete.
Migration Scheme 3: Migrating the Consumption FirstUse Smart Connect to synchronize the two Kafka instances, migrate the consumer first and then the producer to the new Kafka instance.
diff --git a/docs/dms/umn/kafka-dnat.html b/docs/dms/umn/kafka-dnat.html
index ad6d9e303..2338b537b 100644
--- a/docs/dms/umn/kafka-dnat.html
+++ b/docs/dms/umn/kafka-dnat.html
@@ -6,32 +6,105 @@
This section describes how to access Kafka over a public network using DNAT.
PrerequisitesYou have created EIPs. The number of EIPs is the same as the number of brokers in the Kafka instance. For details about how to create an EIP, see Assigning an EIP.
- Step 1: Obtain Information About the Kafka Instance- Log in to the console.
- Click
 in the upper left corner to select a region. Select the region where your Kafka instance is located.
-
- - Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired Kafka instance to view its details.
- In the Connection area on the Basic Information tab page, view and record the private network access addresses of the Kafka instance. In the Network area, view and record the VPC and subnet where the Kafka instance is located.
Figure 1 Kafka instance information
+Step 1: Obtain Information About the Kafka Instance- Log in to the console.
- Click in the upper left corner to select the region where your instance is located.
- Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired instance to go to the instance details page.
- In the Connection area on the Basic Information page, view and record the private network access addresses of the Kafka instance. In the Network area, view and record the VPC and subnet where the Kafka instance is located.
Figure 1 Kafka instance information
Step 2: Create a Public NAT Gateway- Click Service List and choose Network > NAT Gateway.
- Click Create Public NAT Gateway.
- Set the following parameters:
- Region: Select the region that the Kafka instance is in.
- Name: Enter a name for the public NAT gateway.
- VPC: Select the VPC recorded in 5.
- Subnet: Select the subnet recorded in 5.
-Set other parameters as required. For details, see Creating a NAT Gateway.
-Figure 2 Create Public NAT Gateway
+Step 2: Create a Public NAT Gateway- Click Service List and choose Network > NAT Gateway.
- Click Create Public NAT Gateway.
- Set parameters by referring to Table 1 and other parameters as required. For details, see Creating a Public NAT Gateway.
Figure 2 Create Public NAT Gateway
+
+Table 1 Public NAT gateway creation parametersParameter
+ |
+Description
+ |
+
|---|
+
+Region
+ |
+Region where the public NAT gateway is located. Select the region that the Kafka instance is in.
+ |
+
+Name
+ |
+Enter a name for the public NAT gateway. Enter up to 64 characters. Only letters, digits, underscores (_), hyphens (-), and periods (.) are allowed.
+ |
+
+VPC
+ |
+VPC where the public NAT gateway resides. Select the VPC recorded in 5.
+ |
+
+Subnet
+ |
+Subnet in the VPC where the public NAT gateway resides. Select the subnet recorded in 5.
+ |
+
+
+
+ - Click Create Now.
- Confirm the specifications and click Submit.
Step 3: Add a DNAT Rule- On Public NAT Gateways page, locate the row containing the newly created public NAT gateway and click Add Rule in the Operation column.
- On the DNAT Rules tab page, click Add DNAT Rule.
Figure 3 Public NAT gateway details
- - Set the following parameters:
- Scenario: Select VPC.
- Port Type: Select Specific port.
- Protocol: Select TCP.
- EIP: Select an EIP.
- Outside Port: Enter 9011.
- Instance Type: Select Custom.
- Private IP Address: Enter one of the private network addresses of the Kafka instance recorded in 5.
- Inside Port: Enter 9011.
-For details about more parameters, see Adding a DNAT Rule.
-Figure 4 Adding a DNAT rule
+ - Set parameters by referring to Table 2. For details about more parameters, see Adding a DNAT Rule.
Figure 4 Adding a DNAT rule
+
+Table 2 Adding a DNAT ruleParameter
+ |
+Description
+ |
+
|---|
+
+Scenario
+ |
+Select VPC. The servers in a VPC will share an EIP to provide services accessible from the Internet through the DNAT rule.
+ |
+
+Port Type
+ |
+Select Specific port. The public NAT gateway forwards requests to your servers only from the outside port and to the inside port configured here, and only if they use the right protocol.
+ |
+
+Protocol
+ |
+Select TCP.
+ |
+
+EIP
+ |
+Select the created EIP.
+ |
+
+Outside Port
+ |
+Enter 9011.
+ |
+
+Instance Type
+ |
+Instance type for providing services over external public networks. Select Custom.
+ |
+
+Private IP Address
+ |
+Enter one of the private network addresses of the Kafka instance recorded in 5.
+ |
+
+Inside Port
+ |
+Enter 9011.
+ |
+
+
+
+ - Click OK.
View the DNAT rule status in the DNAT rule list. If Status is Running, the rule has been added successfully.
- Create DNAT rules for other private network addresses of the Kafka instance recorded in 5. Configure a unique EIP for each DNAT rule.
For details about how to create a DNAT rule, see 2 to 4.
- After all DNAT rules are created, click the DNAT Rules tab to view the created DNAT rules and record the EIPs corresponding to the private IP addresses.
Step 4: Map EIPs to the Port 9011 of Private IP Addresses- Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired Kafka instance to view its details.
- In the Advanced Settings section on the Basic Information tab page, click Modify next to Cross-VPC Access.
- Change the values of advertised.listeners IP Address/Domain Name to the EIPs in the DNAT rules. Ensure that the mapping between the private network addresses and the EIPs is consistent with that recorded in 6. Then click Save.
Figure 5 Changing the advertised.listeners IP address (for DNAT access)
+Step 4: Map EIPs to the Port 9011 of Private IP Addresses- Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired Kafka instance to view its details.
- In the Advanced Settings area on the Basic Information page, click Modify.
- Change the values of advertised.listeners IP Address/Domain Name to the EIPs in the DNAT rules. Ensure that the mapping between the private network addresses and the EIPs is consistent with that recorded in 6. Then click Save.
Figure 5 Changing the advertised.listeners IP address (for DNAT access)
Step 5: Verify ConnectivityCheck whether messages can be created and retrieved by referring to Connecting to Kafka Using the Client (Plaintext Access) or Connecting to Kafka Using the Client (Ciphertext Access).
Notes:
- The address for connecting to a Kafka instance is in the format of "advertised.listeners IP:9011". For example, the addresses for connecting to the Kafka instance shown in Figure 5 are 100.xxx.xxx.20:9011,100.xxx.xxx.21:9011,100.xxx.xxx.23:9011.
- Configure security group rules for the Kafka instance to allow inbound access over port 9011.
- Public access must be enabled on the client connected to the Kafka instance.
diff --git a/docs/dms/umn/kafka-faq-0001.html b/docs/dms/umn/kafka-faq-0001.html
index 3f9338d29..011f1b9bd 100644
--- a/docs/dms/umn/kafka-faq-0001.html
+++ b/docs/dms/umn/kafka-faq-0001.html
@@ -1,9 +1,8 @@
How Do I Obtain the Public Access Address After Public Access Is Enabled?
- Click the name of your Kafka instance. In the Connection section on the Basic Information tab page, view Instance Address (Public Network).
+ Click the name of your Kafka instance. In the Connection section on the Basic Information page, view Instance Address (Public Network).
For details about how to connect to a Kafka instance, see Connecting to an Instance.
-
diff --git a/docs/dms/umn/kafka-faq-0005.html b/docs/dms/umn/kafka-faq-0005.html
index 4b7e605bb..9d915bd8f 100644
--- a/docs/dms/umn/kafka-faq-0005.html
+++ b/docs/dms/umn/kafka-faq-0005.html
@@ -1,8 +1,7 @@
How Do I Increase the Partition Quantity?
- You can increase the partition quantity by adding brokers.
- To do so, go to the Kafka console, locate the row that contains the desired instance, and choose More > Modify Specifications. On the page that is displayed, add brokers as required. For details, see Modifying Kafka Instance Specifications.
+ You can increase the partition quantity by increasing broker flavors or adding brokers. For details, see Modifying Cluster Kafka Instance Specifications.
diff --git a/docs/dms/umn/kafka-faq-0033.html b/docs/dms/umn/kafka-faq-0033.html
index 182eae736..8b8cd1bb3 100644
--- a/docs/dms/umn/kafka-faq-0033.html
+++ b/docs/dms/umn/kafka-faq-0033.html
@@ -11,7 +11,7 @@
|
|---|
100 MB/s
|
-3000
+ | 3,000
|
300 MB/s
@@ -24,7 +24,7 @@
| 20,000
|
-1200 MB/s
+ | 1,200 MB/s
|
20,000
|
@@ -41,42 +41,42 @@
kafka.2u4g.cluster.small
|
-2000
+ | 2,000
|
kafka.2u4g.single.small
|
-2000
+ | 2,000
|
kafka.2u4g.cluster
|
-2000
+ | 2,000
|
kafka.2u4g.single
|
-2000
+ | 2,000
|
kafka.4u8g.cluster
|
-4000
+ | 4,000
|
kafka.8u16g.cluster
|
-4000
+ | 4,000
|
kafka.12u24g.cluster
|
-4000
+ | 4,000
|
kafka.16u32g.cluster
|
-4000
+ | 4,000
|
diff --git a/docs/dms/umn/kafka-faq-0035.html b/docs/dms/umn/kafka-faq-0035.html
index 37c7bed98..e078520b1 100644
--- a/docs/dms/umn/kafka-faq-0035.html
+++ b/docs/dms/umn/kafka-faq-0035.html
@@ -1,7 +1,7 @@
Does Specification Modification Affect Services?
-Table 1 describes the impact of increasing specifications. It takes 5 to 10 minutes to modify specifications on one broker. The more brokers, the longer time the modification takes.
+ Table 1 describes the impact of increasing specifications.
Table 1 Impact of specification modificationModified Object
|
@@ -11,7 +11,7 @@
|---|
Bandwidth or broker quantity
|
-- Increasing the bandwidth or adding brokers does not affect the original brokers or services.
- When you increase the bandwidth or change the broker quantity, the storage space is proportionally expanded based on the current disk space. For example, assume that the original number of brokers of an instance is 3 and the disk size of each broker is 200 GB. If the broker quantity changes to 10 and the disk size of each broker is still 200 GB, the total disk size becomes 2000 GB.
- New topics are created on new brokers, and the original topics are still on the original brokers, resulting in unbalanced partitions. You can reassign partitions to migrate the replicas of the original topic partitions to the new brokers.
+ | - Increasing the bandwidth or adding brokers does not affect the original brokers or services.
- When you increase the bandwidth or change the broker quantity, the storage space is proportionally expanded based on the current disk space. For example, assume that the original number of brokers of an instance is 3 and the disk size of each broker is 200 GB. If the broker quantity changes to 10 and the disk size of each broker is still 200 GB, the total disk size becomes 2,000 GB.
- New topics are created on new brokers, and the original topics are still on the original brokers, resulting in unbalanced partitions. You can reassign partitions to migrate the replicas of the original topic partitions to the new brokers.
|
Storage space
@@ -24,7 +24,7 @@
| - Single-replica topics do not support message production during this period. Services will be interrupted.
- If a topic has multiple replicas, modifying the broker flavor does not interrupt services, but may cause disorder of partition messages. Evaluate this impact and avoid peak hours.
- Broker rolling restarts will cause partition leader changes, interrupting connections for less than a minute when the network is stable. For multi-replica topics, configure the retry mechanism on the producer client. To do so:
-
+It takes 5 to 10 minutes to modify specifications on one broker. The more brokers, the longer time the modification takes.
|
diff --git a/docs/dms/umn/kafka-faq-0038.html b/docs/dms/umn/kafka-faq-0038.html
index aba39f09b..b39a7d9a9 100644
--- a/docs/dms/umn/kafka-faq-0038.html
+++ b/docs/dms/umn/kafka-faq-0038.html
@@ -8,7 +8,6 @@
Figure 1 Consumer access permissions
Solution:
- If all consumers must be in the same consumer group (group.id is the same), grant the same topic access permissions to all the consumers.
- If the consumers do not need to be in the same consumer group, change the value of group.id to ensure that each consumer is in a separate consumer group.
-
diff --git a/docs/dms/umn/kafka-faq-0041.html b/docs/dms/umn/kafka-faq-0041.html
index 18cb176aa..39f17761d 100644
--- a/docs/dms/umn/kafka-faq-0041.html
+++ b/docs/dms/umn/kafka-faq-0041.html
@@ -2,9 +2,8 @@
How Do I View the Number of Accumulated Messages?
View the number of accumulated messages using any of the following methods:
- - On the Consumer Groups page of an instance, click the name of the consumer group whose accumulated messages are to be viewed. The consumer group details page is displayed. On the Consumer Offset tab page, view the number of messages accumulated in each topic of your target consumer group. For details, see Viewing Consumer Offsets.
- On the Monitoring tab page of an instance, click the By Consumer Group tab. Select the desired consumer group for Consumer Group and All topics for Topic. The Consumer Available Messages metric reflects the number of messages accumulated in all topics of this consumer group. For details about viewing the monitoring data, see Viewing Kafka Metrics.
- On the Consumer Groups tab page of the Cloud Eye console, click the By Consumer Group tab. Select the desired consumer group for Consumer Group and All topics for Topic. The Consumer Available Messages metric reflects the number of messages accumulated in all topics of this consumer group. For details about viewing the monitoring data, see Viewing Kafka Metrics.
- In the /bin directory on the Kafka client, run command kafka-consumer-groups.sh --bootstrap-server ${connection-address} --describe --group ${consumer-group-name} to check number of accumulated messages of each topic in a consumer group. LAG indicates the total number of messages accumulated in each topic.
Figure 1 Viewing the total number of messages accumulated in each topic
- If SASL authentication is enabled for the Kafka instance, the --command-config {SASL authentication configuration file consumer.properties} parameter must be added to the preceding commands. For details about the configuration file consumer.properties, see the CLI access instructions provided in Connecting to Kafka Using the Client (Ciphertext Access).
-
+- On the Consumer Groups page of an instance, click the name of the consumer group whose accumulated messages are to be viewed. The consumer group details page is displayed. On the Consumer Offset tab page, view the number of messages accumulated in each topic of your target consumer group. For details, see Viewing Consumer Offsets.
- On the Monitoring page of an instance, click the By Consumer Group tab. Select the desired consumer group for Consumer Group and All topics for Topic. The Consumer Available Messages metric reflects the number of messages accumulated in all topics of this consumer group. For details about viewing the monitoring data, see Viewing Kafka Metrics.
- On the Consumer Groups tab page of the Cloud Eye console, click the By Consumer Group tab. Select the desired consumer group for Consumer Group and All topics for Topic. The Consumer Available Messages metric reflects the number of messages accumulated in all topics of this consumer group. For details about viewing the monitoring data, see Viewing Kafka Metrics.
- In the /bin directory on the Kafka client, run command kafka-consumer-groups.sh --bootstrap-server ${connection-address} --describe --group ${consumer-group-name} to check number of accumulated messages of each topic in a consumer group. LAG indicates the total number of messages accumulated in each topic.
Figure 1 Viewing the total number of messages accumulated in each topic
+If SASL authentication is enabled for the Kafka instance, the --command-config {SASL authentication configuration file consumer.properties} parameter must be added to the preceding commands. For details about the configuration file consumer.properties, see the CLI access instructions provided in Connecting to Kafka Using the Client (Ciphertext Access).
diff --git a/docs/dms/umn/kafka-faq-0063.html b/docs/dms/umn/kafka-faq-0063.html
new file mode 100644
index 000000000..d882f2512
--- /dev/null
+++ b/docs/dms/umn/kafka-faq-0063.html
@@ -0,0 +1,11 @@
+
+
+ Why Do Metrics Fluctuate Significantly (Disk Read/Write Speed, Average Disk Read/Write Time, and CPU Usage)?
+ Metrics including Disk Read Speed, Disk Write Speed, Average Disk Read Time, Average Disk Write Time, and CPU Usage output instantaneous values. These metrics are for reference only. Generally, flushing Kafka data asynchronously to disk consumes disk I/O and CPU usage, causing fluctuations. They do not affect services.
+
+
+
diff --git a/docs/dms/umn/kafka-faq-0064.html b/docs/dms/umn/kafka-faq-0064.html
new file mode 100644
index 000000000..ee2d1e760
--- /dev/null
+++ b/docs/dms/umn/kafka-faq-0064.html
@@ -0,0 +1,11 @@
+
+
+ Why Does JVM Heap Memory Usage of JVM Fluctuate Significantly?
+ It is normal that JVM Heap Memory Usage of JVM is displayed in a sawtooth shape on the monitoring page. The maximum value may further increase or the fluctuation may be greater after scale-out or an engine upgrade. Such fluctuations are caused by normal JVM heap memory reclamation, and do not affect services.
+
+
+
diff --git a/docs/dms/umn/kafka-faq-0065.html b/docs/dms/umn/kafka-faq-0065.html
new file mode 100644
index 000000000..ebbcd65c9
--- /dev/null
+++ b/docs/dms/umn/kafka-faq-0065.html
@@ -0,0 +1,12 @@
+
+
+ Why Are Offsets Not Continuous?
+ If you have enabled idempotence or transactions on the producer client, and produced messages, message offsets are not continuous on the consumer client or on the Message Query page on the Kafka console. This is because enabling idempotence or transactions generates some metadata control messages during message production. These control messages are produced to topics, and are invisible to consumers.
+ By default, idempotence is enabled for Kafka 3.0 and later producer clients. To disable it, set the parameter enable.idempotence to false.
+
+
+
diff --git a/docs/dms/umn/kafka-faq-0066.html b/docs/dms/umn/kafka-faq-0066.html
new file mode 100644
index 000000000..1025557e3
--- /dev/null
+++ b/docs/dms/umn/kafka-faq-0066.html
@@ -0,0 +1,11 @@
+
+
+ Why Is Production Rate Still 0 When There Are Produced Messages?
+ The metric data of the message production rate is reported once every minute. Message production rate = Number of messages produced per minute/60. The value is rounded down. The production rate is displayed as 0 if the number of messages produced within a minute is less than 60.
+
+
+
diff --git a/docs/dms/umn/kafka-faq-0067.html b/docs/dms/umn/kafka-faq-0067.html
new file mode 100644
index 000000000..4e96978f7
--- /dev/null
+++ b/docs/dms/umn/kafka-faq-0067.html
@@ -0,0 +1,15 @@
+
+
+ Why Is the Number of Stacked Messages Monitored as 0 when Messages Are Stacked?
+ SymptomOn the Consumer Offset tab page of the consumer group details page, there are stacked messages in a topic (total stack ≠0), whereas the Consumer Available Messages on the consumer group monitoring page is 0.
+
+ ConclusionThe total stack on the Consumer Offset tab page and the Consumer Available Messages are not synchronously updated.
+ The total stack is sampled immediately. The Consumer Available Messages is sampled every minute. You see the stack number at your time. When a consumer consumes all the stacked messages within 1 minute, Consumer Available Messages displays 0.
+
+
+
+
diff --git a/docs/dms/umn/kafka-faq-0068.html b/docs/dms/umn/kafka-faq-0068.html
new file mode 100644
index 000000000..8aa07c2ae
--- /dev/null
+++ b/docs/dms/umn/kafka-faq-0068.html
@@ -0,0 +1,11 @@
+
+
+ Can I Purchase Partitions Separately?
+ No. However, you can increase the total number of partitions by increasing the broker quantity or flavor. For details, see Modifying Cluster Kafka Instance Specifications.
+
+
+
diff --git a/docs/dms/umn/kafka-faq-0069.html b/docs/dms/umn/kafka-faq-0069.html
new file mode 100644
index 000000000..4fb9b1a0a
--- /dev/null
+++ b/docs/dms/umn/kafka-faq-0069.html
@@ -0,0 +1,12 @@
+
+
+ Why Are Consumer Groups Missing?
+ If you cannot find a consumer group from the console, the possible causes are as follows:
+ - auto.create.groups.enable is set to true, the consumer group status is EMPTY, and no offset has been submitted. The system automatically deletes the consumer group 10 minutes after it is created.
- If the consumer group has never committed an offset, it will be deleted after the Kafka instance restarts.
+
+
+
diff --git a/docs/dms/umn/kafka-faq-0070.html b/docs/dms/umn/kafka-faq-0070.html
new file mode 100644
index 000000000..01aced255
--- /dev/null
+++ b/docs/dms/umn/kafka-faq-0070.html
@@ -0,0 +1,17 @@
+
+
+ Why Are the Earliest Messages Not Automatically Deleted When the Disk Capacity Reaches 95%?
+ SymptomEach broker of a Kafka instance has a 100 GB disk. The automatic deletion policy has been configured and a topic with 180 partitions has been created. The earliest messages are not automatically deleted when the disk usage reaches 95%.
+
+ Possible CauseEach broker of a Kafka instance uses a 33 GB disk to store logs and ZooKeeper data. Such a disk was formatted during instance creation and certain space is used. As a result, the data storage space is approximately 66 GB. Several segment files constitute each partition of a topic. Each segment file has maximum storage of 500 MB. To delete a message, Kafka deletes segment files instead of a message. Kafka reserves at least one segment file in each partition for message storage. In this way, at least one segment file is kept. Increasing partitions to 132 (66 GB/500 MB = 132) may cause the disk usage to reach the upper limit.
+ When the disk usage reaches 95% (62.7 GB), the messages might be stored in one of those kept segment files. In this case, the messages are not deleted.
+
+
+
+
+
diff --git a/docs/dms/umn/kafka-faq-180604024.html b/docs/dms/umn/kafka-faq-180604024.html
index cd4cc558c..aec8df3b6 100644
--- a/docs/dms/umn/kafka-faq-180604024.html
+++ b/docs/dms/umn/kafka-faq-180604024.html
@@ -25,7 +25,7 @@
9092
|
- 0.0.0.0/0
+ | IP address or IP address group of the Kafka client
|
Accessing a Kafka instance over a private network within a VPC (in plaintext)
|
@@ -38,7 +38,7 @@
9192
|
- ::/0
+ | IP address or IP address group of the Kafka client
|
Accessing a Kafka instance within a VPC (with SSL encryption disabled)
|
@@ -51,7 +51,7 @@
9093
|
- 0.0.0.0/0
+ | IP address or IP address group of the Kafka client
|
Accessing a Kafka instance over a private network within a VPC (in ciphertext)
|
@@ -64,7 +64,7 @@
9193
|
- ::/0
+ | IP address or IP address group of the Kafka client
|
Accessing a Kafka instance within a VPC (with SSL encryption enabled)
|
@@ -155,7 +155,7 @@
9011
|
- 0.0.0.0/0
+ | IP address or IP address group of the Kafka client
|
Accessing a Kafka instance using DNAT (in cipher- or plaintext)
|
@@ -166,7 +166,7 @@
9092
|
- 0.0.0.0/0
+ | IP address or IP address group of the Kafka client
|
Accessing a Kafka instance using a peering connection across VPCs (in plaintext)
|
@@ -177,7 +177,7 @@
9093
|
- 0.0.0.0/0
+ | IP address or IP address group of the Kafka client
|
Accessing a Kafka instance using a peering connection across VPCs (in ciphertext)
|
@@ -210,7 +210,7 @@
9094
|
- 0.0.0.0/0
+ | IP address or IP address group of the Kafka client
|
Accessing a Kafka instance over a public network (in plaintext)
|
@@ -223,7 +223,7 @@
9095
|
- 0.0.0.0/0
+ | IP address or IP address group of the Kafka client
|
Accessing a Kafka instance over a public network (in ciphertext)
|
@@ -236,7 +236,7 @@
9192
|
- ::/0
+ | IP address or IP address group of the Kafka client
|
Accessing a Kafka instance over a public network (without SSL)
|
@@ -249,7 +249,7 @@
9193
|
- ::/0
+ | IP address or IP address group of the Kafka client
|
Accessing a Kafka instance over a public network (with SSL)
|
diff --git a/docs/dms/umn/kafka-faq-190416001.html b/docs/dms/umn/kafka-faq-190416001.html
index 99a73e17c..7184b5bf5 100644
--- a/docs/dms/umn/kafka-faq-190416001.html
+++ b/docs/dms/umn/kafka-faq-190416001.html
@@ -22,6 +22,8 @@
How Do I Modify message.max.bytes?
+ Why Are Offsets Not Continuous?
+
diff --git a/docs/dms/umn/kafka-faq-191030002.html b/docs/dms/umn/kafka-faq-191030002.html
index 9e366e539..a16b04765 100644
--- a/docs/dms/umn/kafka-faq-191030002.html
+++ b/docs/dms/umn/kafka-faq-191030002.html
@@ -52,6 +52,8 @@
How Do I Bind an EIP Again?
+ Why Are the Earliest Messages Not Automatically Deleted When the Disk Capacity Reaches 95%?
+
diff --git a/docs/dms/umn/kafka-faq-191030003.html b/docs/dms/umn/kafka-faq-191030003.html
index 1fd2e76b7..43a3140d4 100644
--- a/docs/dms/umn/kafka-faq-191030003.html
+++ b/docs/dms/umn/kafka-faq-191030003.html
@@ -14,8 +14,6 @@
Do Kafka Instances Support Batch Importing Topics or Automatic Topic Creation?
- Why Do Deleted Topics Still Exist?
-
Can I View the Disk Space Used by a Topic?
Can I Add ACL Permissions for Topics?
@@ -28,6 +26,8 @@
What Can I Do If a Consumer Fails to Retrieve Messages from a Topic Due to Insufficient Permissions?
+ Can I Purchase Partitions Separately?
+
diff --git a/docs/dms/umn/kafka-faq-191030004.html b/docs/dms/umn/kafka-faq-191030004.html
index 30720d734..97763b035 100644
--- a/docs/dms/umn/kafka-faq-191030004.html
+++ b/docs/dms/umn/kafka-faq-191030004.html
@@ -10,6 +10,14 @@
Why Is a Consumer Group Still on the Monitoring Page After Being Deleted?
+ Why Do Metrics Fluctuate Significantly (Disk Read/Write Speed, Average Disk Read/Write Time, and CPU Usage)?
+
+ Why Does JVM Heap Memory Usage of JVM Fluctuate Significantly?
+
+ Why Is Production Rate Still 0 When There Are Produced Messages?
+
+ Why Is the Number of Stacked Messages Monitored as 0 when Messages Are Stacked?
+
diff --git a/docs/dms/umn/kafka-faq-200423001.html b/docs/dms/umn/kafka-faq-200423001.html
index d4bb36e1e..a78c9f626 100644
--- a/docs/dms/umn/kafka-faq-200423001.html
+++ b/docs/dms/umn/kafka-faq-200423001.html
@@ -14,6 +14,8 @@
Can I Delete Unnecessary Topics in a Consumer Group?
+ Why Are Consumer Groups Missing?
+
diff --git a/docs/dms/umn/kafka-faq-200426003.html b/docs/dms/umn/kafka-faq-200426003.html
index 62c2460fc..2adeb8ffd 100644
--- a/docs/dms/umn/kafka-faq-200426003.html
+++ b/docs/dms/umn/kafka-faq-200426003.html
@@ -1,7 +1,7 @@
Why Can't I View the Subnet and Security Group Information When Creating a DMS Instance?
- This may be because you do not have the Server Administrator and VPC Administrator permissions. For details about how to add permissions to a user group, see Viewing and Modifying User Group Information.
+ The possible cause is that you do not have the Server Administrator and VPC Administrator permissions. For details about how to add permissions to a user group, see Viewing and Modifying User Group Information.
diff --git a/docs/dms/umn/kafka-faq-200426005.html b/docs/dms/umn/kafka-faq-200426005.html
index 5ad0b37e4..497c71fe9 100644
--- a/docs/dms/umn/kafka-faq-200426005.html
+++ b/docs/dms/umn/kafka-faq-200426005.html
@@ -2,8 +2,8 @@
How Do I Select Storage Space for a Kafka Instance?
The storage space is the space for storing messages (including messages in replicas), logs and metadata. To select a storage space, specify the disk type and disk size. For more information, see Disk Types and Performance.
- For example, if the required disk size to store data for the retention period is 100 GB, the disk capacity must be at least: 100 GB x Number of replicas + 100 GB (reserved space). In a Kafka cluster, each node uses a 33 GB disk to store logs and ZooKeeper data. Therefore, the actual available storage space is less than the created storage space.
- The number of replicas (3 by default) can be configured when you create a topic. If automatic topic creation has been enabled, each automatically created topic has three replicas by default. You can change this quantity by setting default.replication.factor on the Parameters tab page.
+ For example, if the required disk size to store data for the retention period is 100 GB, the disk capacity must be at least: 100 GB × Number of replicas + 100 GB (reserved space). In a Kafka cluster, each node uses a 33 GB disk to store logs and ZooKeeper data. Therefore, the actual available storage space is less than the created storage space.
+ The number of replicas (3 by default) can be configured when you create a topic. If automatic topic creation has been enabled, each automatically created topic has three replicas by default. You can change this quantity by setting default.replication.factor on the Parameters page.
diff --git a/docs/dms/umn/kafka-faq-200426007.html b/docs/dms/umn/kafka-faq-200426007.html
index f6a011185..559cc7878 100644
--- a/docs/dms/umn/kafka-faq-200426007.html
+++ b/docs/dms/umn/kafka-faq-200426007.html
@@ -3,9 +3,9 @@
Which Capacity Threshold Policy Should I Use?
The following policies are supported:
- Stop production
When the memory usage reaches the disk capacity threshold (95%), new messages will no longer be created, but existing messages can still be retrieved until they are discarded. The default retention time is three days. This policy is suitable for scenarios where no data losses can be tolerated.
- - Automatically delete
When the memory usage reaches the disk capacity threshold (95%), messages can be created and retrieved, but 10% of the earliest messages will be deleted to ensure sufficient disk space. This policy is suitable for scenarios where no service interruption can be tolerated. Data may be lost.
+ - Automatically delete
When the memory usage reaches the disk capacity threshold (95%), messages can be produced and consumed, but 10% of the earliest messages will be deleted to ensure sufficient disk space. This policy is suitable for scenarios where no service interruption can be tolerated. Data may be lost.
- Select a proper policy based on requirements for data and service reliability. Both policies are only used for handling extreme scenarios. To avoid extreme scenarios, create sufficient disk space in the first place.
+ Select a proper policy as required for data and service reliability. Both policies are only used for handling extreme scenarios. To avoid extreme scenarios, create sufficient disk space in the first place.
diff --git a/docs/dms/umn/kafka-faq-200426009.html b/docs/dms/umn/kafka-faq-200426009.html
index 554d0f67e..b43d3f4ad 100644
--- a/docs/dms/umn/kafka-faq-200426009.html
+++ b/docs/dms/umn/kafka-faq-200426009.html
@@ -2,7 +2,7 @@
What Is the ZooKeeper Address of a Kafka Instance?
Kafka instances are managed using ZooKeeper. Opening ZooKeeper may cause misoperations and service losses. ZooKeeper is used only within Kafka clusters and does not provide services externally.
- You can use open-source Kafka clients to connect to Kafka instances and call the native APIs to create and retrieve messages.
+ You can use open-source Kafka clients to connect to Kafka instances and call the native APIs to produce and consume messages.
diff --git a/docs/dms/umn/kafka-faq-200426014.html b/docs/dms/umn/kafka-faq-200426014.html
index 06e6a9d1a..a2159fa3b 100644
--- a/docs/dms/umn/kafka-faq-200426014.html
+++ b/docs/dms/umn/kafka-faq-200426014.html
@@ -1,7 +1,7 @@
How Do I Change the SASL_SSL Setting of a Kafka Instance?
-
+
diff --git a/docs/dms/umn/kafka-faq-200426016.html b/docs/dms/umn/kafka-faq-200426016.html
index 1c767e461..e4b980652 100644
--- a/docs/dms/umn/kafka-faq-200426016.html
+++ b/docs/dms/umn/kafka-faq-200426016.html
@@ -1,9 +1,9 @@
How Many Connection Addresses Does a Kafka Instance Have by Default?
- The number of connection addresses of a Kafka instance is the same as the number of brokers of the instance. The following table lists the number of brokers corresponding to each flavor.
+ The number of connection addresses of a Kafka instance is the same as the number of brokers of the instance. The following table lists the number of brokers corresponding to each flavor.
- Table 1 Kafka instance specifications (cluster)Flavor
+Table 1 Cluster Kafka instance specificationsFlavor
|
Brokers
|
@@ -15,7 +15,7 @@
Maximum Client Connections per Broker
|
-Storage Space
+ | Storage Space (GB)
|
Traffic per Broker (MB/s)
|
@@ -31,9 +31,9 @@
15
|
-2000
+ | 2,000
|
-300 GB–300,000 GB
+ | 300–300,000
|
40
|
@@ -48,9 +48,9 @@
20
|
-2000
+ | 2,000
|
-300 GB–300,000 GB
+ | 300–300,000
|
100
|
@@ -65,9 +65,9 @@
100
|
-4000
+ | 4,000
|
-300 GB–600,000 GB
+ | 300–600,000
|
200
|
@@ -82,9 +82,9 @@
150
|
-4000
+ | 4,000
|
-300 GB–1,500,000 GB
+ | 300–1,500,000
|
375
|
@@ -95,13 +95,13 @@
200,000
|
-1500
+ | 1,500
|
200
|
-4000
+ | 4,000
|
-300 GB–1,500,000 GB
+ | 300–1,500,000
|
625
|
@@ -112,13 +112,13 @@
250,000
|
-2000
+ | 2,000
|
200
|
-4000
+ | 4,000
|
-300 GB–1,500,000 GB
+ | 300–1,500,000
|
750
|
@@ -127,56 +127,56 @@
|---|
Table 2 Kafka instance specifications (single-node)Flavor
+Table 2 Single-node Kafka instance specificationsFlavor
|
-Brokers
+ | Brokers
|
-TPS per Broker
+ | Max. TPS per Broker
|
-Maximum Partitions per Broker
+ | Max. Partitions per Broker
|
-Recommended Consumer Groups per Broker
+ | Recommended Consumer Groups per Broker
|
-Maximum Client Connections per Broker
+ | Max. Client Connections per Broker
|
-Storage Space
+ | Storage Space (GB)
|
-Traffic per Broker (MB/s)
+ | Traffic Limit per Broker (MB/s)
|
|---|
-kafka.2u4g.single.small
+ | kafka.2u4g.single.small
|
-1
+ | 1
|
-20,000
+ | 20,000
|
-100
+ | 100
|
-15
+ | 15
|
-2000
+ | 2,000
|
-100 GB–10,000 GB
+ | 100–10,000
|
-40
+ | 40
|
-kafka.2u4g.single
+ | kafka.2u4g.single
|
-1
+ | 1
|
-30,000
+ | 30,000
|
-250
+ | 250
|
-20
+ | 20
|
-2000
+ | 2,000
|
-100 GB–10,000 GB
+ | 100–10,000
|
-100
+ | 100
|
diff --git a/docs/dms/umn/kafka-faq-200426024.html b/docs/dms/umn/kafka-faq-200426024.html
index 2e7ae79e4..0f3611b52 100644
--- a/docs/dms/umn/kafka-faq-200426024.html
+++ b/docs/dms/umn/kafka-faq-200426024.html
@@ -4,7 +4,7 @@
The number of topics is related to the total number of topic partitions and the number of partitions in each topic. There is an upper limit on the aggregate number of partitions of topics. When this limit is reached, no more topics can be created.
The partition limit varies depending on the flavor, as shown in the following table.
- Table 1 Kafka instance specifications (cluster)Flavor
+Table 1 Cluster Kafka instance specificationsFlavor
|
Brokers
|
@@ -16,7 +16,7 @@
Maximum Client Connections per Broker
|
-Storage Space
+ | Storage Space (GB)
|
Traffic per Broker (MB/s)
|
@@ -32,9 +32,9 @@
15
|
-2000
+ | 2,000
|
-300 GB–300,000 GB
+ | 300–300,000
|
40
|
@@ -49,9 +49,9 @@
20
|
-2000
+ | 2,000
|
-300 GB–300,000 GB
+ | 300–300,000
|
100
|
@@ -66,9 +66,9 @@
100
|
-4000
+ | 4,000
|
-300 GB–600,000 GB
+ | 300–600,000
|
200
|
@@ -83,9 +83,9 @@
150
|
-4000
+ | 4,000
|
-300 GB–1,500,000 GB
+ | 300–1,500,000
|
375
|
@@ -96,13 +96,13 @@
200,000
|
-1500
+ | 1,500
|
200
|
-4000
+ | 4,000
|
-300 GB–1,500,000 GB
+ | 300–1,500,000
|
625
|
@@ -113,13 +113,13 @@
250,000
|
-2000
+ | 2,000
|
200
|
-4000
+ | 4,000
|
-300 GB–1,500,000 GB
+ | 300–1,500,000
|
750
|
@@ -128,56 +128,56 @@
|---|
Table 2 Kafka instance specifications (single-node)Flavor
+Table 2 Single-node Kafka instance specificationsFlavor
|
-Brokers
+ | Brokers
|
-TPS per Broker
+ | Max. TPS per Broker
|
-Maximum Partitions per Broker
+ | Max. Partitions per Broker
|
-Recommended Consumer Groups per Broker
+ | Recommended Consumer Groups per Broker
|
-Maximum Client Connections per Broker
+ | Max. Client Connections per Broker
|
-Storage Space
+ | Storage Space (GB)
|
-Traffic per Broker (MB/s)
+ | Traffic Limit per Broker (MB/s)
|
|---|
-kafka.2u4g.single.small
+ | kafka.2u4g.single.small
|
-1
+ | 1
|
-20,000
+ | 20,000
|
-100
+ | 100
|
-15
+ | 15
|
-2000
+ | 2,000
|
-100 GB–10,000 GB
+ | 100–10,000
|
-40
+ | 40
|
-kafka.2u4g.single
+ | kafka.2u4g.single
|
-1
+ | 1
|
-30,000
+ | 30,000
|
-250
+ | 250
|
-20
+ | 20
|
-2000
+ | 2,000
|
-100 GB–10,000 GB
+ | 100–10,000
|
-100
+ | 100
|
diff --git a/docs/dms/umn/kafka-faq-200426025.html b/docs/dms/umn/kafka-faq-200426025.html
index c687f6cce..4b32d6bc8 100644
--- a/docs/dms/umn/kafka-faq-200426025.html
+++ b/docs/dms/umn/kafka-faq-200426025.html
@@ -4,7 +4,7 @@
Kafka manages messages by partition. If there are too many partitions, message creation, storage, and retrieval will be fragmented, affecting the performance and stability. If the total number of partitions of topics reaches the upper limit, you cannot create more topics.
The partition limit varies depending on the flavor, as shown in the following table.
- Table 1 Kafka instance specifications (cluster)Flavor
+Table 1 Cluster Kafka instance specificationsFlavor
|
Brokers
|
@@ -16,7 +16,7 @@
Maximum Client Connections per Broker
|
-Storage Space
+ | Storage Space (GB)
|
Traffic per Broker (MB/s)
|
@@ -32,9 +32,9 @@
15
|
-2000
+ | 2,000
|
-300 GB–300,000 GB
+ | 300–300,000
|
40
|
@@ -49,9 +49,9 @@
20
|
-2000
+ | 2,000
|
-300 GB–300,000 GB
+ | 300–300,000
|
100
|
@@ -66,9 +66,9 @@
100
|
-4000
+ | 4,000
|
-300 GB–600,000 GB
+ | 300–600,000
|
200
|
@@ -83,9 +83,9 @@
150
|
-4000
+ | 4,000
|
-300 GB–1,500,000 GB
+ | 300–1,500,000
|
375
|
@@ -96,13 +96,13 @@
200,000
|
-1500
+ | 1,500
|
200
|
-4000
+ | 4,000
|
-300 GB–1,500,000 GB
+ | 300–1,500,000
|
625
|
@@ -113,13 +113,13 @@
250,000
|
-2000
+ | 2,000
|
200
|
-4000
+ | 4,000
|
-300 GB–1,500,000 GB
+ | 300–1,500,000
|
750
|
@@ -128,56 +128,56 @@
|---|
Table 2 Kafka instance specifications (single-node)Flavor
+Table 2 Single-node Kafka instance specificationsFlavor
|
-Brokers
+ | Brokers
|
-TPS per Broker
+ | Max. TPS per Broker
|
-Maximum Partitions per Broker
+ | Max. Partitions per Broker
|
-Recommended Consumer Groups per Broker
+ | Recommended Consumer Groups per Broker
|
-Maximum Client Connections per Broker
+ | Max. Client Connections per Broker
|
-Storage Space
+ | Storage Space (GB)
|
-Traffic per Broker (MB/s)
+ | Traffic Limit per Broker (MB/s)
|
|---|
-kafka.2u4g.single.small
+ | kafka.2u4g.single.small
|
-1
+ | 1
|
-20,000
+ | 20,000
|
-100
+ | 100
|
-15
+ | 15
|
-2000
+ | 2,000
|
-100 GB–10,000 GB
+ | 100–10,000
|
-40
+ | 40
|
-kafka.2u4g.single
+ | kafka.2u4g.single
|
-1
+ | 1
|
-30,000
+ | 30,000
|
-250
+ | 250
|
-20
+ | 20
|
-2000
+ | 2,000
|
-100 GB–10,000 GB
+ | 100–10,000
|
-100
+ | 100
|
diff --git a/docs/dms/umn/kafka-faq-200426026.html b/docs/dms/umn/kafka-faq-200426026.html
index 8530897ed..f81404dc7 100644
--- a/docs/dms/umn/kafka-faq-200426026.html
+++ b/docs/dms/umn/kafka-faq-200426026.html
@@ -1,7 +1,7 @@
Why Do I Fail to Create Topics?
-Possible cause: The aggregate number of partitions of created topics has reached the upper limit. The maximum number of partitions varies by instance specifications. For details, see Specifications.
+ Possible cause: The aggregate number of partitions of created topics has reached the upper limit. The limit varies with instance specifications. For details, see Single-node Kafka Instances and Cluster Kafka Instances.
Solution: Scale up the instance or delete unnecessary topics.
diff --git a/docs/dms/umn/kafka-faq-200426028.html b/docs/dms/umn/kafka-faq-200426028.html
deleted file mode 100644
index aab07dfa9..000000000
--- a/docs/dms/umn/kafka-faq-200426028.html
+++ /dev/null
@@ -1,12 +0,0 @@
-
-
- Why Do Deleted Topics Still Exist?
- Possible cause: Automatic topic creation has been enabled and a consumer is connecting to the topic. If no existing topics are available for message creation, new topics will be automatically created.
- Solution: Disable automatic topic creation.
-
-
-
diff --git a/docs/dms/umn/kafka-faq-200426030.html b/docs/dms/umn/kafka-faq-200426030.html
index de04616e0..65eb4f50f 100644
--- a/docs/dms/umn/kafka-faq-200426030.html
+++ b/docs/dms/umn/kafka-faq-200426030.html
@@ -2,7 +2,7 @@
Can I View the Disk Space Used by a Topic?
Yes. Use either of the following methods to check the disk space used by a topic:
- - In the row containing the desired Kafka instance, click View Metric to go to the Cloud Eye console. On the By Topic tab page, set Topic to the name of the desired topic, and Monitoring Type to Basic monitoring. The Message Size metric reflects the message size of the selected topic.
- Click the desired Kafka instance to view its details. Choose Monitoring in the navigation pane. On the By Topic tab page, set Topic to the name of the topic whose disk space you want to view and Monitoring Type to Basic monitoring. The Message Size metric reflects the message size of the selected topic.
+ - In the row containing the desired Kafka instance, click View Metric to go to the Cloud Eye console. On the By Topic tab page, set Topic to the name of the desired topic, and Monitoring Type to Basic monitoring. The Message Size metric reflects the message size of the selected topic.
- Click the desired Kafka instance to go to the instance details page. In the navigation pane, choose Monitoring. On the By Topic tab page, set Topic to the name of the topic whose disk space you want to view and Monitoring Type to Basic monitoring. The Message Size metric reflects the message size of the selected topic.
diff --git a/docs/dms/umn/kafka-faq-200426032.html b/docs/dms/umn/kafka-faq-200426032.html
index a3da359c3..9204594c0 100644
--- a/docs/dms/umn/kafka-faq-200426032.html
+++ b/docs/dms/umn/kafka-faq-200426032.html
@@ -1,7 +1,7 @@
Can I Add ACL Permissions for Topics?
- If you have enabled SASL_SSL for your Kafka instance, you can configure ACL permissions for your topics. On the Topics tab page of the Kafka console, click Grant User Permission in the row that contains the topic for which you want to configure user permissions.
+ If you have enabled SASL_SSL for your Kafka instance, you can configure ACL permissions for your topics. On the Topics page of the Kafka console, click Grant User Permission in the row that contains the topic for which you want to configure user permissions.
For details, see Configuring Kafka Topic Permissions.
diff --git a/docs/dms/umn/kafka-faq-200426036.html b/docs/dms/umn/kafka-faq-200426036.html
index a0bc91e48..f17627499 100644
--- a/docs/dms/umn/kafka-faq-200426036.html
+++ b/docs/dms/umn/kafka-faq-200426036.html
@@ -2,19 +2,19 @@
Why Does Message Poll Often Fail During Rebalancing?
Rebalancing is a process where partitions of topics are re-allocated for a consumer group.
- In normal cases, rebalancing occurs inevitably when a consumer is added to or removed from a consumer group. However, if a consumer is regarded as abnormal and removed from the consumer group, message retrieval may fail.
+ In normal cases, rebalancing occurs inevitably when a consumer is added to or removed from a consumer group. However, if a consumer is regarded as abnormal by the server and removed from the consumer group, message consumption may fail.
This may happen in the following scenarios:
- Heartbeat requests are not sent in time.
A consumer sends heartbeat requests to the broker at the interval specified by heartbeat.interval.ms. If the broker does not receive any heartbeat request from the consumer within the period specified by session.timeout.ms, the broker considers that the consumer is abnormal and removes the consumer from the consumer group, triggering rebalancing.
- - The interval between retrievals is too long.
The maximum number of messages that a consumer can retrieve at a time is specified by max.poll.records. In most cases, a client processes the retrieved data before starting the next retrieval. The processing may be prolonged when a large number of messages are retrieved at a time and cannot be processed within the time specified by max.poll.interval.ms, or when an exception occurs during the process (for example, data needs to be written to the backend database, but the backend database pressure is too high, resulting in high latency). If the consumer does not send the next retrieval request within the time specified by max.poll.interval.ms, the broker considers that the consumer is inactive and removes it from the consumer group, triggering rebalancing.
+ - The interval between retrievals is too long.
The maximum number of messages that a consumer can consume at a time is specified by max.poll.records. In most cases, a client processes the consumed data before starting the next consumption. The processing may be prolonged when a large number of messages are consumed at a time and cannot be processed within the time specified by max.poll.interval.ms, or when an exception occurs during the process (for example, data needs to be written to the backend database, but the backend database pressure is too high, resulting in slow SQL and high latency). If the consumer does not send the next consumption request within the time specified by max.poll.interval.ms, the broker considers that the consumer is inactive and removes it from the consumer group, triggering rebalancing.
Solutions and Troubleshooting MethodsScenario 1: Heartbeat requests are not sent in time.
- Solution: On the consumer client, set the value of session.timeout.ms to three times the value of heartbeat.interval.ms.
+ Locating: If the client is in Java, check GC logs to see whether FullGC takes a long time. If yes, the heartbeat thread may be blocked. As a result, the server fails to check heartbeats and starts rebalancing.
+ Solution: Check for client issues. FullGC indicates that memory leakage may have occurred on the client.
Scenario 2: The interval between retrievals is too long.
Troubleshooting methods:
- - Check the time required for processing a single message and whether the time required for processing a specified number (max.poll.records) of messages exceeds the time specified by max.poll.interval.ms.
- Check whether message processing requires network connections, such as writing data to the database and calling backend APIs, and whether the backend is normal in rebalancing scenarios.
- Solution: On the consumer client, decrease the value of max.poll.records.
+ - Check the time required for processing a single message and whether the time required for processing a specified number (max.poll.records) of messages exceeds the time specified by max.poll.interval.ms.
- Check whether message processing requires network connections, such as writing data to the database and calling backend APIs, and whether the downstream consumer system is normal in rebalancing scenarios.
+ Solution: On the consumer client, decrease the value of max.poll.records. If message processing takes longer, increase the value of max.poll.interval.ms.
-
diff --git a/docs/dms/umn/kafka-faq-200426100.html b/docs/dms/umn/kafka-faq-200426100.html
index 9050c2b73..ad93d2159 100644
--- a/docs/dms/umn/kafka-faq-200426100.html
+++ b/docs/dms/umn/kafka-faq-200426100.html
@@ -3,7 +3,7 @@
What Can I Do If Kafka Messages Are Accumulated?
Symptom: An alarm is generated for the Accumulated Messages metric.
Solution:
- - Log in to the Kafka console and click the instance for which the alarm is generated. The instance details page is displayed.
- Choose Monitoring in the navigation pane.
- On the By Consumer Group tab page, view Consumer Available Messages to find the consumer group with accumulated messages.
- In the navigation pane, choose Consumer Groups.
- Check whether there are consumers in the consumer group where messages are accumulated. If yes, contact the service party to accelerate their consumption. If no, contact the customer to delete unused consumer groups.
+ - Log in to the Kafka console and click the instance for which the alarm is generated. The instance details page is displayed.
- In the navigation pane, choose Monitoring.
- On the By Consumer Group tab page, view Consumer Available Messages to find the consumer group with accumulated messages.
- In the navigation pane, choose Consumer Groups.
- Check whether there are consumers in the consumer group where messages are accumulated. If yes, contact the service party to accelerate their consumption. If no, contact the customer to delete unused consumer groups.
diff --git a/docs/dms/umn/kafka-pd-0052.html b/docs/dms/umn/kafka-pd-0052.html
index c895a08ee..ec13b5ce0 100644
--- a/docs/dms/umn/kafka-pd-0052.html
+++ b/docs/dms/umn/kafka-pd-0052.html
@@ -2,120 +2,104 @@
Comparing Single-node and Cluster Kafka Instances
A single-node Kafka instance has only one broker. These instances do not guarantee performance or reliability and are for trial use or testing only. In the production environment, use cluster instances.
- Parameters for CreationParameter settings that are unique to single-node instances are listed in Table 1.
+ Table 1 compares single-node and cluster instances by features and functions.
- Table 1 Parameters of a single-node instanceParameter
+Table 1 Comparing single-node and cluster instancesItem
|
-Description
+ | Single-node
+ |
+Cluster
|
|---|
-AZ
+ | Version
|
-Only one AZ
+ | 2.7
+ |
+1.1.0, 2.3.0, 2.7, and 3.x
|
-Version
+ | AZ
|
-Only v2.7
+ | Single
+ |
+1, or 3 or more
|
-Broker Flavor
+ | Brokers
|
-kafka.2u4g.single.small and kafka.2u4g.single
+ | 1
+ |
+3 and more
|
-Brokers
+ | Access mode
|
-Only one
+ | Plaintext access
+ |
+Plaintext and ciphertext access
|
-Storage space per broker
+ | Modifying instance specifications
|
-Disk types: high I/O and ultra-high I/O; Disk size: 100–10,000 GB
+ | ×
+ |
+√
|
-Ciphertext Access
+ | Resetting Kafka password
|
-Not supported
+ | ×
+ |
+√
+ |
+
+Viewing disk usage
+ |
+×
+ |
+√
+ |
+
+Reassigning partitions
+ |
+×
+ |
+√
+ |
+
+Configuring topic permissions
+ |
+×
+ |
+√
+ |
+
+Managing users
+ |
+×
+ |
+√
+ |
+
+Smart Connect
+ |
+×
+ |
+√
+ |
+
+Modifying configuration parameters
+ |
+×
+ |
+√
|
Comparing Instance FunctionsTable 2 compares the functions of single-node and cluster instances.
-
- Table 2 Comparing functionsFunction
- |
-Single-node Instance
- |
-Cluster Instance
- |
-
|---|
-
-Modifying instance specifications
- |
-×
- |
-√
- |
-
-Changing the instance access mode
- |
-Only enabling/disabling public plaintext access
- |
-Options:
-- Enabling private ciphertext access
- Enabling/Disabling private plaintext access
- Enabling/Disabling public plaintext access
- Enabling/Disabling public ciphertext access
- |
-
-Resetting Kafka password
- |
-×
- |
-√
- |
-
-Viewing disk usage
- |
-×
- |
-√
- |
-
-Reassigning partitions
- |
-×
- |
-√
- |
-
-Configuring topic permissions
- |
-×
- |
-√
- |
-
-Managing users
- |
-×
- |
-√
- |
-
-Modifying configuration parameters
- |
-×
- |
-√
- |
-
-
-
-
-
diff --git a/docs/dms/umn/kafka-pd-0053.html b/docs/dms/umn/kafka-pd-0053.html
new file mode 100644
index 000000000..30b640454
--- /dev/null
+++ b/docs/dms/umn/kafka-pd-0053.html
@@ -0,0 +1,18 @@
+
+
+ Kafka Instance Specifications
+
+
+
diff --git a/docs/dms/umn/kafka-pd-0056.html b/docs/dms/umn/kafka-pd-0056.html
new file mode 100644
index 000000000..d135a3abf
--- /dev/null
+++ b/docs/dms/umn/kafka-pd-0056.html
@@ -0,0 +1,79 @@
+
+
+ Single-node Kafka Instances
+ Instance SpecificationsA single-node Kafka instance has one broker, is compatible with open-source Kafka 2.7, and is applicable to test scenarios. Do not use it for production services.
+ In the following table:
+ - For Kafka instances, the number of transactions per second (TPS) is the maximum number of messages that can be written per second.
- TPS performance: number of messages processed per second assuming that the size of a message is 1 KB.
- Test scenario: private access in plaintext. The disk type is ultra-high I/O.
+
+
+ Table 1 Single-node Kafka instance specificationsFlavor
+ |
+Brokers
+ |
+Max. TPS per Broker
+ |
+Max. Partitions per Broker
+ |
+Recommended Consumer Groups per Broker
+ |
+Max. Client Connections per Broker
+ |
+Storage Space (GB)
+ |
+Traffic Limit per Broker (MB/s)
+ |
+
|---|
+
+kafka.2u4g.single.small
+ |
+1
+ |
+20,000
+ |
+100
+ |
+15
+ |
+2,000
+ |
+100–10,000
+ |
+40
+ |
+
+kafka.2u4g.single
+ |
+1
+ |
+30,000
+ |
+250
+ |
+20
+ |
+2,000
+ |
+100–10,000
+ |
+100
+ |
+
+
+
+
+
+ Storage Space EstimationKafka instances can store messages in multiple replicas. The storage space is consumed by message replicas, logs, and metadata. When creating an instance, specify its storage space based on the expected service message size, the number of replicas, and reserved disk space. Each Kafka broker reserves 33 GB disk space for storing logs and metadata.
+ For example, if the expected service message size is 100 GB, the number of replicas is 2, and the number of brokers is 1, the disk size should be at least 233 GB (100 GB × 2 + 33 GB × 1).
+
+ Topic Quantity CalculationThere are limits on the topic quantity and the aggregate number of partitions in the topics. When the partition quantity limit is reached, you can no longer create topics.
+ The number of topics is related to the maximum number of partitions allowed and the specified number of partitions in each topic (see Table 1).
+ The maximum number of partitions allowed for an instance with kafka.2u4g.single is 250.
+ - If the number of partitions of each topic in the instance is 2, the maximum number of topics is 250/2 = 125.
- If the number of partitions of each topic in the instance is 1, the maximum number of topics is 250/1 = 250.
+
+
+
+
diff --git a/docs/dms/umn/kafka-pd-190605001.html b/docs/dms/umn/kafka-pd-190605001.html
index c856a0e97..aaef23cee 100644
--- a/docs/dms/umn/kafka-pd-190605001.html
+++ b/docs/dms/umn/kafka-pd-190605001.html
@@ -4,7 +4,7 @@
Apache Kafka is distributed message middleware that features high throughput, data persistence, horizontal scalability, and stream data processing. It adopts the publish-subscribe pattern and is widely used for log collection, data streaming, online/offline system analytics, and real-time monitoring.
DMS is a message queuing service based on Apache Kafka. It provides Kafka instances with isolated compute, storage, and bandwidth resources. DMS allows you to apply and configure resources based on service requirements. It can be used out of the box and frees you from deployment and O&M so that you can focus on the agile development of your applications.
- Readers' GuideThis documentation introduces DMS for Kafka and its differences from Apache Kafka. You will learn about the detailed information about the specifications, console operations, and client access to instances of DMS for Kafka.
+ Readers' GuideThis documentation introduces DMS for Kafka and its differences from Apache Kafka. You will learn about the detailed information about the specifications, console operations, and client access to instances of DMS for Kafka.
For more information about the basic knowledge of Kafka or technical details about creating and retrieving messages, please go to the official Apache Kafka website.
diff --git a/docs/dms/umn/kafka-pd-190605002.html b/docs/dms/umn/kafka-pd-190605002.html
index d31c925b2..545984929 100644
--- a/docs/dms/umn/kafka-pd-190605002.html
+++ b/docs/dms/umn/kafka-pd-190605002.html
@@ -15,6 +15,8 @@
Tag Management Service (TMS)TMS is a visualized service for fast and unified cross-region tagging and categorization of cloud services.
Tags facilitate Kafka instance identification and management.
Key Management Service (KMS)When creating a Kafka instance, you can specify whether to enable disk encryption. Enabling disk encryption improves data security. Disk encryption depends on the keys provided by KMS.
+VPC EndpointA client can access a Kafka instance over a private network: When a client and a Kafka instance are deployed across VPCs in one region, connect the client and the Kafka instance across VPCs using a VPC endpoint.
+NAT GatewayA Kafka instance can communicate with a client over a public network: Configure port mapping from EIPs to specified instance ports using destination NAT (DNAT) of NAT Gateway.
diff --git a/docs/dms/umn/kafka-pd-190605003.html b/docs/dms/umn/kafka-pd-190605003.html
index c36145930..aa7ae9c80 100644
--- a/docs/dms/umn/kafka-pd-190605003.html
+++ b/docs/dms/umn/kafka-pd-190605003.html
@@ -1,57 +1,59 @@
Notes and Constraints
- This section describes the notes and constraints on DMS.
+ This section describes the notes and constraints on Distributed Message Service (DMS). Use your Kafka instances as prescribed to avoid program exceptions.
+  Any instability caused by ignorance of the notes and constraints is not covered by the SLA.
+ Instance
- Table 1 Instance notes and constraintsItem
+Table 1 Instance notes and constraintsItem
|
-Notes and Constraints
+ | Notes and Constraints
|
|---|
-Kafka ZooKeeper
+ | Kafka ZooKeeper
|
-Kafka clusters are managed using ZooKeeper. Opening ZooKeeper may cause misoperations and service losses. Currently, ZooKeeper is used only within Kafka clusters and does not provide services externally.
+ | Kafka clusters are managed using ZooKeeper. Opening ZooKeeper may cause misoperations and service losses. Currently, ZooKeeper is used only within Kafka clusters and does not provide services externally.
|
-Version
+ | Version
|
-- The service version can be 2.3.0, 2.7, or 3.x. Kafka instances cannot be upgraded once they are created.
- Clients later than version 0.10 are supported. Use a version that is consistent with the service version.
+ | - The service version can be 2.3.0, 2.7, or 3.x. Kafka instances cannot be upgraded once they are created.
- Clients later than version 0.10 are supported. Use a version that is consistent with the service version.
|
-Logging in to the VM where the Kafka brokers reside
+ | Logging in to the VM where the Kafka brokers reside
|
-Not supported
+ | Not supported
|
-Storage
+ | Storage
|
-- The storage space can be expanded but cannot be reduced.
- You can expand the storage space up to 20 times.
+ | - The storage space of cluster instances can be expanded but cannot be reduced.
- You can expand the storage space up to 20 times.
- The storage space of single-node instances cannot be changed.
|
-Broker quantity
+ | Brokers
|
-The broker quantity can be increased but cannot be decreased.
+ | The broker quantity of cluster instances can be increased but cannot be decreased. The broker quantity of single-node instances cannot be changed.
|
-Broker flavor
+ | Broker flavor
|
-- The broker flavor can be increased but cannot be decreased.
- Single-replica topics do not support message production during this period. Services will be interrupted.
- If a topic has multiple replicas, scaling up the broker flavor does not interrupt services, but may cause disorder of partition messages. Evaluate this impact and avoid peak hours.
- Broker rolling restarts will cause partition leader changes, interrupting connections for less than a minute when the network is stable. For multi-replica topics, configure the retry mechanism on the producer client.
+ | - The broker flavor of cluster instances can be increased but cannot be decreased.
- Single-replica topics do not support message production during this period. Services will be interrupted.
- If a topic has multiple replicas, scaling up the broker flavor does not interrupt services, but may cause disorder of partition messages. Evaluate this impact and avoid peak hours.
- Broker rolling restarts will cause partition leader changes, interrupting connections for less than a minute when the network is stable. For multi-replica topics, configure the retry mechanism on the producer client.
- The broker flavor of single-node instances cannot be changed.
|
-VPC, subnet, and AZ
+ | VPC, subnet, and AZ
|
-After an instance is created, its VPC, subnet, and AZ cannot be modified.
+ | After an instance is created, its VPC, subnet, and AZ cannot be modified.
|
-Kerberos authentication
+ | Kerberos authentication
|
-Not supported
+ | Not supported
|
-Client connections from each IP address
+ | Client connections from each IP address
|
-Each Kafka broker allows a maximum of 1000 connections from each IP address by default. Excess connections will be rejected.
+ | Each Kafka broker allows a maximum of 1000 connections from each IP address by default. Excess connections will be rejected.
|
@@ -59,71 +61,71 @@
Topic
- Table 2 Topic notes and constraintsItem
+Table 2 Topic notes and constraintsItem
|
-Notes and Constraints
+ | Notes and Constraints
|
|---|
-Total number of topic partitions
+ | Total number of topic partitions
|
-The total number of topic partitions is related to the instance specifications. For details, see Specifications.
+ | The total number of topic partitions is related to the instance specifications. For details, see Cluster Kafka Instances.
Kafka manages messages by partition. If there are too many partitions, message creation, storage, and retrieval will be fragmented, affecting the performance and stability. If the total number of partitions of topics reaches the upper limit, you cannot create more topics.
|
-Number of partitions in a topic
+ | Number of partitions in a topic
|
-- Based on the open-source Kafka constraints, the number of partitions in a topic can be increased but cannot be decreased.
- To ensure performance, a partition number within 200 is recommended for each topic.
+ | - Based on the open-source Kafka constraints, the number of partitions in a topic can be increased but cannot be decreased.
- To ensure performance, a partition number within 200 is recommended for each topic.
|
-Topic quantity
+ | Topic quantity
|
-The topic quantity is related to the total number of topic partitions and number of partitions in each topic. For details, see Specifications.
+ | The topic quantity is related to the total number of topic partitions and number of partitions in each topic. For details, see Cluster Kafka Instances.
|
-Automatic topic creation
+ | Automatic topic creation
|
-Supported. If this option is enabled, a topic will be automatically created when a message is produced in or consumed from a topic that does not exist. By default, the topic has the following parameters:
-- Partitions: 3
- Replicas: 3
- Aging Time: 72
- Synchronous Replication and Synchronous Flushing disabled
- Message Timestamp: CreateTime
- Max.Message Size (bytes): 10,485,760
-After you change the value of the log.retention.hours, default.replication.factor, or num.partitions parameter, the value will be used in later topics that are automatically created.
+ | Supported. If this option is enabled, a topic will be automatically created when a message is produced in or consumed from a topic that does not exist. By default, the topic has the following parameters:
+- Partitions: 1 for single-node instances and 3 for cluster instances
- Replicas: 1 for single-node instances and 3 for cluster instances
- Aging Time: 72
- Synchronous Replication and Synchronous Flushing disabled
- Message Timestamp: CreateTime
- Max. Message Size (bytes): 10,485,760
+For cluster instances, after you change the value of the log.retention.hours (retention period), default.replication.factor (replica quantity), or num.partitions (partition quantity) parameter, the value will be used in later topics that are automatically created. These parameters cannot be changed for single-node instances.
For example, assume that num.partitions is changed to 5, an automatically created topic has the following parameters:
-- Partitions: 5
- Replicas: 3
- Aging Time: 72
- Synchronous Replication and Synchronous Flushing disabled
- Message Timestamp: CreateTime
- Max.Message Size (bytes): 10,485,760
+- Partitions: 5
- Replicas: 3
- Aging Time: 72
- Synchronous Replication and Synchronous Flushing disabled
- Message Timestamp: CreateTime
- Max. Message Size (bytes): 10,485,760
|
-Synchronous replication
+ | Synchronous replication
|
-If a topic has only one replica, synchronous replication cannot be enabled.
+ | If a topic has only one replica, synchronous replication cannot be enabled.
|
-Replica quantity
+ | Replica quantity
|
-Single-replica topics are not recommended. If an instance node is faulty, an internal service error may be reported when you query messages in a topic with only one replica. Therefore, you are not advised to use a topic with only one replica.
+ | Single-replica topics are not recommended for cluster instances. If an instance node is faulty, an internal service error may be reported when you query messages in a topic with only one replica. Therefore, you are not advised to use a topic with only one replica.
|
-Aging time
+ | Aging time
|
-The value of the log.retention.hours parameter takes effect only if the aging time has not been set for the topic.
-For example, if the aging time of Topic01 is set to 60 hours and log.retention.hours is set to 72 hours, the actual aging time of Topic01 is 60 hours.
+ | The log.retention.hours parameter takes effect only when the aging time has not been set for the topic.
+For example, if the aging time of Topic01 is set to 60 hours and log.retention.hours is set to 72 hours, the actual aging time of Topic01 is 60 hours.
|
-Batch importing and exporting topics
+ | Batch importing and exporting topics
|
-Batch export is supported, but batch import is not supported.
+ | Batch export is supported, but batch import is not supported.
|
-Topic name
+ | Topic name
|
-If a topic name starts with a special character, for example, an underscore (_) or a number sign (#), monitoring data cannot be displayed.
+ | If a topic name starts with a special character, for example, an underscore (_) or a number sign (#), monitoring data cannot be displayed.
|
-Delay queues
+ | Delay queues
|
-Not supported
+ | Not supported
|
-Broker faults
+ | Broker faults
|
-When some brokers of an instance are faulty, topics cannot be created, modified, or deleted, but can be queried.
+ | When some brokers of an instance are faulty, topics cannot be created, modified, or deleted, but can be queried.
|
@@ -131,30 +133,30 @@
Consumer Group
- Table 3 Consumer group notes and constraintsItem
+Table 3 Consumer group notes and constraintsItem
|
-Notes and Constraints
+ | Notes and Constraints
|
|---|
-Creating consumer groups, consumers, and producers
+ | Creating consumer groups, consumers, and producers
|
-Consumer groups, consumers, and producers are generated automatically when you use the instance.
+ | - When parameter auto.create.groups.enable is set to true, you do not need to create a consumer group, producer, or consumer because they are generated automatically when you use the instance.
- When parameter auto.create.groups.enable is set to false, you need to create a consumer group, but do not need to create a producer or consumer.
|
-Resetting the consumer offset
+ | Resetting the consumer offset
|
-Messages may be retrieved more than once after the offset is reset.
+ | Messages may be retrieved more than once after the offset is reset.
|
-Consumer group name
+ | Consumer group name
|
-If a consumer group name starts with a special character, for example, an underscore (_) or a number sign (#), monitoring data cannot be displayed.
+ | If a consumer group name starts with a special character, for example, an underscore (_) or a number sign (#), monitoring data cannot be displayed.
|
-Broker faults
+ | Broker faults
|
-When some instance brokers are faulty, consumer groups cannot be created or deleted, or consumption progress cannot be reset, but consumer groups can be queried.
+ | When some instance brokers are faulty, consumer groups cannot be created or deleted, or consumption progress cannot be reset, but consumer groups can be queried.
|
@@ -162,15 +164,15 @@
Message
- Table 4 Message notes and constraintsItem
+Table 4 Message notes and constraintsItem
|
-Notes and Constraints
+ | Notes and Constraints
|
|---|
-Message size
+ | Message size
|
-The maximum length of a message is 10 MB. If the length exceeds 10 MB, the production fails.
+ | The maximum length of a message is 10 MB. If the length exceeds 10 MB, the production fails.
|
@@ -178,20 +180,20 @@
User
- Table 5 User notes and constraintsItem
+Table 5 User notes and constraintsItem
|
-Notes and Constraints
+ | Notes and Constraints
|
|---|
-Number of users
+ | Number of users
|
-The maximum users that can be created for a Kafka instance is 20 or 500. Check the console for the actual limit.
+ | The maximum users that can be created for a Kafka instance is 20 or 500. Check the console for the actual limit.
|
-Broker faults
+ | Broker faults
|
-When some instance brokers are faulty, users cannot be created or deleted, or password cannot be reset, but users can be queried.
+ | When some instance brokers are faulty, users cannot be created or deleted, or password cannot be reset, but users can be queried.
|
diff --git a/docs/dms/umn/kafka-pd-200720001.html b/docs/dms/umn/kafka-pd-200720001.html
index 145a9b9e2..02fd0534e 100644
--- a/docs/dms/umn/kafka-pd-200720001.html
+++ b/docs/dms/umn/kafka-pd-200720001.html
@@ -57,7 +57,7 @@
Feature-rich
|
-While maintaining 100% open-source compatibility, DMS further optimizes open-source code to improve performance and reliability, and provides message querying, and many other features.
+ | While maintaining 100% open-source compatibility, DMS further optimizes open-source code to improve performance and reliability, and provides message querying, migration, and many other features.
|
Functionality is limited and requires self-development.
|
diff --git a/docs/dms/umn/kafka-qs-0409001.html b/docs/dms/umn/kafka-qs-0409001.html
index f64c7c4ff..1998dcc1f 100644
--- a/docs/dms/umn/kafka-qs-0409001.html
+++ b/docs/dms/umn/kafka-qs-0409001.html
@@ -11,33 +11,33 @@
This section uses the example of creating a topic on the console.
Step 4: Connect to a Kafka Instance to Produce and Consume MessagesBefore connecting to a Kafka instance with SASL_SSL enabled, download the certificate and configure the connection in the client configuration file.
-Step 1: Preparations- Grant Kafka instance permissions.
To achieve fine-grained management of your cloud resources, create Identity and Access Management (IAM) user groups and users and grant specified permissions to the users. For more information, see Creating a User and Granting DMS for Kafka Permissions.
+Step 1: Preparations- Grant Kafka instance permissions.
To achieve fine-grained management of your cloud resources, create Identity and Access Management (IAM) user groups and users and grant specified permissions to the users. For more information, see Creating an IAM User and Granting DMS for Kafka Permissions.
- Create a VPC and subnet.
Before creating a Kafka instance, ensure that a VPC and a subnet are available. For details about how to create a VPC and a subnet, see Creating a VPC.
- The VPC must be created in the same region as the Kafka instance.
-
+The VPC must be created in the same region as the Kafka instance.
- Create a security group and add security group rules.
Before creating a Kafka instance, ensure that a security group is available. For details about how to create a security group, see Creating a Security Group.
+The security group must be created in the same region as the Kafka instance.
To connect to Kafka instances, add the security group rules described in Table 1. Other rules can be added based on site requirements.
- Table 1 Security group rulesDirection
+Table 1 Security group rulesDirection
|
-Protocol
+ | Protocol
|
-Port
+ | Port
|
-Source address
+ | Source address
|
-Description
+ | Description
|
|---|
-Inbound
+ | Inbound
|
-TCP
+ | TCP
|
-9093
+ | 9093
|
-0.0.0.0/0
+ | 0.0.0.0/0
|
-Accessing a Kafka instance over a private network within a VPC (in ciphertext)
+ | Accessing a Kafka instance over a private network within a VPC (in ciphertext)
|
@@ -47,13 +47,13 @@
- Construct a client for message production and consumption.
This section uses a Linux elastic cloud server (ECS) as the client. Before creating a Kafka instance, create an ECS with an EIP, install the JDK, configure environment variables, and download an open-source Kafka client. - Log in to the console, click
 in the upper left corner, click Elastic Cloud Server under Computing, and then create an ECS.For details about how to create an ECS, see Creating an ECS. If you already have an available ECS, skip this step.
- - Log in to an ECS as user root.
- Install Java JDK and configure the environment variables JAVA_HOME and PATH.
- Download a JDK.
Use Oracle JDK instead of ECS's default JDK (for example, OpenJDK), because ECS's default JDK may not be suitable. Obtain Oracle JDK 1.8.111 or later from Oracle's official website.
+ - Log in to an ECS as user root.
- Install Java JDK and configure the environment variables JAVA_HOME and PATH.
- Download a JDK.
Use Oracle JDK instead of ECS's default JDK (for example, OpenJDK), because ECS's default JDK may not be suitable. Obtain Oracle JDK 1.8.111 or later from Oracle's official website.
- Decompress the JDK.
tar -zxvf jdk-8u321-linux-x64.tar.gz
Change jdk-8u321-linux-x64.tar.gz to your JDK version.
- Open the .bash_profile file.
vim ~/.bash_profile
- - Add the following content:
export JAVA_HOME=/root/jdk1.8.0_321
-export PATH=$JAVA_HOME/bin:$PATH
+ - Press i and add the following content.
export JAVA_HOME=/root/jdk1.8.0_321
+export PATH=$JAVA_HOME/bin:$PATH
Change /root/jdk1.8.0_321 to the path where you install JDK.
- Press Esc. Enter the following line and press Enter. Save the .bash_profile file and exit.
:wq
- Run the following command to make the change take effect:
source .bash_profile
@@ -61,7 +61,7 @@ export PATH=$JAVA_HOME/bin:$PATH
If the following message is returned, the JDK is installed. java version "1.8.0_321"
- - Download an open-source Kafka client.
wget https://archive.apache.org/dist/kafka/2.7.2/kafka_2.12-2.7.2.tgz
+ - Download an open-source Kafka client.
wget https://archive.apache.org/dist/kafka/2.7.2/kafka_2.12-2.7.2.tgz
- Run the following command to decompress the package:
tar -zxf kafka_2.12-2.7.2.tgz
@@ -77,7 +77,7 @@ export PATH=$JAVA_HOME/bin:$PATH
Region
|
DMS for Kafka in different regions cannot communicate with each other over an intranet. Select a nearest location for low latency and fast access.
-
+Select eu-de.
|
Project
@@ -115,12 +115,6 @@ export PATH=$JAVA_HOME/bin:$PATH
Select 2.7.
|
-CPU Architecture
- |
-x86
-Retain the default value.
- |
-
Broker Flavor
|
Select a broker flavor as required.
@@ -195,7 +189,7 @@ export PATH=$JAVA_HOME/bin:$PATH
| When this parameter is enabled, SASL authentication is required when a client connects to the Kafka instance.
- Ciphertext Access is enabled.
- SASL_SSL is selected. Username and Password can be set. Enter "test" for the username. The username cannot be changed once ciphertext access is enabled.
A username should contain 4 to 64 characters, start with a letter, and contain only letters, digits, hyphens (-), and underscores (_).
A password must meet the following requirements:
-- Contains 8 to 32 characters.
- Contains at least three types of the following characters: uppercase letters, lowercase letters, digits, and special characters `~! @#$ %^&*()-_=+\|[{}];:'",<.>? and spaces, and cannot start with a hyphen (-).
- Cannot be the username spelled forwards or backwards.
+- Contains 8 to 32 characters.
- Contains at least three types of the following characters: uppercase letters, lowercase letters, digits, and special characters `~!@#$%^&*()-_=+\|[{}];:'",<.>? and spaces, and cannot start with a hyphen (-).
- Cannot be the username spelled forwards or backwards.
- SASL/PLAIN is enabled.
|
@@ -209,7 +203,7 @@ export PATH=$JAVA_HOME/bin:$PATH
- Skip Advanced Settings.
- Click Create.
- Confirm the instance settings.
- Return to the DMS for Kafka page and check whether the instance has been created.
It takes 3 to 15 minutes to create an instance. During this period, the instance status is Creating.
+ - Skip Advanced Settings.
- Click Create.
- Confirm the instance information and then submit the request.
- Return to the Kafka Instances page and check whether the instance has been created.
It takes 3 to 15 minutes to create an instance. During this period, the instance status is Creating.
- If the instance is created successfully, its status changes to Running.
- If the instance is in the Failed state, delete it and try creating another one. If the instance creation fails again, contact customer service.
Instances that fail to be created do not occupy other resources.
@@ -227,7 +221,7 @@ export PATH=$JAVA_HOME/bin:$PATH
Customize a name that contains 3 to 200 characters, starts with a letter or underscore (_), and contains only letters, digits, periods (.), hyphens (-), and underscores (_).
The name must be different from preset topics:
-- __consumer_offsets
- __transaction_state
- __trace
- __connect-status
- __connect-configs
- __connect-offsets
+- __consumer_offsets
- __transaction_state
- __trace
- __connect-status
- __connect-configs
- __connect-offsets
- __dms_dial_test
Cannot be changed once the topic is created.
Enter topic-01.
|
@@ -286,11 +280,11 @@ password="**********";
sasl.mechanism=PLAIN
security.protocol=SASL_SSL
-ssl.truststore.location={ssl_truststore_path}
+ssl.truststore.location={ssl_truststore_path}
ssl.truststore.password=dms@kafka
ssl.endpoint.identification.algorithm=
Description:
-- username and password are specified when enabling ciphertext access during instance creation.
- ssl.truststore.location is the path for storing the certificate obtained in 1.b.
- ssl.truststore.password is certified by the server, which must be set to dms@kafka and cannot be changed.
- ssl.endpoint.identification.algorithm decides whether to verify the certificate domain name. In this example, leave this parameter blank, which indicates disabling domain name verification.
+- username and password are specified when enabling ciphertext access during instance creation.
- ssl.truststore.location is the path for storing the certificate obtained in 1.b.
- ssl.truststore.password is certified by the server, which must be set to dms@kafka and cannot be changed.
- ssl.endpoint.identification.algorithm decides whether to verify the certificate domain name. In this example, leave this parameter blank, which indicates disabling domain name verification.
- Go to the /bin directory on the Kafka client.
cd ../bin
- Produce messages.
./kafka-console-producer.sh --broker-list ${connection address} --topic ${topic name} --producer.config ../config/producer.properties
diff --git a/docs/dms/umn/kafka-scenarios.html b/docs/dms/umn/kafka-scenarios.html
index 68cccf92f..38a465650 100644
--- a/docs/dms/umn/kafka-scenarios.html
+++ b/docs/dms/umn/kafka-scenarios.html
@@ -2,6 +2,7 @@
Application Scenarios
Kafka is popular message-oriented middleware that features highly reliable, asynchronous message delivery. It is widely used for transmitting data between different systems in many industries, including enterprise application, payment, telecommunications, e-commerce, social networking, instant messaging, video, Internet of Things, and Internet of Vehicle.
+ For synchronization links that require real-time results, Remote Procedure Call (RPC) is recommended.
Asynchronous CommunicationNon-core or less important messages are sent asynchronously to receiving systems, so that the main service process is not kept waiting for the results of other systems, allowing for faster responses.
For example, Kafka can be used to send a notification email and SMS message after a user has registered with a website, providing fast responses throughout the registration process.
Figure 1 Serial registration and notification
diff --git a/docs/dms/umn/kafka-trouble-0001.html b/docs/dms/umn/kafka-trouble-0001.html
index 229417eaf..23c470755 100644
--- a/docs/dms/umn/kafka-trouble-0001.html
+++ b/docs/dms/umn/kafka-trouble-0001.html
@@ -5,7 +5,7 @@
Root CauseThe disk usage of the broker is too high.
-
diff --git a/docs/dms/umn/kafka-ug-00001.html b/docs/dms/umn/kafka-ug-00001.html
index 0c02ef001..5e16a9a50 100644
--- a/docs/dms/umn/kafka-ug-00001.html
+++ b/docs/dms/umn/kafka-ug-00001.html
@@ -8,12 +8,18 @@
|
|---|
-2024-09-25
+ | 2025-07-01
+ |
+Added:
+
+ |
+
+2024-09-25
|
Added:
Modified:
-
+
|
2024-07-19
@@ -25,9 +31,9 @@
| 2024-06-14
|
Added the following content:
-
+
Modified the following content:
-
+
|
2022-12-01
@@ -68,7 +74,7 @@
| 2019-11-30
|
This release incorporates the following changes:
-- Optimized description about Kafka premium instances in section What Is DMS?.
- Added description about DMS functions in section "Functions".
- Added description about the differences between Kafka queues and Kafka premium instances in section "Comparing Kafka Queues and Kafka Premium Instances".
- Added description about the specifications of Kafka queues and Kafka premium instances in section Specifications.
- Added explanation of the relationship between Kafka premium instances and VPCs in section Related Services.
- Optimized description about DMS queues and added description about Kafka premium instances in section Basic Concepts.
- Added recommendations on Kafka client configurations in Best Practices.
- Removed the Region parameter in section "Creating a Queue".
- Optimized description about queue policy parameters in section "Managing Queue Policies".
- Optimized description about Kafka premium instance operations and parameters in chapter "Managing Kafka Premium Instances".
- Optimized description about DMS metrics in section Kafka Metrics.
- Added description about Kafka premium instance operations supported by CTS in section Viewing Kafka Audit Logs.
+- Optimized description about Kafka premium instances in section What Is DMS?.
- Added description about DMS functions in section "Functions".
- Added description about the differences between Kafka queues and Kafka premium instances in section "Comparing Kafka Queues and Kafka Premium Instances".
- Added description about the specifications of Kafka queues and Kafka premium instances in section Cluster Kafka Instances.
- Added explanation of the relationship between Kafka premium instances and VPCs in section Related Services.
- Optimized description about DMS queues and added description about Kafka premium instances in section Basic Concepts.
- Added recommendations on Kafka client configurations in Best Practices.
- Removed the Region parameter in section "Creating a Queue".
- Optimized description about queue policy parameters in section "Managing Queue Policies".
- Optimized description about Kafka premium instance operations and parameters in chapter "Managing Kafka Premium Instances".
- Optimized description about DMS metrics in section Kafka Metrics.
- Added description about Kafka premium instance operations supported by CTS in section Viewing Kafka Audit Logs.
|
2019-11-08
diff --git a/docs/dms/umn/kafka-ug-0001.html b/docs/dms/umn/kafka-ug-0001.html
index 7a4be1bb4..1690657ba 100644
--- a/docs/dms/umn/kafka-ug-0001.html
+++ b/docs/dms/umn/kafka-ug-0001.html
@@ -11,30 +11,133 @@
Options:
- PLAINTEXT: There is no authentication required in such a connection and data is transmitted in plaintext.
- SASL_SSL: Clients can connect to a Kafka instance with SASL and the data will be encrypted using the SSL certificate.
- SASL_PLAINTEXT: Clients can connect to a Kafka instance with SASL and the data will be transmitted in plaintext.
-Creating a VPC Endpoint Service- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
-
- - Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired Kafka instance to view its details.
- In the Advanced Settings section on the Basic Information tab page, obtain the listeners IP addresses and port IDs of the instance for Cross-VPC Access.
Figure 2 Cross-VPC access–related listeners IP addresses and corresponding port IDs of the Kafka instance
- - In the Network section on the Basic Information tab page, view the VPC to which the Kafka instance belongs.
Figure 3 Viewing the VPC to which the Kafka instance belongs
+Creating a VPC Endpoint Service- Log in to the console.
- Click in the upper left corner to select the region where your instance is located.
- Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired instance to go to the instance details page.
- In the Advanced Settings area on the Basic Information page, obtain the listeners IP addresses and port IDs of the instance for Cross-VPC Access.
Figure 2 Cross-VPC access–related listeners IP addresses and corresponding port IDs of the Kafka instance
+ - In the Network area on the Basic Information page, view the VPC to which the Kafka instance belongs.
Figure 3 Viewing the VPC to which the Kafka instance belongs
- Click the VPC to obtain the VPC ID on the VPC console.
Figure 4 Obtaining the VPC ID
- - Call the VPC Endpoint API to create a VPC endpoint service. For details, see Creating a VPC Endpoint Service.
POST https://{endpoint}/v1/{project_id}/vpc-endpoint-services
-Set the following request parameter to the specified values, and other parameters as required.
-- port_id: one of the port IDs obtained in 5.
- vpc_id: VPC ID obtained in 7.
- server_type: VM
- client_port: 9011
- server_port: 9011
- protocol: TCP
- approval_enabled: false
- service_type: interface
- endpoint: VPCEP endpoint obtained from Regions and Endpoints. The region must be the same as that of the Kafka instance.
- project_id: project ID obtained from Obtaining a Project ID. The region must be the same as that of the Kafka instance.
+ - Call the VPC Endpoint API to create a VPC endpoint service. For details, see Creating a VPC Endpoint Service.
POST https://{endpoint}/v1/{project_id}/vpc-endpoint-services
+Set request parameters by referring to Table 1, and other parameters as required.
+
+Table 1 VPC Endpoint service creation parametersParameter
+ |
+Description
+ |
+
|---|
+
+port_id
+ |
+ID of the backend resource.
+Enter a port ID obtained in 5.
+ |
+
+vpc_id
+ |
+ID of the VPC of the backend resource.
+Enter a VPC ID obtained in 7.
+ |
+
+server_type
+ |
+Resource type.
+Enter VM.
+ |
+
+client_port
+ |
+Access port.
+Enter 9011.
+ |
+
+server_port
+ |
+Service port.
+Enter 9011.
+ |
+
+protocol
+ |
+Port mapping protocol.
+Enter TCP.
+ |
+
+approval_enabled
+ |
+Whether approval is required.
+Entering false indicates that no approval is required and the connected VPC endpoint will be in the accepted state.
+ |
+
+service_type
+ |
+Service type.
+Enter interface.
+ |
+
+endpoint
+ |
+VPCEP endpoint obtained from Regions and Endpoints. The region must be the same as that of the Kafka instance.
+ |
+
+project_id
+ |
+project ID obtained from Obtaining a Project ID. The region must be the same as that of the Kafka instance.
+ |
+
+
+
+ Record the value of service_name in the response. This parameter indicates the name of the VPC endpoint service.
- Repeat 8 to create VPC endpoint services for other port IDs obtained in 5 and record the VPC endpoint service names.
(Optional) Adding a WhitelistThe VPC endpoint service can be used across accounts through a whitelist.
If the Kafka client and Kafka instance belong to different accounts, add the ID of the account to which the Kafka client belongs to the whitelist of the endpoint service. For details, see Add a Whitelist Record.
Creating a VPC Endpoint- Click Service List. Then choose Network > VPC Endpoint.
- Click Create VPC Endpoint.
- Set the following parameters:
- Region: Select the region that the Kafka instance is in.
- Service Category: Select Find a service by name.
- VPC Endpoint Service Name: Enter the VPC endpoint service name recorded in 8 and click Verify. If Service name found is displayed, proceed with subsequent operations.
- VPC: Select the VPC that the Kafka client is in.
- Subnet: Select the subnet that the Kafka client is in.
- Private IP Address: Select Automatic.
-Retain the default values for other parameters. For details, see Creating a VPC Endpoint.
- - Click Create Now.
- Confirm the configurations and submit the request.
- Go back to the VPC endpoint list and check whether the status of the created VPC endpoint has changed to Accepted. The Accepted state means that the VPC endpoint has been connected to the VPC endpoint service.
Figure 5 Checking the VPC endpoint status
+Creating a VPC Endpoint- Click Service List. Then choose Network > VPC Endpoint.
- Click Create VPC Endpoint.
- Set parameters by referring to Table 2. Retain the default values for other parameters. For details, see Creating a VPC Endpoint.
+
+
Table 2 VPC endpoint creation parametersParameter
+ |
+Description
+ |
+
|---|
+
+Region
+ |
+Region where the endpoint is located. Select the region that the Kafka instance is in.
+ |
+
+Service Category
+ |
+- Cloud services: Select this option if the VPC endpoint service to be accessed is a cloud service.
- Find a service by name: Select this option if the VPC endpoint service to be accessed is a private service of your own.
+Select Find a service by name.
+ |
+
+VPC Endpoint Service Name
+ |
+Enter the VPC endpoint service name recorded in 8 and click Verify. If Service name found is displayed, proceed with subsequent operations.
+ |
+
+VPC
+ |
+VPC where the endpoint is located. Select the VPC that the Kafka client is in.
+ |
+
+Subnet
+ |
+Subnet where the endpoint is located. Select the subnet that the Kafka client is in.
+ |
+
+Private IP Address
+ |
+IPv4 address of the endpoint. Select Automatic.
+ |
+
+
+
+ - Click Create Now.
- Confirm the configurations and submit the request.
- Go back to the VPC endpoint list and check whether the status of the created VPC endpoint has changed to Accepted. The Accepted state means that the VPC endpoint has been connected to the VPC endpoint service.
Figure 5 Checking the VPC endpoint status
- Click the VPC endpoint ID. On the Summary tab page, obtain the private IP address.
You can use the private IP address to access the VPC endpoint service.
Figure 6 Viewing the private IP address
- Repeat 1 to 7 to create a VPC endpoint for each VPC endpoint service created in 9, and view and record the private IP addresses of the VPC endpoint services.
Modifying Parameter advertised.listeners IP- Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired Kafka instance to view its details.
- On the Advanced Settings section of the Basic Information tab page, click Modify for Cross-VPC Access to change the value of advertised.listeners IP address to the private IP addresses recorded in 7 and 8. Click Save.
Each IP address must match the corresponding port ID. Otherwise, the network will be disconnected.
-
-Figure 7 Changing the advertised.listeners IP addresses
+Modifying Parameter advertised.listeners IP- Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired Kafka instance to view its details.
- In the Advanced Settings area on the Basic Information page, click Modify for Cross-VPC Access to change the value of advertised.listeners IP address to the private IP addresses recorded in 7 and 8. Each IP address must match the corresponding port ID. Otherwise, the network will be disconnected. After the modification, click Save.
Figure 7 Changing the advertised.listeners IP addresses
Verifying ConnectivityCheck whether messages can be created and retrieved by referring to Connecting to Kafka Using the Client (Plaintext Access) or Connecting to Kafka Using the Client (Ciphertext Access).
diff --git a/docs/dms/umn/kafka-ug-0002.html b/docs/dms/umn/kafka-ug-0002.html
index c7c972b2a..d1c6c3e9a 100644
--- a/docs/dms/umn/kafka-ug-0002.html
+++ b/docs/dms/umn/kafka-ug-0002.html
@@ -2,27 +2,22 @@
Configuring Kafka Topic Permissions
Kafka instances with ciphertext access enabled support access control list (ACL) for topics. You can differentiate user permissions by granting users different permissions in a topic.
- This section describes how to grant topic permissions to users after ciphertext access is enabled for Kafka instances.
- This function is unavailable for single-node instances.
-
- Constraints- If parameter allow.everyone.if.no.acl.found is set to true and no topic is granted for a user, all users can subscribe to or publish messages to the topic. If permissions for a topic has been granted to one or more users, only these users can subscribe to or publish messages to the topic. The value of allow.everyone.if.no.acl.found can be modified.
- If allow.everyone.if.no.acl.found is set to false, only the authorized users can subscribe to or publish messages to the topic. The value of allow.everyone.if.no.acl.found can be modified.
- If both the default and individual user permissions are configured for a topic, the union of the permissions is used.
+ This section describes how to grant topic permissions to users after ciphertext access is enabled for a Kafka instance.
+ Notes and Constraints- If parameter allow.everyone.if.no.acl.found is set to true and no topic is granted for a user, all users can subscribe to or publish messages to the topic. If permissions for a topic have been granted to one or more users, only these users can subscribe to or publish messages to the topic. The value of allow.everyone.if.no.acl.found can be modified.
- If allow.everyone.if.no.acl.found is set to false, only the initial user (set when ciphertext access is enabled for the first time) and other authorized users have the permission to subscribe to or publish messages to topics. The value of allow.everyone.if.no.acl.found can be modified.
- If both the default and individual user permissions are configured for a topic, the union of the permissions is used.
- Unavailable for single-node instances.
- Permissions may be temporarily invalid during the configuration, throwing client error message "AuthorizationException". In this case, set a retry mechanism on the client. For details, see Suggestions on Using the Kafka Client.
- Procedure- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
-
- - Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired Kafka instance to view the instance details.
- In the navigation pane, choose Topics.
- In the row containing the desired topic, click Grant User Permission.
- Grant topic permissions to users.
- To grant the same permissions to all users, select Default permissions and then select permissions. As shown in the following figure, all users have the permission to publish messages to this topic.
Figure 1 Granting the same permissions to all users
- - To grant different permissions to different users, do not select Default permissions. In the Users area of the Grant User Permission dialog box, select target users. If there are many users, enter the username in the search box for a quick search. In the Selected area, configure permissions (Subscribe, Publish, or Publish/Subscribe) for the users. As shown in the following figure, only the test, send, and receive users can subscribe to or publish messages to this topic. The send_receive user cannot subscribe to or publish messages to this topic.
Figure 2 Granting permissions to individual users
+Setting Permissions for a Topic- Log in to the console.
- Click in the upper left corner to select the region where your instance is located.
- Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired Kafka instance to view the instance details.
- In the navigation pane, choose Topics.
- In the row containing the desired topic, click Grant User Permission.
- Grant topic permissions to users.
- To grant the same permissions to all users, select Default permissions and then select permissions. As shown in the following figure, all users have the permission to publish messages to this topic.
Figure 1 Granting the same permissions to all users
+ - To grant different permissions to different users, do not select Default permissions. In the Users area of the Grant User Permission dialog box, select target users. If there are many users, enter the username in the search box for a quick search. In the Selected area, configure permissions (Subscribe, Publish, or Publish/Subscribe) for the users. As shown in the following figure, only the test, send, and receive users can subscribe to or publish messages to this topic. The send_receive user cannot subscribe to or publish messages to this topic.
Figure 2 Granting permissions to individual users
-If both the default and individual user permissions are configured for a topic, the union of the permissions is used. As shown in the following figure, the test and receive users can subscribe to and publish messages to this topic.
+If both the default and individual user permissions are configured for a topic, the union of the permissions is used. As shown in the following figure, the test and receive users can subscribe to and publish messages to this topic, while other users can only publish messages to this topic.
Figure 3 Granting topic permissions to users
- Click OK.
On the Topics tab page, click  next to the topic name to view the authorized users and their permissions.
Figure 4 Viewing authorized users and their permissions
Deleting Topic Permissions- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
-
- - Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired Kafka instance to view the instance details.
- In the navigation pane, choose Topics.
- In the row containing the desired topic, click Grant User Permission.
- In the Selected area of the displayed Grant User Permission dialog box, locate the row that contains the user whose permissions are to be removed, click Delete, and click OK.
+ Deleting Permissions for a Topic- Log in to the console.
- Click in the upper left corner to select the region where your instance is located.
- Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired Kafka instance to view the instance details.
- In the navigation pane, choose Topics.
- In the row containing the desired topic, click Grant User Permission.
- In the Selected area of the displayed Grant User Permission dialog box, locate the row containing the user whose permissions are to be removed, click Delete, and click OK.
To check whether the user permission has been deleted, click before the topic on the topic list page. Deleted users are no longer displayed.
+
diff --git a/docs/dms/umn/kafka-ug-0003.html b/docs/dms/umn/kafka-ug-0003.html
index 5eb238113..0802eed6e 100644
--- a/docs/dms/umn/kafka-ug-0003.html
+++ b/docs/dms/umn/kafka-ug-0003.html
@@ -3,60 +3,50 @@
Configuring Kafka ACL Users
Kafka instances with ciphertext access enabled support access control list (ACL) for topics. You can differentiate user permissions by granting users different permissions in a topic.
This section describes how to create users, reset the password, and delete users with ciphertext access enabled. For details about how to grant topic permissions for users, see Configuring Kafka Topic Permissions.
- The maximum number of users that can be created for a Kafka instance is 20 or 500. Check the console for the actual limit.
There are two ways to create a user on the console. Accordingly, there are two ways to reset the user's password:
-
- Prerequisites- Ciphertext access has been enabled for the Kafka instance.
- Kafka users can be configured only for Kafka instances in the Running state.
+
+ Notes and Constraints- Single-node instances do not support user creation, user password reset, or user deletion.
- The initial user set when ciphertext access is enabled for the first time cannot be deleted.
- Resetting a user password will interrupt services. Change the user password in the client configuration file or code as soon as possible.
- The maximum number of users that can be created for a Kafka instance is 20 or 500. Check the console for the actual limit.
- Constraints- Single-node instances do not support user creation, user password reset, or user deletion.
- Resetting a user password will interrupt services. Change the user password in the client configuration file or code as soon as possible.
+ Prerequisites- Ciphertext access has been enabled for the Kafka instance.
- The Kafka instance is in the Running state.
- Creating a User- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
-
- - Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired instance to go to the instance details page.
- On the Users page, click Create User.
- Set user information by referring to Configuring Kafka ACL Users.
- Table 1 User creation parametersParameter
+Creating a User- Log in to the console.
- Click in the upper left corner to select the region where your instance is located.
- Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired instance to go to the instance details page.
- In the navigation pane, choose Users.
- Click Create User in the upper left corner.
- Set user information by referring to Table 1.
+ Table 1 User creation parametersParameter
|
-Description
+ | Description
|
|---|
-Username
+ | Username
|
-The username used to access a Kafka instance, you can customize a name that complies with the rules: 4–64 characters; starts with a letter; can contain only letters, digits, hyphens (-), and underscores (_).
+ | The username used to access a Kafka instance, you can customize a name that complies with the rules: 4–64 characters; starts with a letter; can contain only letters, digits, hyphens (-), and underscores (_).
|
-Password
+ | Password
|
-The password used to access a Kafka instance. A password must meet the following requirements:
-- Contains 8 to 32 characters.
- Contains at least three of the following character types: uppercase letters, lowercase letters, digits, and special characters `~! @#$ %^&*()-_=+\|[{}];:'",<.>? and spaces, and cannot start with a hyphen (-).
- Cannot be the username spelled forward or backward.
+ | The password used to access a Kafka instance. A password must meet the following requirements:
+- Contains 8 to 32 characters.
- Contains at least three types of the following characters: uppercase letters, lowercase letters, digits, and special characters `~!@#$%^&*()-_=+\|[{}];:'",<.>? and spaces, and cannot start with a hyphen (-).
- Cannot be the username spelled forward or backward.
|
- Click OK.
After the user is created, grant permissions to the user by referring to Configuring Kafka Topic Permissions.
+ - Click OK.
View the new user on the user list page.
+After the user is created, grant permissions to the user by referring to Configuring Kafka Topic Permissions.
Resetting the Password (for the Initial User)- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
-
- - Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Reset the password for the initial user in either of the following ways.
- Choose More > Reset Kafka Password in the row containing the desired Kafka instance.
- Click the desired Kafka instance to view its details. Choose More > Reset Kafka Password in the upper left corner.
- Click the desired Kafka instance to view its details. On the Basic Information page, click Reset Password next to Username in the Connection section.
- Click the desired Kafka instance to view its details. On the Users page, click Reset Password in the row containing the desired user.
- - Enter and confirm a new password, and click OK.
- If the password is successfully reset, a success message is displayed.
- If the password fails to be reset, a failure message is displayed. In this case, reset the password again. If you still fail to reset the password after multiple attempts, contact customer service.
- The system will display a success message only after the password is successfully reset on all brokers.
-
+Resetting the Password (for the Initial User)- Log in to the console.
- Click in the upper left corner to select the region where your instance is located.
- Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Reset the password for the initial user in either of the following ways.
- Choose More > Reset Kafka Password in the row containing the desired Kafka instance.
- Click the desired Kafka instance to go to the instance details page. Choose More > Reset Kafka Password in the upper right corner.
- Click the desired Kafka instance to go to the instance details page. On the Basic Information page, click Reset Password next to Username in the Connection section.
- Click the desired Kafka instance to go to the instance details page. On the Users page, click Reset Password in the row containing the desired user.
+ - Enter and confirm a new password, and click OK.
Resetting the User Password (for Non-initial Users)- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
-
- - Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired Kafka instance to view its details.
- On the Users page, click Reset Password in the row containing the desired user.
- Enter and confirm a new password, and click OK.
- If the password is successfully reset, a success message is displayed.
- If the password fails to be reset, a failure message is displayed. In this case, reset the password again. If you still fail to reset the password after multiple attempts, contact customer service.
+Resetting the Password (for Non-initial Users)- Log in to the console.
- Click in the upper left corner to select the region where your instance is located.
- Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired Kafka instance to go to the instance details page.
- On the Users page, click Reset Password in the row containing the desired user.
- Enter and confirm a new password, and click OK.
- If the password is successfully reset, a success message is displayed.
- If the password fails to be reset, a failure message is displayed. In this case, reset the password again. If you still fail to reset the password after multiple attempts, contact customer service.
The system will display a success message only after the password is successfully reset on all brokers.
Deleting a User- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
-
- - Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired Kafka instance to view its details.
- In the navigation pane, choose Users.
- Delete a user in either of the following ways:
- In the row containing the desired user, click Delete.
- Select one or more users and click Delete above the list.
- The initial user set when ciphertext access is enabled for the first time cannot be deleted.
-
- - In the displayed Delete User dialog box, click OK to delete the user.
+ Deleting a User- Log in to the console.
- Click in the upper left corner to select the region where your instance is located.
- Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired instance to go to the instance details page.
- In the navigation pane, choose Users.
- Delete a user in either of the following ways:
- In the row containing the desired user, click Delete.
- Select one or more users and click Delete above the list.
+ - In the displayed Delete User dialog box, click OK to delete the user.
The user is deleted if it is not displayed in the user list.
+
diff --git a/docs/dms/umn/kafka-ug-0004.html b/docs/dms/umn/kafka-ug-0004.html
index 71be31fb0..cc9902256 100644
--- a/docs/dms/umn/kafka-ug-0004.html
+++ b/docs/dms/umn/kafka-ug-0004.html
@@ -2,11 +2,9 @@
Viewing Kafka Disk Usage
This section describes how to view the disk usage of each broker of a Kafka instance on the console.
- This function is unavailable for single-node instances.
-
- Procedure- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
-
- - Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click a Kafka instance to go to the Basic Information page.
- Go to the Disk Usage Statistics page.
You can query topics that use the most disk space or topics that have used a specified amount or percentage of disk space.
+Notes and ConstraintsUnavailable for single-node instances.
+ Procedure- Log in to the console.
- Click in the upper left corner to select the region where your instance is located.
- Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the name of the desired Kafka instance to go to the Basic Information page.
- Go to the Monitoring > Disk Usage Statistics page.
You can query topics that use the most disk space or topics that have used a specified amount or percentage of disk space.
In the upper right corner of the page, click View Metric. On the displayed Cloud Eye page, you can view metrics of Kafka instances.
Figure 1 Viewing disk usage
diff --git a/docs/dms/umn/kafka-ug-0006.html b/docs/dms/umn/kafka-ug-0006.html
index 9e150d319..298faa4d3 100644
--- a/docs/dms/umn/kafka-ug-0006.html
+++ b/docs/dms/umn/kafka-ug-0006.html
@@ -1,59 +1,79 @@
Changing Kafka Partition Quantity
- After creating a topic, you can increase the number of partitions as required.
- Changing the number of partitions does not restart the instance or affect services.
-
+ After creating a topic, you can change the number of partitions as required. Changing the number of partitions does not restart the instance or affect services.
Methods for changing the partition quantity:
-
- Modifying Topic Partitions on the Console- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
-
- - Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired Kafka instance to view the instance details.
- In the navigation pane, choose Topics.
- Modify the number of partitions using either of the following methods:
- Select one or more topics and click Edit Topic in the upper left corner.
- In the row containing the desired topic, click Edit.
- - In the Edit Topic dialog box, enter the number of partitions and click OK.
- The number of partitions can only be increased.
- To ensure performance, the Kafka console allows a maximum of 200 partitions for each topic.
- The total partitions of all topics cannot exceed the maximum partitions of an instance. The maximum partitions of an instance vary by instance specifications. For details, see Specifications.
-
+
+Notes and Constraints- The number of partitions can only be increased.
- The partition quantity of topics of a single-node or cluster Kafka instance is limited. When the partition quantity limit is reached, you can no longer create topics. The quantity varies with instance specifications. For details, see Cluster Kafka Instances and Single-node Kafka Instances.
- For an instance with ciphertext access enabled, if allow.everyone.if.no.acl.found is set to false, the topic partition quantity can be modified on the client only by the initial user (set in first ciphertext access enablement).
+ Repartitioning a Topic on the Console- Log in to the console.
- Click in the upper left corner to select the region where your instance is located.
- Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired instance to go to the instance details page.
- In the navigation pane, choose Topics.
- Modify the number of partitions using either of the following methods:
- Select one or more topics and click Edit Topic in the upper left corner.
- In the row containing the desired topic, click Edit.
+ - In the Edit Topic dialog box, enter the number of partitions and click OK.
To ensure performance, a maximum of 200 partitions is allowed for each topic on the Kafka console.
+View the partition quantity on the Topics page.
Modifying Topic Partitions on the ClientIf your Kafka client version is later than 2.2, you can use kafka-topics.sh to change the partition quantity.
- For an instance with ciphertext access enabled, if allow.everyone.if.no.acl.found is set to false, topic partition quantity cannot be modified through the client.
-
-
|
|---|
|
|
|---|
|
|---|
|
|---|
|
|---|
|
|---|
|
|---|
|
|---|
|
|---|
|
|---|
|
|---|
|
|---|
|
|---|
|
|---|
|
