yexinru/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
Files
a41a4e0331fbc0beff63bb28fa59228cc9a751df
doc-exports/docs/dms_rocketmq/umn/hrm_ug_046.html
chenjunjie cd96953687 RocketMQ UMN 20250723 version 
Reviewed-by: Liudmila Denisova <ldenisov@noreply.gitea.eco.tsi-dev.otc-service.com>
Co-authored-by: chenjunjie <chenjunjie@huawei.com>
Co-committed-by: chenjunjie <chenjunjie@huawei.com>
2025-09-04 09:57:06 +00:00

26 KiB
Raw Blame History

Migrating RocketMQ Instance Metadata from Others

Migrate metadata from others' RocketMQ to DMS for RocketMQ.

You can use one of the following migration methods as required:

Prerequisites

  • A RocketMQ instance has been created.
  • A Linux host is available, Java Development Kit 1.8.111 or later has been installed on the host, and related environment variables have been configured.
  • A network environment is prepared.

    A RocketMQ instance can be accessed within a VPC or over a public network. For public network access, the producer and consumer must have public access permissions. Configure the security group as follows.

    Table 1 Security group rules

    Direction

    Protocol

    Port

    Source

    Description

    Inbound

    TCP

    8200

    IP address or IP address group of the RocketMQ client

    The port is used for public network access to instances using TCP.

    Inbound

    TCP

    8081

    The port is used for public network access to instances using gRPC.

    Inbound

    TCP

    10101

    The port is used for public access to service nodes using TCP.
  • The client server can access the Internet to download the RocketMQ software package.

Method 1: Run the mqadmin Command to Export the Source Instance Metadata and Then Create a Migration Task in DMS for RocketMQ

Metadata of self-hosted, or another DMS for RocketMQ instance

  1. Log in to the prepared Linux host and download the RocketMQ software package.

    wget https://archive.apache.org/dist/rocketmq/5.3.0/rocketmq-all-5.3.0-bin-release.zip

  2. Decompress the software package.

    unzip rocketmq-all-5.3.0-bin-release.zip

  3. (Optional) If ACL is enabled for the RocketMQ instance, authentication is required when you run the mqadmin command.

    Switch to the directory where the decompressed software package is stored and add the following content to the conf/tools.yml file:
    accessKey:*******
secretKey:*******

    accessKey and secretKey are the username and secret key set on the Users page of the console. For details, see Creating a User.

  4. Go to the directory where the decompressed software package is stored and run the following command to query the cluster name:

    • If SSL is disabled, run the following command:
      sh ./bin/mqadmin clusterList -n {nameserver address and port}

      For example, if the nameserver address and port number are 192.168.0.65:8100, run the following command:

      sh ./bin/mqadmin clusterList -n 192.168.0.65:8100
    • If SSL is enabled, run the following command:
      JAVA_OPT=-Dtls.enable=true sh ./bin/mqadmin clusterList -n {nameserver address and port}

      For example, if the nameserver address and port are 192.168.0.65:8100, run the following command:

      JAVA_OPT=-Dtls.enable=true sh ./bin/mqadmin clusterList -n 192.168.0.65:8100

  5. Run the following command to export metadata:

    • If SSL is disabled, run the following command:
      sh ./bin/mqadmin exportMetadata -n {nameserver address and port number} -c {RocketMQ cluster name} -f {Path for storing the exported metadata file}

      For example, if the nameserver address and port number are 192.168.0.65:8100, the RocketMQ cluster name is DmsCluster, and the path for storing exported metadata files is /tmp/rocketmq/export, run the following command:

      sh ./bin/mqadmin exportMetadata -n 192.168.0.65:8100 -c DmsCluster -f /tmp/rocketmq/export
    • If SSL is enabled, run the following command:
      JAVA_OPT=-Dtls.enable=true sh ./bin/mqadmin exportMetadata -n {nameserver address and port number} -c {RocketMQ cluster name} -f {path for storing the exported metadata file}

      For example, if the nameserver address and port number are 192.168.0.65:8100, the RocketMQ cluster name is DmsCluster, and the path for storing exported metadata files is /tmp/rocketmq/export, run the following command:

      JAVA_OPT=-Dtls.enable=true sh ./bin/mqadmin exportMetadata -n 192.168.0.65:8100 -c DmsCluster -f /tmp/rocketmq/export

Migrate metadata on the console.

  1. Log in to the console.
  2. Click in the upper left corner to select a region.

    DMS for RocketMQ instances in different regions cannot communicate with each other over an intranet. Select a nearest location for low latency and fast access.

  3. Click and choose Application > Distributed Message Service for RocketMQ to open the console of DMS for RocketMQ.
  4. Click a RocketMQ instance to go to the instance details page.
  5. In the navigation pane, choose Cloud Migration > Metadata Migration.
  6. Click Create Migration Task.
  7. Configure the migration task by referring to Table 2.

    Table 2 Migration task parameters

    Parameter

    Description

    Task Type

    Select From self-built RocketMQ to cloud.

    Task Name

    Unique name of the migration task.

    A task name must meet the following requirements:
    • Contains 4 to 64 characters.
    • Contains only letters, digits, hyphens (-), and underscores (_), and must start with a letter.

    Overwrite
    • If this option is enabled, configurations in the metadata file with the same name as the uploaded file will be modified.

      Assume that Topic01 on the source instance has three read queues, and Topic01 on the DMS instance has two read queues. If Overwrite is enabled, Topic01 on the DMS instance will have three read queues after migration.

    • If this option is disabled, migration of the metadata file with the same name as the uploaded file will fail.

      Assume that the source instance has Topic01 and Topic02, and the DMS instance has Topic01 and Topic03. If Overwrite is disabled, migration of the source Topic01 will fail.

    Metadata

    See 5.

  8. Click OK.

    After the migration is complete, view Task Status in the migration task list.

    • If Task Status is Complete, all metadata has been successfully migrated.

      Produce and consume messages by referring to or to ensure that metadata is available.

    • If Task Status is Failed, some or all metadata fails to be migrated. Click the migration task name to go to the migration task details page. In the Migration Result area, view the name of the topic or consumer group that fails to be migrated, the failure cause, and the solution. After the fault is rectified, perform the following steps.

  9. Migrate the production service to the RocketMQ instance.

    Change the metadata connection address on the production client to the metadata connection address of the RocketMQ instance and then restart the production service. New messages will be sent to the RocketMQ instance.

  10. Migrate the consumption service to the RocketMQ instance.

    After all messages in the consumer group are consumed, change the metadata connection address of the consumer client to the metadata connection address of the RocketMQ instance. New messages will be consumed from the RocketMQ instance.

  11. If there are multiple source RocketMQ instances, migrate services from them one by one by referring to 1 to 10.

Method 2: Export the Source Topics and Consumer Groups and Import Them to DMS for RocketMQ Using Scripts

Metadata of others', self-hosted, or another DMS for RocketMQ instance

  1. Log in to the console of another vendor and export the lists of source topics and consumer groups.
  2. Create the topics.txt and groups.txt files and add the source topic list and consumer group list to the files respectively. Each line contains a topic or consumer group name. For example:

    topic-01
topic-02
...
topic-n

    The groups.txt file cannot contain blank lines (for example, a newline character at the end of a consumer group name). Otherwise, consumer groups with empty names will be created when the lists are imported to the RocketMQ instance.

Import the source topics and consumer groups to DMS for RocketMQ using the following script:

  1. Log in to the prepared Linux host and download the RocketMQ software package.

    wget https://archive.apache.org/dist/rocketmq/5.3.0/rocketmq-all-5.3.0-bin-release.zip

  2. Decompress the software package.

    unzip rocketmq-all-5.3.0-bin-release.zip

  3. (Optional) If ACL is enabled for the RocketMQ instance, authentication is required when you run the mqadmin command.

    Switch to the directory where the decompressed software package is stored and add the following content to the conf/tools.yml file:
    accessKey:*******
secretKey:*******

    accessKey and secretKey are the username and secret key set on the Users page of the console.

  4. Go to the bin directory of the decompressed software package and upload topics.txt and groups.txt to this directory.
  5. Run the following script to import the source topics and consumer groups to DMS for RocketMQ:

    #!/bin/bash

# Read groups from groups.txt file
groups=()
while read -r group; do
  groups+=("$group")
done < "groups.txt"

# Read topics from topic.txt file
topics=()
while read -r topic; do
  topics+=("$topic")
done < "topics.txt"

# Add topics
for topic in "${topics[@]}"; do
  echo "Adding topic: $topic"
  sh mqadmin updateTopic -n <namesrvIp:8100> -c DmsCluster -t "$topic"
done

# Add consumer groups
for group in "${groups[@]}"; do
  echo "Adding consumer group: $group"
  sh mqadmin updateSubGroup -n <namesrvIp:8100> -c DmsCluster -g "$group"
done

    namesrvIp:8100 indicates the address of the RocketMQ instance.

  6. Log in to the console.
  7. Click in the upper left corner to select a region.

    DMS for RocketMQ instances in different regions cannot communicate with each other over an intranet. Select a nearest location for low latency and fast access.

  8. Click and choose Application > Distributed Message Service for RocketMQ to open the console of DMS for RocketMQ.
  9. Click a RocketMQ instance to go to the instance details page.
  10. In the navigation pane, choose Cloud Migration > Metadata Migration.
  11. Click the task name to go to the task details page. Migration Result shows whether the topic and consumer group list are imported.

    Produce and consume messages by referring to or to ensure that metadata is available.

  12. Migrate the production service to the RocketMQ instance.

    Change the metadata connection address on the production client to the metadata connection address of the RocketMQ instance and then restart the production service. New messages will be sent to the RocketMQ instance.

  13. Migrate the consumption service to the RocketMQ instance.

    After all messages in the consumer group are consumed, change the metadata connection address of the consumer client to the metadata connection address of the RocketMQ instance. New messages will be consumed from the RocketMQ instance.

Parent topic: Migrating Metadata