Reassigning Kafka Partitions

Operation Impact

• Partition reassignment on topics with a large amount of data consumes a large amount of network and storage bandwidth. As a result, service requests may time out or the latency may increase. Therefore, you are advised to perform reassignment during off-peak hours. Compare the current instance load based on the instance specifications to decide whether the remaining instance capacity can support partition reassignment. Do not reassign partitions when there is insufficient bandwidth or when the CPU usage is greater than 90%. To view data volume and CPU usage of a topic, see Message Size and CPU Usage on the monitoring page. For details, see Viewing Kafka Metrics.
• A throttle refers to the upper limit of the bandwidth for replication of a topic, to ensure that other topics on the instance are not affected. Note that throttles apply to replication triggered by both normal message production and partition reassignment. If the throttle is too small, normal message production may be affected, and partition reassignment may never complete. If partitions are continuously reassigned, contact customer service.
• You cannot delete topics whose reassignment tasks have started. Otherwise, the tasks will never complete.
• You cannot modify the partition quantity of topics whose reassignment tasks have started.
• Reassignment tasks cannot be manually stopped. Please wait until they complete.
• If partition reassignment has been scheduled, reassignment cannot be scheduled again for any topic in this instance until this reassignment is executed.
• After partition reassignment, the metadata of the topic changes. If the producer does not support the retry mechanism, a few requests will fail, causing some messages to fail to be produced.
• Reassignment takes a long time if the topic has a large amount of data. You are advised to decrease the topic retention period based on the topic consumption so that historical data of the topic can be deleted in a timely manner to accelerate the migration. To view data volume of a topic, see Message Size on the monitoring page. For details, see Viewing Kafka Metrics.

Preparing for Partition Reassignment

• To reduce the amount of data to be migrated, decrease the topic retention period without affecting services and wait for messages to age. After the reassignment is complete, you can restore the retention period. To change the topic retention period, see Changing Kafka Message Retention Period.
• The target broker should have sufficient disk space. To check available disk space of each broker, see Viewing Kafka Disk Usage. If the remaining disk capacity of the target broker is close to the amount of data to be migrated to the broker, expand the disk capacity before the reassignment.

Auto Reassignment

1. Log in to the console.
2. Click in the upper left corner to select a region.
   

   Select the region where your Kafka instance is located.

3. Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
4. Click the desired Kafka instance to view the instance details.
5. In the navigation pane, choose Topics.
6. Reassign partitions using either of the following methods:
   • Select one or more topics and choose Reassign > Auto above the topic list.
   • In the row that contains the desired topic, choose More > Reassign > Auto.

7. Set automatic reassignment parameters.
   • In the Brokers area, select the brokers to assign the topic's partition replicas to.
   • In the Topics area, enter the number of replicas to be automatically reassigned. The number of replicas must be less than or equal to the number of brokers.
   • Specify throttle. The default value is -1, indicating that there is no throttle If the instance has low workload (for example, only 30/300 MB/s is used), you are not advised to limit the bandwidth. If a throttle is required, you are advised to set it to a value greater than or equal to the total production bandwidth of the to-be-reassigned topic multiplied by the maximum number of replicas of the to-be-reassigned topic. For details, see Calculating a Throttle.
   • For Execute, specify when to execute the reassignment. Now means to execute it immediately. As scheduled means to execute it at the scheduled time.
   Figure 1 Setting automatic reassignment parameters
   

8. (Optional) Click Calculate. Time Required indicates how long automatic balancing will take.
9. Click OK.

   The following table lists how to check whether reassignment is complete (scheduled and non-scheduled tasks):

   Table 1 Checking the reassignment result

   Task Type	Reassignment Result
   Background tasks	In the upper left corner of the topic list, click View details and the Background Tasks > Background tasks page is displayed. The reassignment task is complete when it is in the Successful state.
   Scheduled tasks	The Background Tasks > Scheduled tasks page is displayed. This page only shows whether scheduled tasks start to execute instead of whether they are successful. When the task status is Pending, reassignment has not been executed. When the task status is Successful, reassignment has started. When the task status is Cancel, reassignment has been canceled. Click Background tasks tab page. When the task status is Successful, reassignment has completed.

   Figure 2 Background Tasks page
   
   

   • You cannot delete topics whose reassignment tasks have started. Otherwise, the tasks will never complete.
   • You cannot modify the partition quantity of topics whose reassignment tasks have started.
   • Reassignment tasks cannot be manually stopped. Please wait until they complete.
   • If partition reassignment has been scheduled, reassignment cannot be scheduled again for any topic in this instance until this reassignment is executed.

Manual Reassignment

1. Log in to the console.
2. Click in the upper left corner to select a region.
   

   Select the region where your Kafka instance is located.

3. Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
4. Click the desired Kafka instance to view the instance details.
5. In the navigation pane, choose Topics.
6. Reassign partitions using either of the following methods:
   • Select a topic and choose Reassign > Manual above the topic list. Manual reassignment does not support batch operations.
   • In the row that contains the desired topic, choose More > Reassign > Manual.

7. Set manual reassignment parameters.
   • In the upper right corner of the Manual dialog box, click Delete Replica or Add Replica to reduce or increase the number of replicas for each partition of the topic.
   • Under the name of the replica to be reassigned, click the broker name or and select the target broker to migrate the replica to. Assign replicas of the same partition to different brokers.
   • Specify throttle. The default value is -1, indicating that there is no throttle If the instance has low workload (for example, only 30/300 MB/s is used), you are not advised to limit the bandwidth. If a throttle is required, you are advised to set it to a value greater than or equal to the total production bandwidth of the to-be-reassigned topic multiplied by the maximum number of replicas of the to-be-reassigned topic. For details, see Calculating a Throttle.
   • For Execute, specify when to execute the reassignment. Now means to execute it immediately. As scheduled means to execute it at the scheduled time.
   Figure 3 Setting manual reassignment parameters
   

8. (Optional) Click Calculate. Time Required indicates how long manual balancing will take.
9. Click OK.

   The following table lists how to check whether reassignment is complete (scheduled and non-scheduled tasks):

   Table 2 Checking the reassignment result

   Task Type	Reassignment Result
   Background tasks	In the upper left corner of the topic list, click View details and the Background Tasks > Background tasks page is displayed. The reassignment task is complete when it is in the Successful state.
   Scheduled tasks	The Background Tasks > Scheduled tasks page is displayed. This page only shows whether scheduled tasks start to execute instead of whether they are successful. When the task status is Pending, reassignment has not been executed. When the task status is Successful, reassignment has started. When the task status is Cancel, reassignment has been canceled. Click Background tasks tab page. When the task status is Successful, reassignment has completed.

   Figure 4 Background Tasks page
   
   

   • You cannot delete topics whose reassignment tasks have started. Otherwise, the tasks will never complete.
   • You cannot modify the partition quantity of topics whose reassignment tasks have started.
   • Reassignment tasks cannot be manually stopped. Please wait until they complete.
   • If partition reassignment has been scheduled, reassignment cannot be scheduled again for any topic in this instance until this reassignment is executed.

Re-scheduling Partition Reassignment

1. On the Scheduled tasks tab page on the Background Tasks page, click the drop-down box in the upper left corner, select a time period, enter the desired topic name in the search box, and press Enter.
   Figure 5 Querying reassignment schedules
   

2. In the row that contains the desired task, click Modify.
3. In the Change Schedule dialog box, change the schedule or cancel the scheduled task.
   • To change the schedule, select a time and click OK.
   • To cancel the task, select Cancel (as shown in Figure 6) and click OK.
     Figure 6 Canceling a scheduled reassignment task
     

Calculating a Throttle

Throttles are affected by the execution duration of the reassignment, leader/follower distribution of partition replicas, and message production rate.

• A throttle limits the replication traffic of all partitions in a broker.
• Replicas added after the assignment are regarded as followers, and existing replicas are regarded as leaders. Throttles on leaders and followers are separated.
• Throttles do not distinguish between replication caused by normal message production and that caused by partition reassignment. Therefore, the traffic generated in both cases is throttled.

Assume that the partition reassignment task needs to be completed within 200s and each replica has 100 MB data. Calculate the throttle in the following scenarios:

Scenario 1: Topic 1 has two partitions and two replicas, and Topic 2 has one partition and one replica. All leader replicas are on the same broker. One replica needs to be added for Topic 1 and Topic 2 respectively.

Figure 7 Reassignment scenario 1

As shown in Figure 7, three replicas fetch data from Broker 0. Each replica on Broker 0 has 100 MB data. Broker 0 has only leader replicas, and Broker 1 and Broker 2 have only follower replicas.

• Bandwidth required by Broker 0 to complete partition reassignment within 200s = (100 MB + 100 MB + 100 MB)/200s = 1.5 MB/s
• Bandwidth required by Broker 1 to complete partition reassignment within 200s = 100 MB/200s = 0.5 MB/s
• Bandwidth required by Broker 2 to complete partition reassignment within 200s = (100 MB + 100 MB)/200s = 1 MB/s

In conclusion, to complete the partition reassignment task within 200s, set the throttle to a value greater than or equal to 1.5 MB/s. The bandwidth should be set to be greater than or equal to 2 MB/s because the limit on it on the console must be an integer.

Scenario 2: Topic 1 has two partitions and one replica, and Topic 2 has two partitions and one replica. Leader replicas are on different brokers. One replica needs to be added for Topic 1 and Topic 2 respectively.

Figure 8 Reassignment scenario 2