vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/mrs/component-operation-guide/mrs_01_1037.html
Yang, Tong 6182f91ba8 MRS component operation guide_normal 2.0.38.SP20 version 
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
Co-authored-by: Yang, Tong <yangtong2@huawei.com>
Co-committed-by: Yang, Tong <yangtong2@huawei.com>
2022-12-09 14:55:21 +00:00

137 lines
17 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<a name="mrs_01_1037"></a><a name="mrs_01_1037"></a>
<h1 class="topictitle1">Configuring Kafka HA and High Reliability Parameters</h1>
<div id="body1590134593889"><div class="section" id="mrs_01_1037__section4605114719235"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_1037__p853295072316">For the Kafka message transmission assurance mechanism, different parameters are available for meeting different performance and reliability requirements. This section describes how to configure Kafka high availability (HA) and high reliability parameters.</p>
<p id="mrs_01_1037__p1944684185813">This section applies to MRS 3.x or later.</p>
</div>
<div class="section" id="mrs_01_1037__section82251011192415"><h4 class="sectiontitle">Impact on the System</h4><ul id="mrs_01_1037__ul659418222557"><li id="mrs_01_1037__li13184133185618">Impact of HA and high performance configurations:<div class="notice" id="mrs_01_1037__note12467372339"><span class="noticetitle"><img src="public_sys-resources/notice_3.0-en-us.png"> </span><div class="noticebody"><p id="mrs_01_1037__p624614379331">After HA and high performance are configured, the data reliability decreases. Specifically, data may be lost of disks or nodes are faulty.</p>
</div></div>
</li><li id="mrs_01_1037__li9595926195613">Impact of high reliability configurations:<ul id="mrs_01_1037__ul01551542105518"><li id="mrs_01_1037__li3156184212557">Deteriorated performance<p id="mrs_01_1037__p9158164210557"><a name="mrs_01_1037__li3156184212557"></a><a name="li3156184212557"></a>If <strong id="mrs_01_1037__b58766828642342">ack</strong> is set to <strong id="mrs_01_1037__b097418269262">–1</strong>, data written is considered as successful only when data is written to multiple replicas. As a result, the delay of a single message increases and the client processing capability decreases. The impact is subject to the actual test data.</p>
</li><li id="mrs_01_1037__li13159842195517">Reduced availability<p id="mrs_01_1037__p16160442135518"><a name="mrs_01_1037__li13159842195517"></a><a name="li13159842195517"></a>A replica that is not in the ISR list cannot be elected as a leader. If the leader goes offline and other replicas are not in the ISR list, the partition remains unavailable until the leader node recovers. When the node where a replica of a partition is located is faulty, the minimum number of successful replicas cannot be met. As a result, service writing fails.</p>
</li></ul>
</li><li id="mrs_01_1037__li12022151215">If parameters are at the service level, Kafka needs to be restarted. You are advised to modify the service-level configuration in the change window.</li></ul>
</div>
<div class="section" id="mrs_01_1037__section8755824172417"><h4 class="sectiontitle">Parameter Description</h4><p id="mrs_01_1037__p683811247427"></p>
<ul id="mrs_01_1037__ul4795143184314"><li id="mrs_01_1037__li24114814181">If services require high availability and high performance,<div class="p" id="mrs_01_1037__p54172188187"><a name="mrs_01_1037__li24114814181"></a><a name="li24114814181"></a>set the parameters listed in <a href="#mrs_01_1037__table97904317438">Table 1</a> on the server. For details about the parameter configuration entry, see <a href="mrs_01_2125.html">Modifying Cluster Service Configuration Parameters</a>.
<div class="tablenoborder"><a name="mrs_01_1037__table97904317438"></a><a name="table97904317438"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1037__table97904317438" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Server HA and high performance parameters</caption><thead align="left"><tr id="mrs_01_1037__row47883344310"><th align="left" class="cellrowborder" valign="top" width="26%" id="mcps1.3.3.3.1.1.3.2.4.1.1"><p id="mrs_01_1037__p127881364313">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="10%" id="mcps1.3.3.3.1.1.3.2.4.1.2"><p id="mrs_01_1037__p4788183154315">Default Value</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="64%" id="mcps1.3.3.3.1.1.3.2.4.1.3"><p id="mrs_01_1037__p16788143124315">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_1037__row147893334313"><td class="cellrowborder" valign="top" width="26%" headers="mcps1.3.3.3.1.1.3.2.4.1.1 "><p id="mrs_01_1037__p578816354311">unclean.leader.election.enable</p>
</td>
<td class="cellrowborder" valign="top" width="10%" headers="mcps1.3.3.3.1.1.3.2.4.1.2 "><p id="mrs_01_1037__p1878893164311">true</p>
</td>
<td class="cellrowborder" valign="top" width="64%" headers="mcps1.3.3.3.1.1.3.2.4.1.3 "><p id="mrs_01_1037__p47891531433">Specifies whether a replica that is not in the ISR can be selected as the leader. If this parameter is set to <strong id="mrs_01_1037__b172611602042342">true</strong>, data may be lost.</p>
</td>
</tr>
<tr id="mrs_01_1037__row97894310431"><td class="cellrowborder" valign="top" width="26%" headers="mcps1.3.3.3.1.1.3.2.4.1.1 "><p id="mrs_01_1037__p15789193144315">auto.leader.rebalance.enable</p>
</td>
<td class="cellrowborder" valign="top" width="10%" headers="mcps1.3.3.3.1.1.3.2.4.1.2 "><p id="mrs_01_1037__p1378913104316">true</p>
</td>
<td class="cellrowborder" valign="top" width="64%" headers="mcps1.3.3.3.1.1.3.2.4.1.3 "><p id="mrs_01_1037__p87897310436">Specifies whether the leader automated balancing function is used.</p>
<p id="mrs_01_1037__p478993204313">If this parameter is set to <strong id="mrs_01_1037__b91991258192717">true</strong>, the controller periodically balances the leader of each partition on all nodes and assigns the leader to a replica with a higher priority.</p>
</td>
</tr>
<tr id="mrs_01_1037__row137901939430"><td class="cellrowborder" valign="top" width="26%" headers="mcps1.3.3.3.1.1.3.2.4.1.1 "><p id="mrs_01_1037__p1179020319438">min.insync.replicas</p>
</td>
<td class="cellrowborder" valign="top" width="10%" headers="mcps1.3.3.3.1.1.3.2.4.1.2 "><p id="mrs_01_1037__p37901833431">1</p>
</td>
<td class="cellrowborder" valign="top" width="64%" headers="mcps1.3.3.3.1.1.3.2.4.1.3 "><p id="mrs_01_1037__p379023104318">Specifies the minimum number of replicas to which data is written when <strong id="mrs_01_1037__b35981548542342">acks</strong> is set to <strong id="mrs_01_1037__b173008488242342">-1</strong> for the Producer.</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<p id="mrs_01_1037__p109515341697">Set the parameters listed in <a href="#mrs_01_1037__table337919499593">Table 2</a> in the client configuration file <strong id="mrs_01_1037__b168902365119">producer.properties</strong>. The path for storing <strong id="mrs_01_1037__b65061128145114">producer.properties</strong> is /<strong id="mrs_01_1037__b196221651182919">opt</strong><strong id="mrs_01_1037__b26221751172913"></strong><strong id="mrs_01_1037__b862218516296">/client/Kafka/kafka/config/producer.properties</strong>, where <strong id="mrs_01_1037__b388417913018">/opt</strong><strong id="mrs_01_1037__b20884498304"></strong><strong id="mrs_01_1037__b18884599308">/client</strong> indicates the installation directory of the Kafka client.</p>
<div class="tablenoborder"><a name="mrs_01_1037__table337919499593"></a><a name="table337919499593"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1037__table337919499593" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Client HA and high performance parameters</caption><thead align="left"><tr id="mrs_01_1037__row1038014497593"><th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.3.3.1.3.2.4.1.1"><p id="mrs_01_1037__p136922014503">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.3.3.1.3.2.4.1.2"><p id="mrs_01_1037__p1693914904">Default Value</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.3.3.1.3.2.4.1.3"><p id="mrs_01_1037__p069313141805">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_1037__row938044912597"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.3.3.1.3.2.4.1.1 "><p id="mrs_01_1037__p432736202">acks</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.3.3.1.3.2.4.1.2 "><p id="mrs_01_1037__p9327160010">1</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.3.3.1.3.2.4.1.3 "><p id="mrs_01_1037__p113271268017">The leader needs to check whether the message has been received and determine whether the required operation has been processed. This parameter affects message reliability and performance.</p>
<ul id="mrs_01_1037__ul16327116802"><li id="mrs_01_1037__li23271265015">If this parameter is set to <strong id="mrs_01_1037__b203014382842342">0</strong>, the producer does not wait for any response from the server, and the message is considered successful.</li><li id="mrs_01_1037__li4327866015">If this parameter is set to <strong id="mrs_01_1037__b27448186742342">1</strong>, when the leader of the replica verifies that data has been written into the cluster, the leader returns a response without waiting for data to be written to all replicas. In this case, if the leader is abnormal when the leader makes the confirmation but replica synchronization is not complete, data will be lost.</li><li id="mrs_01_1037__li63275611020">If this parameter is set to <strong id="mrs_01_1037__b383117495524">-1</strong>, the message is considered to be successfully received only when all synchronized replicas are confirmed. If the <span class="parmname" id="mrs_01_1037__parmname53271161102"><b>min.insync.replicas</b></span> parameter is also configured, data can be written into multiple replicas. In this case, records will not be lost as long as one replica remains active.</li></ul>
</td>
</tr>
</tbody>
</table>
</div>
</li><li id="mrs_01_1037__li10626184012194">To ensure high data reliability for services,<div class="p" id="mrs_01_1037__p3594171616215"><a name="mrs_01_1037__li10626184012194"></a><a name="li10626184012194"></a>set the parameters listed in <a href="#mrs_01_1037__table1779493174311">Table 3</a> on the server. For details about the parameter configuration entry, see <a href="mrs_01_2125.html">Modifying Cluster Service Configuration Parameters</a>.
<div class="tablenoborder"><a name="mrs_01_1037__table1779493174311"></a><a name="table1779493174311"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1037__table1779493174311" frame="border" border="1" rules="all"><caption><b>Table 3 </b>Server HA parameters</caption><thead align="left"><tr id="mrs_01_1037__row1179111310434"><th align="left" class="cellrowborder" valign="top" width="26%" id="mcps1.3.3.3.2.1.3.2.4.1.1"><p id="mrs_01_1037__p17911232439">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="10%" id="mcps1.3.3.3.2.1.3.2.4.1.2"><p id="mrs_01_1037__p779153174317">Recommended Value</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="64%" id="mcps1.3.3.3.2.1.3.2.4.1.3"><p id="mrs_01_1037__p8791123194313">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_1037__row679111311437"><td class="cellrowborder" valign="top" width="26%" headers="mcps1.3.3.3.2.1.3.2.4.1.1 "><p id="mrs_01_1037__p167913354317">unclean.leader.election.enable</p>
</td>
<td class="cellrowborder" valign="top" width="10%" headers="mcps1.3.3.3.2.1.3.2.4.1.2 "><p id="mrs_01_1037__p27919315433">false</p>
</td>
<td class="cellrowborder" valign="top" width="64%" headers="mcps1.3.3.3.2.1.3.2.4.1.3 "><p id="mrs_01_1037__p207917319434">A replica that is not in the ISR list cannot be elected as a leader.</p>
</td>
</tr>
<tr id="mrs_01_1037__row679219324315"><td class="cellrowborder" valign="top" width="26%" headers="mcps1.3.3.3.2.1.3.2.4.1.1 "><p id="mrs_01_1037__p16792123194320">min.insync.replicas</p>
</td>
<td class="cellrowborder" valign="top" width="10%" headers="mcps1.3.3.3.2.1.3.2.4.1.2 "><p id="mrs_01_1037__p079216318433">2</p>
</td>
<td class="cellrowborder" valign="top" width="64%" headers="mcps1.3.3.3.2.1.3.2.4.1.3 "><p id="mrs_01_1037__p107921316433">Specifies the minimum number of replicas to which data is written when <strong id="mrs_01_1037__b46170830842342">acks</strong> is set to <strong id="mrs_01_1037__b56729317942342">-1</strong> for the Producer.</p>
<p id="mrs_01_1037__p37928344318">Ensure that the value of <strong id="mrs_01_1037__b41181864942342">min.insync.replicas</strong> is equal to or less than that of <strong id="mrs_01_1037__b210620680542342">replication.factor</strong>.</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<p id="mrs_01_1037__p8407834112418">Set the parameters listed in <a href="#mrs_01_1037__table22137713255">Table 4</a> in the client configuration file <strong id="mrs_01_1037__b1449848145516">producer.properties</strong>. The path for storing <strong id="mrs_01_1037__b14498198205513">producer.properties</strong> is <strong id="mrs_01_1037__b1093813490300">/opt</strong><strong id="mrs_01_1037__b11938649113018"></strong><strong id="mrs_01_1037__b20938134943014">/client/Kafka/kafka/config/producer.properties</strong>, where <strong id="mrs_01_1037__b763720193111">/opt</strong><strong id="mrs_01_1037__b86376183119"></strong><strong id="mrs_01_1037__b196371315313">/client</strong> indicates the installation directory of the Kafka client.</p>
<div class="tablenoborder"><a name="mrs_01_1037__table22137713255"></a><a name="table22137713255"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1037__table22137713255" frame="border" border="1" rules="all"><caption><b>Table 4 </b>Server HA parameters</caption><thead align="left"><tr id="mrs_01_1037__row14213179258"><th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.3.3.2.3.2.4.1.1"><p id="mrs_01_1037__p10917143910257">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.3.3.2.3.2.4.1.2"><p id="mrs_01_1037__p1991712395259">Recommended Value</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.3.3.2.3.2.4.1.3"><p id="mrs_01_1037__p891753915257">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_1037__row192149713259"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.3.3.2.3.2.4.1.1 "><p id="mrs_01_1037__p1997943012257">acks</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.3.3.2.3.2.4.1.2 "><p id="mrs_01_1037__p597943018259">-1</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.3.3.2.3.2.4.1.3 "><p id="mrs_01_1037__p1979193012520">The leader needs to check whether the message has been received and determine whether the required operation has been processed.</p>
<p id="mrs_01_1037__p29798306259">If this parameter is set to <strong id="mrs_01_1037__b145035423942342">-1</strong>, the message is considered to be successfully received only when all replicas in the ISR list have confirmed to receive the message. This parameter is used along with <span class="parmname" id="mrs_01_1037__parmname8979163012250"><b>min.insync.replicas</b></span> to ensure that multiple copies are successfully written. As long as one copy is active, the record will not be lost. If this parameter is set to <strong id="mrs_01_1037__b101053400555">-1</strong>, the production performance deteriorates. Therefore, you need to set this parameter based on the actual situation.</p>
</td>
</tr>
</tbody>
</table>
</div>
</li></ul>
</div>
<div class="section" id="mrs_01_1037__section9377162414465"><h4 class="sectiontitle">Configuration Suggestions</h4><p id="mrs_01_1037__p19856345466">Configure parameters based on requirements on reliability and performance in the following service scenarios:</p>
<ul id="mrs_01_1037__ul45061357164616"><li id="mrs_01_1037__li1850635764616">For valued data, you are advised to configure RAID1 or RAID5 for Kafka data directory disks to improve data reliability when a single disk is faulty.</li><li id="mrs_01_1037__li735281413473">For parameters that can be modified at the topic level, the service level configurations are used by default.<p id="mrs_01_1037__p698359464"><a name="mrs_01_1037__li735281413473"></a><a name="li735281413473"></a>These parameters can be separately configured based on topic reliability requirements. For example, log in to the Kafka client as user <strong id="mrs_01_1037__b12254195853215">root</strong>, and run the following command to configure the reliability parameter with topic named test in the client installation directory:</p>
<p id="mrs_01_1037__p694925013378"><strong id="mrs_01_1037__b794945014376">cd Kafka/kafka/bin</strong></p>
<p id="mrs_01_1037__p20949350133712"><strong id="mrs_01_1037__b169491350103715">kafka-topics.sh --zookeeper 192.168.1.205:2181/kafka --alter --topic test --config unclean.leader.election.enable=false --config min.insync.replicas=2</strong></p>
<p id="mrs_01_1037__p10131635194618"><strong id="mrs_01_1037__b133614187142342">192.168.1.205</strong> indicates the ZooKeeper service IP address.</p>
</li></ul>
<ul id="mrs_01_1037__ul18131182215472"><li id="mrs_01_1037__li1713112234710">If parameters are at the service level, Kafka needs to be restarted. You are advised to modify the service-level configuration in the change window.</li></ul>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0375.html">Using Kafka</a></div>
</div>
</div>
View Git Blame Copy Permalink