forked from docs/doc-exports
zhengxiu
93d856d5c5
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: zhengxiu <zhengxiu@huawei.com> Co-committed-by: zhengxiu <zhengxiu@huawei.com>
93 lines
22 KiB
HTML
93 lines
22 KiB
HTML
<a name="EN-US_TOPIC_0000001938377520"></a><a name="EN-US_TOPIC_0000001938377520"></a>
|
|
|
|
<h1 class="topictitle1">Switching AZs for an Elasticsearch Cluster</h1>
|
|
<div id="body0000001438549444"><p id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_p67196974819">CSS supports cross-AZ deployment. By switching AZs for a cluster, you can make more resources available to the cluster or improve the cluster's availability.</p>
|
|
<div class="p" id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_p184312303171">You can <strong id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_b12888619193013">Add AZ</strong> or <strong id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_b17504133310308">Migrate AZ</strong>.<ul id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_ul04225613559"><li id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_li1642135611554"><strong id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_b1421516153117">Add AZ</strong>: Add one or two AZs for a single-AZ cluster, or add one AZ for a dual-AZ cluster to enhance cluster availability.</li><li id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_li288110413110"><strong id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_b470415993115">Migrate AZ</strong>: Migrate data from the current AZ to another AZ that has sufficient resources for the cluster.</li></ul>
|
|
</div>
|
|
<div class="p" id="EN-US_TOPIC_0000001938377520__p164548389129">The AZ change process is as follows:<ol id="EN-US_TOPIC_0000001938377520__ol20517163781210"><li id="EN-US_TOPIC_0000001938377520__li1951673771215"><p id="EN-US_TOPIC_0000001938377520__p180717562128"><a name="EN-US_TOPIC_0000001938377520__li1951673771215"></a><a name="li1951673771215"></a>Migrate data from one node to other available nodes.</p>
|
|
</li><li id="EN-US_TOPIC_0000001938377520__li6787180143">Rebuild the node for the target AZ. Keep the node ID, IP address, and specifications unchanged.</li><li id="EN-US_TOPIC_0000001938377520__li5735161913148">Add the new node into the cluster. The system automatically triggers a shard reallocation, migrating some of the shards to the new node. Then repeat these steps on all the other nodes.</li></ol>
|
|
</div>
|
|
<p id="EN-US_TOPIC_0000001938377520__p448412286149">The nodes are changed one at a time to prevent service interruptions.</p>
|
|
<div class="section" id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_section5831519171516"><h4 class="sectiontitle">Constraints</h4><ul id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_ul11656830101213"><li id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_li18656830171214">To ensure service continuity, the total number of data nodes plus cold data nodes in the cluster must be at least three.</li><li id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_li11171172161312">During the change, nodes are brought offline one by one and then new nodes are created. Ensure that the total disk capacity of the remaining nodes is sufficient for handling all the data of the cluster, and that the nodes' disk usage stays below 80%.</li><li id="EN-US_TOPIC_0000001938377520__li19930831191612">To make sure that all index shards in the cluster can be allocated to the remaining data nodes and cold data nodes, the <strong id="EN-US_TOPIC_0000001938377520__b73874237124">total number of data nodes and cold data nodes must be greater than the maximum number of index replicas plus 1</strong>.</li><li id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_li2553635174511">Before a change completes, some nodes may have already been moved to a new AZ. In this case, the AZs before and after the change are both displayed. After the change succeeds, the new AZs and their nodes will be displayed properly.</li><li id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_li14591175318713">When you add nodes for a cluster (<strong id="EN-US_TOPIC_0000001938377520__b45894549142">Add AZ</strong>), the current AZ must be retained. When adding one or two AZs for a single-AZ cluster, you must change AZs for all nodes. When adding one AZ for a dual-AZ cluster, you can choose to change AZs for specific types of nodes or all nodes in the cluster. For example, for a dual-AZ cluster, you can use three AZs for its master nodes, while still using two AZs for the rest of its nodes. To complete this AZ change, the system will try to move as few nodes as possible to rebuild the cluster. During this process, the YML configuration file of the nodes that are not modified is also updated. You need to restart the cluster to make the change take effect.</li><li id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_li18603846193211">When migrating an AZ (<strong id="EN-US_TOPIC_0000001938377520__b176451835172413">Migrate AZ</strong>), you can select only one target AZ. For this operation, you can choose to migrate only specific types of nodes or all nodes in the cluster. For example, for a dual-AZ cluster, you may move the master nodes from one AZ to the other, while keeping the rest of the nodes unchanged. If you migrate from one AZ to another, there is no need to restart the cluster. If multiple AZs are migrated, you must restart the cluster to make the change take effect.</li></ul>
|
|
</div>
|
|
<div class="section" id="EN-US_TOPIC_0000001938377520__section1581211581159"><h4 class="sectiontitle">Change Impact</h4><p id="EN-US_TOPIC_0000001938377520__p6910133541820">Before changing a cluster's AZs, it is essential to assess the potential impacts and review operational recommendations. This enables proper scheduling of the change, minimizing service interruptions.</p>
|
|
<ul id="EN-US_TOPIC_0000001938377520__ul1499813106326"><li id="EN-US_TOPIC_0000001938377520__li1999951063217">Performance<p id="EN-US_TOPIC_0000001938377520__p12755222133911"><a name="EN-US_TOPIC_0000001938377520__li1999951063217"></a><a name="li1999951063217"></a>Changing AZs for a cluster does not interrupt services. However, data migration that occurs during this process consumes I/O performance, and taking individual nodes offline still has some impact on the overall cluster performance.</p>
|
|
<div class="p" id="EN-US_TOPIC_0000001938377520__p16986568357">To minimize this impact, it is advisable to adjust the data migration rate based on the cluster's traffic cycle: increase the data migration rate during off-peak hours to shorten the task duration, and decrease it <strong id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001965496905_b7158153214314">before</strong> peak hours arrive to ensure optimal cluster performance. The data migration rate is determined by the <span class="parmname" id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001965496905_parmname525325404618"><b>indices.recovery.max_bytes_per_sec</b></span> parameter. The default value of this parameter is the number of vCPUs multiplied by 32 MB. For example, for four vCPUs, the data migration rate is 128 MB. Set this parameter to a value between 40 MB and 1000 MB based on your service requirements.<pre class="screen" id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001965496905_screen67611747479">PUT /_cluster/settings
|
|
{
|
|
"transient": {
|
|
"indices.recovery.max_bytes_per_sec": "1000MB"
|
|
}
|
|
}</pre>
|
|
</div>
|
|
</li><li id="EN-US_TOPIC_0000001938377520__li643812278112">Impact on request handling<p id="EN-US_TOPIC_0000001938377520__p163869181212"><a name="EN-US_TOPIC_0000001938377520__li643812278112"></a><a name="li643812278112"></a>While a node is offline, requests sent to it may fail. To mitigate this impact, the following measures may be taken:</p>
|
|
<ul id="EN-US_TOPIC_0000001938377520__ul2415022526"><li id="EN-US_TOPIC_0000001938377520__li4662191251411">Use a VPC endpoint or a dedicated load balancer to handle access requests to your cluster, which makes sure that requests are automatically routed to available nodes.</li><li id="EN-US_TOPIC_0000001938377520__li56621012111410">Enable an exponential backoff & retry mechanism on the client (configure three retries).</li><li id="EN-US_TOPIC_0000001938377520__li174152255211">Perform this operation during off-peak hours.</li></ul>
|
|
</li><li id="EN-US_TOPIC_0000001938377520__li1697691517153">Characteristics of this process<p id="EN-US_TOPIC_0000001938377520__p1297941510158"><a name="EN-US_TOPIC_0000001938377520__li1697691517153"></a><a name="li1697691517153"></a>Once started, an AZ change task cannot be stopped until it succeeds or fails.</p>
|
|
</li></ul>
|
|
</div>
|
|
<div class="section" id="EN-US_TOPIC_0000001938377520__section1147814313341"><h4 class="sectiontitle">Change Duration</h4><p id="EN-US_TOPIC_0000001938377520__p176511815124816">The following formula can be used to estimate how long it will take to change AZs for a cluster:</p>
|
|
<p id="EN-US_TOPIC_0000001938377520__p2382153812219"><strong id="EN-US_TOPIC_0000001938377520__b10340534143116">Change duration (min) = 15 (min) x Total number of nodes involved + Data migration duration (min)</strong></p>
|
|
<p id="EN-US_TOPIC_0000001938377520__p13772163437">where, 15 minutes indicates how long non-data migration operations (e.g., initialization) typically take per node. It is an empirical value.</p>
|
|
<p id="EN-US_TOPIC_0000001938377520__p165262364916"><strong id="EN-US_TOPIC_0000001938377520__b1244886698114513">Data migration duration (min) = Total data size of the nodes to be migrated (MB) ÷ [Total number of vCPUs of the data nodes x 32 (MB/s) x 60 (s)]</strong></p>
|
|
<div class="p" id="EN-US_TOPIC_0000001938377520__p45261339497">where,<ul id="EN-US_TOPIC_0000001938377520__ul10582458124811"><li id="EN-US_TOPIC_0000001938377520__li1858235894812">32 MB/s indicates that each vCPU can process 32 MB of data per second. It is an empirical value.</li><li id="EN-US_TOPIC_0000001938377520__li4582458174816">The formulas above use estimates under ideal conditions. The actual migration speed depends on cluster load.</li></ul>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="EN-US_TOPIC_0000001938377520__section13834181516910"><h4 class="sectiontitle">Prerequisites</h4><ul id="EN-US_TOPIC_0000001938377520__ul48347151793"><li id="EN-US_TOPIC_0000001938377520__li20834315793">The cluster status is <strong id="EN-US_TOPIC_0000001938377520__b1797620419316">Available</strong>, and there are no ongoing tasks.</li><li id="EN-US_TOPIC_0000001938377520__li1083411514916">Make sure that the cluster has not had any non-standard modifications, such as the configuration of custom return routes and system parameters. Such modifications, if ever made, will be lost after the AZ change, which may affect your services.</li><li id="EN-US_TOPIC_0000001938377520__li392016144316">All mission-critical data has been backed up. For details, see <a href="css_01_0267.html">Creating Snapshots to Back Up the Data of an Elasticsearch Cluster</a>.</li></ul>
|
|
</div>
|
|
<div class="section" id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_section8588923151510"><h4 class="sectiontitle">Changing AZs</h4><ol id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_ol373518361145"><li id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_li1173513611411"><span id="EN-US_TOPIC_0000001938377520__ph179116118620">Log in to the CSS management console.</span></li><li id="EN-US_TOPIC_0000001938377520__li4250043448">In the navigation pane on the left, choose <span class="uicontrol" id="EN-US_TOPIC_0000001938377520__uicontrol31668037685139"><b>Clusters > Elasticsearch</b></span>.</li><li id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001285669680_li17735636141417">In the cluster list, find the target cluster, and choose <strong id="EN-US_TOPIC_0000001938377520__b476896113419">More</strong> > <strong id="EN-US_TOPIC_0000001938377520__b167699623411">Modify Configuration</strong> in the <strong id="EN-US_TOPIC_0000001938377520__b16769967343">Operation</strong> column. The <strong id="EN-US_TOPIC_0000001938377520__b15769762342">Modify Configuration</strong> page is displayed.</li><li id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_li239201117417">Click the <span class="uicontrol" id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_uicontrol163971144111"><b>Change AZ</b></span> tab.</li><li id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_li13735736141416">On the <strong id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_b1571097157">Change AZ</strong> page, set parameters.<div class="caution" id="EN-US_TOPIC_0000001938377520__note127148267391"><span class="cautiontitle"><img src="public_sys-resources/caution_3.0-en-us.png"> </span><div class="cautionbody"><p id="EN-US_TOPIC_0000001938377520__p1882132495218">If the number of data nodes plus cold data nodes in a cluster is not divisible by the number of AZs, data in the cluster may be unevenly distributed after the AZ change is completed. For example, if the cluster has two AZs and three data nodes, one node may need to store twice as much data as the other two nodes. This uneven distribution of data may affect the cluster's query or write performance.</p>
|
|
</div></div>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_table1436065715216" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameters for changing AZs</caption><thead align="left"><tr id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_row1636125711210"><th align="left" class="cellrowborder" valign="top" width="18.12%" id="mcps1.3.9.2.5.3.2.3.1.1"><p id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_p173613573215">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="81.88%" id="mcps1.3.9.2.5.3.2.3.1.2"><p id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_p123616570215">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_row106621836114213"><td class="cellrowborder" valign="top" width="18.12%" headers="mcps1.3.9.2.5.3.2.3.1.1 "><p id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_p1666233624218">Operation Type</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="81.88%" headers="mcps1.3.9.2.5.3.2.3.1.2 "><ul id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_ul1772564194319"><li id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_li12725184174318"><strong id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_b642915457368">Add AZ</strong>: Add one or two AZs for a single-AZ cluster, or add one AZ for a dual-AZ cluster.<p id="EN-US_TOPIC_0000001938377520__p132361721175712">To complete this AZ change, the system will try to move as few nodes as possible to rebuild the cluster. During this process, the YML configuration file of the nodes that are not modified is also updated. You need to restart the cluster to make the change take effect.</p>
|
|
</li><li id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_li272524134318"><strong id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_b12993154203619">Migrate AZ</strong>: Migrate data from one AZ to another.<p id="EN-US_TOPIC_0000001938377520__p16991183318579">After adding AZs, you need to restart the cluster to make the modification take effect.</p>
|
|
</li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_row1569612332424"><td class="cellrowborder" valign="top" width="18.12%" headers="mcps1.3.9.2.5.3.2.3.1.1 "><p id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_p36961330424">Node Type</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="81.88%" headers="mcps1.3.9.2.5.3.2.3.1.2 "><p id="EN-US_TOPIC_0000001938377520__p7370156134315">Select a node type for the AZ change.</p>
|
|
<p id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_p14390154111211">Select one node type or <strong id="EN-US_TOPIC_0000001938377520__b6346163613525">All nodes</strong> to change their AZ. When adding one or two AZs for a single-AZ cluster, you can only select <span class="parmvalue" id="EN-US_TOPIC_0000001938377520__parmvalue868258351114513"><b>All nodes</b></span>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_row066111219312"><td class="cellrowborder" valign="top" width="18.12%" headers="mcps1.3.9.2.5.3.2.3.1.1 "><p id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_p17661612137">Current AZ</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="81.88%" headers="mcps1.3.9.2.5.3.2.3.1.2 "><p id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_p14662122036">Current AZ of a cluster</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_row1871942062614"><td class="cellrowborder" valign="top" width="18.12%" headers="mcps1.3.9.2.5.3.2.3.1.1 "><p id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_p107204205267">Target AZ</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="81.88%" headers="mcps1.3.9.2.5.3.2.3.1.2 "><p id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_p13720152017261">AZ to add or migrate to.</p>
|
|
<ul id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_ul10115145774016"><li id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_li311525712402"><strong id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_b457965844110">Add AZ</strong>: Select up to three AZs, which must include all your current AZs.</li><li id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_li191151757124016"><strong id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_b13401104010435">Migrate AZ</strong>: Select only one target AZ, which cannot be your current AZ.</li></ul>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</li><li id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_li637205501913">Click <strong id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_b1062811591016">Submit</strong>. In the displayed dialog box, choose whether to check for the presence of a full index snapshot, and click <strong id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_b311899101013">OK</strong> to start the change.<div class="note" id="EN-US_TOPIC_0000001938377520__note35651135122010"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="EN-US_TOPIC_0000001938377520__p7217738202917">You are advised to select <strong id="EN-US_TOPIC_0000001938377520__b97851521241">Check full index snapshot</strong>. This ensures that all data has been backed up, so that in case the task fails, the data can be restored using this snapshot.</p>
|
|
</div></div>
|
|
<p id="EN-US_TOPIC_0000001938377520__p1917717467184">When <strong id="EN-US_TOPIC_0000001938377520__b1943294251011">Task Status</strong> in the task list below changes to <strong id="EN-US_TOPIC_0000001938377520__b1943224281016">Successful</strong>, the AZ change is completed.</p>
|
|
</li><li id="EN-US_TOPIC_0000001938377520__li112901934153217">Confirm the result.<ol type="a" id="EN-US_TOPIC_0000001938377520__ol684376327"><li id="EN-US_TOPIC_0000001938377520__li356574532611">In the cluster list, click the name of the target cluster. The cluster information page is displayed.</li><li id="EN-US_TOPIC_0000001938377520__li7340135103320">On the <strong id="EN-US_TOPIC_0000001938377520__b161355313338">Overview</strong> tab, check the cluster's AZs in the <strong id="EN-US_TOPIC_0000001938377520__b213953173320">Configuration</strong> area to determine whether the switchover is successful.</li></ol>
|
|
</li></ol>
|
|
</div>
|
|
<div class="section" id="EN-US_TOPIC_0000001938377520__section850410492147"><h4 class="sectiontitle">Checking the Progress of an AZ Change Task</h4><p id="EN-US_TOPIC_0000001938377520__p112511944131517">In the <strong id="EN-US_TOPIC_0000001938377520__b462415108144">Jobs</strong> area, find the AZ change task.</p>
|
|
<p id="EN-US_TOPIC_0000001938377520__p110621119159">Expand the task, and click <strong id="EN-US_TOPIC_0000001938377520__b1236639707114513">View Progress</strong> to check its progress.</p>
|
|
<p id="EN-US_TOPIC_0000001938377520__p272812451517">If the task status is <strong id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_b1528612274476">Failed</strong>, you can retry or terminate the task.</p>
|
|
<ul id="EN-US_TOPIC_0000001938377520__ul1672834121515"><li id="EN-US_TOPIC_0000001938377520__li147281149158">Retry a task: Click <span class="uicontrol" id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_uicontrol13302163611200"><b>Retry</b></span> in the <strong id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_b125101530154919">Operation</strong> column.</li><li id="EN-US_TOPIC_0000001938377520__li107291417157">Terminate a task: Click <span class="uicontrol" id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_uicontrol1330215365208"><b>Terminate</b></span> in the <strong id="EN-US_TOPIC_0000001938377520__en-us_topic_0000001438549444_b83938304503">Operation</strong> column.<p id="EN-US_TOPIC_0000001938377520__p772964111519">If by the time an AZ change task is terminated, the AZ has not been changed for some nodes, you can try to restore these nodes by performing <a href="css_01_0156.html">Replacing a Specified Node for an Elasticsearch Cluster</a>.</p>
|
|
<div class="caution" id="EN-US_TOPIC_0000001938377520__note153618285161"><span class="cautiontitle"><img src="public_sys-resources/caution_3.0-en-us.png"> </span><div class="cautionbody"><p id="EN-US_TOPIC_0000001938377520__p176918252018">If the AZ change is only partially successful, the cluster will have nodes spanning different AZs, and there is no convenient way to continue to change the AZ for the remaining nodes. To avoid this situation, do not terminate the task in this state.</p>
|
|
</div></div>
|
|
</li></ul>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="css_01_0009.html">Managing Elasticsearch Clusters</a></div>
|
|
</div>
|
|
</div>
|
|
|