forked from docs/doc-exports
zhengxiu
93d856d5c5
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: zhengxiu <zhengxiu@huawei.com> Co-committed-by: zhengxiu <zhengxiu@huawei.com>
108 lines
18 KiB
HTML
108 lines
18 KiB
HTML
<a name="EN-US_TOPIC_0000001938377700"></a><a name="EN-US_TOPIC_0000001938377700"></a>
|
|
|
|
<h1 class="topictitle1">Enhancing the Data Ingestion Performance of Elasticsearch Clusters</h1>
|
|
<div id="body8662426"><div class="section" id="EN-US_TOPIC_0000001938377700__en-us_topic_0000001498504010_section537465010594"><h4 class="sectiontitle">Overview</h4><p id="EN-US_TOPIC_0000001938377700__en-us_topic_0000001498504010_p116588564596">This unique CSS feature significantly enhances data ingestion performance and reduces write rejections through bulk route optimization, text index acceleration, and word segmentation acceleration. You are advised to use this feature for clusters that contain a large number of index shards and text indexes, or have a high inbound data throughput.</p>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="EN-US_TOPIC_0000001938377700__table1753585617564" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Ways to improve data ingestion performance</caption><thead align="left"><tr id="EN-US_TOPIC_0000001938377700__row19536125613569"><th align="left" class="cellrowborder" valign="top" width="20%" id="mcps1.3.1.3.2.4.1.1"><p id="EN-US_TOPIC_0000001938377700__p1053625625619">Method</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="60%" id="mcps1.3.1.3.2.4.1.2"><p id="EN-US_TOPIC_0000001938377700__p85361356105615">Description</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="20%" id="mcps1.3.1.3.2.4.1.3"><p id="EN-US_TOPIC_0000001938377700__p353655614560">Details</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="EN-US_TOPIC_0000001938377700__row753655617563"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.1.3.2.4.1.1 "><p id="EN-US_TOPIC_0000001938377700__p053645617563">Bulk route optimization</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60%" headers="mcps1.3.1.3.2.4.1.2 "><p id="EN-US_TOPIC_0000001938377700__p20142210125710">According to the default routing rule of Elasticsearch, each piece of data in a bulk request is routed to a different shard. When a large amount of data is written in and a large number of index shards exist, excessive internal request forwarding may trigger bulk rejection. Additionally, in a large-scale cluster, the long tail effect causes a high latency for bulk requests.</p>
|
|
<p id="EN-US_TOPIC_0000001938377700__p314231014571">You can specify the <span class="parmname" id="EN-US_TOPIC_0000001938377700__parmname6944736753"><b>index.bulk_routing</b></span> configuration item to enable bulk route optimization. You can use it to reduce the number of requests that need to be internally forwarded. For a cluster that has a large number of shards, this setting can improve write performance and reduce bulk rejection.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.1.3.2.4.1.3 "><p id="EN-US_TOPIC_0000001938377700__p1553665675613"><a href="#EN-US_TOPIC_0000001938377700__section9414161115511">Bulk Route Optimization</a></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="EN-US_TOPIC_0000001938377700__row175361356205616"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.1.3.2.4.1.1 "><p id="EN-US_TOPIC_0000001938377700__p1753675675613">Bulk aggregation optimization</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60%" headers="mcps1.3.1.3.2.4.1.2 "><p id="EN-US_TOPIC_0000001938377700__p12380138319">You can specify the <strong id="EN-US_TOPIC_0000001938377700__b0926327193">index.aggr_perf_batch_size</strong> configuration item to enable or disable bulk aggregation optimization. When bulk aggregation optimization is enabled, documents in a bulk request are written in batches, rather than individually. This helps to reduce the overheads of memory requests, lock requests, and other calls, enhancing data ingestion performance.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.1.3.2.4.1.3 "><p id="EN-US_TOPIC_0000001938377700__p20536175685615"><a href="#EN-US_TOPIC_0000001938377700__section1443152195614">Bulk Aggregation Optimization</a></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="EN-US_TOPIC_0000001938377700__row135366561569"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.1.3.2.4.1.1 "><p id="EN-US_TOPIC_0000001938377700__p95367566561">Text index acceleration</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60%" headers="mcps1.3.1.3.2.4.1.2 "><p id="EN-US_TOPIC_0000001938377700__p1495320741918">You can configure <strong id="EN-US_TOPIC_0000001938377700__b16598211172319">index.native_speed_up</strong> to enable or disable text index acceleration. This setting optimizes the indexing process and memory access to accelerate index building for text fields (text and keyword). When text index acceleration is enabled, you can configure <strong id="EN-US_TOPIC_0000001938377700__b331812346262">index.native_analyzer</strong> to enable word segmentation acceleration as well. For texts that need common word segmentation, you can use the analyzer to accelerate word segmentation.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.1.3.2.4.1.3 "><p id="EN-US_TOPIC_0000001938377700__p45361756175613"><a href="#EN-US_TOPIC_0000001938377700__section15513121414583">Text Index Acceleration</a></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="EN-US_TOPIC_0000001938377700__row12295115135817"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.1.3.2.4.1.1 "><p id="EN-US_TOPIC_0000001938377700__p2029517519586">Index merge task optimization</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60%" headers="mcps1.3.1.3.2.4.1.2 "><p id="EN-US_TOPIC_0000001938377700__p1072614154718">Enabling the three optimization options above will increase the number of index merge tasks in the cluster. You can configure <span class="parmname" id="EN-US_TOPIC_0000001938377700__parmname147732310370"><b>index.merge.scheduler.max_thread_count</b></span> to reduce the impact of index merge tasks on data ingestion performance. The optimization is about increasing the number of shard merging threads and thereby alleviate the throttling of data import.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.1.3.2.4.1.3 "><p id="EN-US_TOPIC_0000001938377700__p129665185812"><a href="#EN-US_TOPIC_0000001938377700__section124450517595">Index Merge Task Optimization</a></p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="EN-US_TOPIC_0000001938377700__en-us_topic_0000001498504010_section10368166125"><h4 class="sectiontitle">Constraints</h4><p id="EN-US_TOPIC_0000001938377700__p944843310255">Only Elasticsearch 7.10.2 clusters support data ingestion performance enhancement.</p>
|
|
</div>
|
|
<div class="section" id="EN-US_TOPIC_0000001938377700__en-us_topic_0000001498504010_section1342011569319"><h4 class="sectiontitle">Prerequisites</h4><p id="EN-US_TOPIC_0000001938377700__p7570529103216">The target cluster is available.</p>
|
|
</div>
|
|
<div class="section" id="EN-US_TOPIC_0000001938377700__section17337522988"><h4 class="sectiontitle">Logging In to Kibana</h4><p id="EN-US_TOPIC_0000001938377700__en-us_topic_0000002338102386_en-us_topic_0000002271391393_p84498183236">Log in to Kibana and go to the command execution page. Elasticsearch clusters support multiple access methods. This topic uses Kibana as an example to describe the operation procedures.</p>
|
|
<ol id="EN-US_TOPIC_0000001938377700__en-us_topic_0000002338102386_en-us_topic_0000002271391393_ol13172153674611"><li id="EN-US_TOPIC_0000001938377700__en-us_topic_0000002338102386_en-us_topic_0000002271391393_en-us_topic_0000001268594549_li5509181942419"><span id="EN-US_TOPIC_0000001938377700__en-us_topic_0000002338102386_ph152259618315">Log in to the CSS management console.</span></li><li id="EN-US_TOPIC_0000001938377700__en-us_topic_0000002338102386_en-us_topic_0000002271391393_li4250043448">In the navigation pane on the left, choose <span class="uicontrol" id="EN-US_TOPIC_0000001938377700__en-us_topic_0000002338102386_uicontrol192689888185117"><b>Clusters > Elasticsearch</b></span>.</li><li id="EN-US_TOPIC_0000001938377700__en-us_topic_0000002338102386_en-us_topic_0000002271391393_en-us_topic_0000001268154521_li6012554340">In the cluster list, find the target cluster, and click <strong id="EN-US_TOPIC_0000001938377700__en-us_topic_0000002338102386_b18928803044129">Kibana</strong> in the <strong id="EN-US_TOPIC_0000001938377700__en-us_topic_0000002338102386_b5255083364129">Operation</strong> column to log in to the Kibana console.</li><li id="EN-US_TOPIC_0000001938377700__en-us_topic_0000002338102386_en-us_topic_0000002271391393_li270075935613">In the left navigation pane, choose <strong id="EN-US_TOPIC_0000001938377700__en-us_topic_0000002338102386_b3927845824129">Dev Tools</strong>.<p id="EN-US_TOPIC_0000001938377700__en-us_topic_0000002338102386_p1810995434510">The left part of the console is the command input box, and the triangle icon in its upper-right corner is the execution button. The right part shows the execution result.</p>
|
|
</li></ol>
|
|
</div>
|
|
<div class="section" id="EN-US_TOPIC_0000001938377700__section9414161115511"><a name="EN-US_TOPIC_0000001938377700__section9414161115511"></a><a name="section9414161115511"></a><h4 class="sectiontitle">Bulk Route Optimization</h4><p id="EN-US_TOPIC_0000001938377700__en-us_topic_0000001549463585_p99041110285">You can specify the <span class="parmname" id="EN-US_TOPIC_0000001938377700__en-us_topic_0000001938377700_parmname6944736753"><b>index.bulk_routing</b></span> configuration item to enable bulk route optimization. You can use it to reduce the number of requests that need to be internally forwarded. For a cluster that has a large number of shards, this setting can improve write performance and reduce bulk rejection.</p>
|
|
<div class="caution" id="EN-US_TOPIC_0000001938377700__note11118101661313"><span class="cautiontitle"><img src="public_sys-resources/caution_3.0-en-us.png"> </span><div class="cautionbody"><p id="EN-US_TOPIC_0000001938377700__p211861617139">After bulk route optimization is enabled (that is, <span class="parmname" id="EN-US_TOPIC_0000001938377700__parmname048175513296"><b>index.bulk_routing</b></span> is set to <span class="parmvalue" id="EN-US_TOPIC_0000001938377700__parmvalue11482155582911"><b>pack</b></span> or <span class="parmvalue" id="EN-US_TOPIC_0000001938377700__parmvalue11482355192914"><b>local_pack</b></span>), data writes are no longer routed based on <span class="parmvalue" id="EN-US_TOPIC_0000001938377700__parmvalue64822559297"><b>_id</b></span>, and routing-related functionality may be affected. For example, ID-based GET requests may fail.</p>
|
|
</div></div>
|
|
<div class="p" id="EN-US_TOPIC_0000001938377700__p432914411097">Run the following command to enable bulk route optimization:<pre class="screen" id="EN-US_TOPIC_0000001938377700__screen159442373917">PUT my_index
|
|
{
|
|
"settings": {
|
|
"index.bulk_routing": "local_pack"
|
|
}
|
|
}</pre>
|
|
</div>
|
|
<div class="p" id="EN-US_TOPIC_0000001938377700__p99450375915">The options for the <strong id="EN-US_TOPIC_0000001938377700__b18878199131419">index.bulk_routing</strong> configuration item are as follows:<ul id="EN-US_TOPIC_0000001938377700__ul1594511378916"><li id="EN-US_TOPIC_0000001938377700__li199451237191"><span class="parmvalue" id="EN-US_TOPIC_0000001938377700__parmvalue101051951494"><b>default</b></span>: Use the default routing mechanism. Records in a bulk request are split and then routed separately.</li><li id="EN-US_TOPIC_0000001938377700__li49452371910"><span class="parmvalue" id="EN-US_TOPIC_0000001938377700__parmvalue74881971392"><b>pack</b></span>: All data of a single bulk request is randomly routed to the same shard.</li><li id="EN-US_TOPIC_0000001938377700__li1194517377912"><span class="parmvalue" id="EN-US_TOPIC_0000001938377700__parmvalue120614112919"><b>local_pack</b></span>: The data of a single bulk request is routed to a local shard of the data node that has received the bulk request. If the node does not contain the corresponding index shard, the data is randomly routed to another node that contains the index shard. This solution depends on the random distribution of client bulk requests and the balanced distribution of primary shards.</li></ul>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="EN-US_TOPIC_0000001938377700__section1443152195614"><a name="EN-US_TOPIC_0000001938377700__section1443152195614"></a><a name="section1443152195614"></a><h4 class="sectiontitle">Bulk Aggregation Optimization</h4><p id="EN-US_TOPIC_0000001938377700__en-us_topic_0000001549503741_p725354043716">You can specify the <strong id="EN-US_TOPIC_0000001938377700__en-us_topic_0000001938377700_b0926327193">index.aggr_perf_batch_size</strong> configuration item to enable or disable bulk aggregation optimization. When bulk aggregation optimization is enabled, documents in a bulk request are written in batches, rather than individually. This helps to reduce the overheads of memory requests, lock requests, and other calls, enhancing data ingestion performance.</p>
|
|
<div class="p" id="EN-US_TOPIC_0000001938377700__p5996132816105">Run the following command to enable bulk aggregation optimization:<pre class="screen" id="EN-US_TOPIC_0000001938377700__screen157941924101014">PUT my_index
|
|
{
|
|
"settings": {
|
|
"index.aggr_perf_batch_size": "128"
|
|
}
|
|
}</pre>
|
|
</div>
|
|
<p id="EN-US_TOPIC_0000001938377700__p7794424201014">The value range of the <span class="parmname" id="EN-US_TOPIC_0000001938377700__parmname163951930111013"><b>index.aggr_perf_batch_size</b></span> configuration item is [1, Integer.MAX_VALUE]. The default value is 1, indicating that bulk aggregation optimization is disabled. If the value is greater than 1, bulk aggregation optimization is enabled, the value of <strong id="EN-US_TOPIC_0000001938377700__b66421432172012">MIN(bulk_doc_size, aggr_perf_batch_size)</strong> indicates the bulk size.</p>
|
|
</div>
|
|
<div class="section" id="EN-US_TOPIC_0000001938377700__section15513121414583"><a name="EN-US_TOPIC_0000001938377700__section15513121414583"></a><a name="section15513121414583"></a><h4 class="sectiontitle">Text Index Acceleration</h4><p id="EN-US_TOPIC_0000001938377700__p1936614335224">You can configure <strong id="EN-US_TOPIC_0000001938377700__en-us_topic_0000001938377700_b16598211172319">index.native_speed_up</strong> to enable or disable text index acceleration. This setting optimizes the indexing process and memory access to accelerate index building for text fields (text and keyword). When text index acceleration is enabled, you can configure <strong id="EN-US_TOPIC_0000001938377700__en-us_topic_0000001938377700_b331812346262">index.native_analyzer</strong> to enable word segmentation acceleration as well. For texts that need common word segmentation, you can use the analyzer to accelerate word segmentation.</p>
|
|
<div class="caution" id="EN-US_TOPIC_0000001938377700__note983052617137"><span class="cautiontitle"><img src="public_sys-resources/caution_3.0-en-us.png"> </span><div class="cautionbody"><ul id="EN-US_TOPIC_0000001938377700__ul8830162610132"><li id="EN-US_TOPIC_0000001938377700__li2083012263135">Word segmentation acceleration can be enabled (set <strong id="EN-US_TOPIC_0000001938377700__b4130173810315">index.native_analyzer</strong> to <strong id="EN-US_TOPIC_0000001938377700__b31318382317">true</strong>) only when text index acceleration is enabled (set <strong id="EN-US_TOPIC_0000001938377700__b17131338153117">index.native_speed_up</strong> to <strong id="EN-US_TOPIC_0000001938377700__b111311381319">true</strong>). Otherwise, word segmentation acceleration will not take effect.</li><li id="EN-US_TOPIC_0000001938377700__li18831132641320">Text index acceleration cannot be enabled for indexes that contain nested fields.</li></ul>
|
|
</div></div>
|
|
<div class="p" id="EN-US_TOPIC_0000001938377700__p16161410121214">Run the following command to enable text index acceleration:<pre class="screen" id="EN-US_TOPIC_0000001938377700__screen72547621215">PUT my_index
|
|
{
|
|
"settings": {
|
|
"index.native_speed_up": true,
|
|
"index.native_analyzer": true
|
|
}
|
|
}</pre>
|
|
</div>
|
|
<p id="EN-US_TOPIC_0000001938377700__p18254126121210">The values of <span class="parmname" id="EN-US_TOPIC_0000001938377700__parmname1484943112317"><b>index.native_speed_up</b></span> and <span class="parmname" id="EN-US_TOPIC_0000001938377700__parmname750610416248"><b>index.native_analyzer</b></span> are <strong id="EN-US_TOPIC_0000001938377700__b343483620227">true</strong> or <strong id="EN-US_TOPIC_0000001938377700__b13661833122215">false</strong>. The default value is <strong id="EN-US_TOPIC_0000001938377700__b201957450228">false</strong>.</p>
|
|
</div>
|
|
<div class="section" id="EN-US_TOPIC_0000001938377700__section124450517595"><a name="EN-US_TOPIC_0000001938377700__section124450517595"></a><a name="section124450517595"></a><h4 class="sectiontitle">Index Merge Task Optimization</h4><p id="EN-US_TOPIC_0000001938377700__p11566721163213">Enabling the three optimization options above will increase the number of index merge tasks in the cluster. You can configure <span class="parmname" id="EN-US_TOPIC_0000001938377700__en-us_topic_0000001938377700_parmname147732310370"><b>index.merge.scheduler.max_thread_count</b></span> to reduce the impact of index merge tasks on data ingestion performance. The optimization is about increasing the number of shard merging threads and thereby alleviate the throttling of data import.</p>
|
|
<div class="p" id="EN-US_TOPIC_0000001938377700__p15109163620127">Run the following command to start index merge task optimization:<pre class="screen" id="EN-US_TOPIC_0000001938377700__screen1082933091214">PUT my_index
|
|
{
|
|
"settings": {
|
|
"index.merge.scheduler.max_thread_count": 8
|
|
}
|
|
}</pre>
|
|
</div>
|
|
<p id="EN-US_TOPIC_0000001938377700__p1582918304123">The value range of <strong id="EN-US_TOPIC_0000001938377700__b1373213992419">index.merge.scheduler.max_thread_count</strong> is [1, node.processors/2]. The default value is 4, and the recommended value is 8.</p>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="css_01_0004.html">Importing Data to an Elasticsearch Cluster</a></div>
|
|
</div>
|
|
</div>
|
|
|