forked from docs/doc-exports
zhengxiu
2125539080
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: zhengxiu <zhengxiu@huawei.com> Co-committed-by: zhengxiu <zhengxiu@huawei.com>
108 lines
20 KiB
HTML
108 lines
20 KiB
HTML
<a name="css_01_0458"></a><a name="css_01_0458"></a>
|
|
|
|
<h1 class="topictitle1">Enhancing the Data Import Performance of OpenSearch Clusters</h1>
|
|
<div id="body0000001955726482"><div class="section" id="css_01_0458__css_01_0397_en-us_topic_0000001498504010_section537465010594"><h4 class="sectiontitle">Overview</h4><p id="css_01_0458__css_01_0397_en-us_topic_0000001498504010_p116588564596">This unique CSS feature significantly improves data import performance and reduces write rejections through bulk route optimization, text index acceleration, and word segmentation acceleration. You are advised to use this feature for clusters that contain a large number of index shards and text indexes, or have a high inbound data throughput.</p>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="css_01_0458__css_01_0397_table1753585617564" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Ways to improve data import performance</caption><thead align="left"><tr id="css_01_0458__css_01_0397_row19536125613569"><th align="left" class="cellrowborder" valign="top" width="20%" id="mcps1.3.1.3.2.4.1.1"><p id="css_01_0458__css_01_0397_p1053625625619">Method</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="60%" id="mcps1.3.1.3.2.4.1.2"><p id="css_01_0458__css_01_0397_p85361356105615">Description</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="20%" id="mcps1.3.1.3.2.4.1.3"><p id="css_01_0458__css_01_0397_p353655614560">Details</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="css_01_0458__css_01_0397_row753655617563"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.1.3.2.4.1.1 "><p id="css_01_0458__css_01_0397_p053645617563">Bulk route optimization</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60%" headers="mcps1.3.1.3.2.4.1.2 "><p id="css_01_0458__css_01_0397_p20142210125710">According to the default routing rule of OpenSearch, each piece of data in a bulk request is routed to a different shard. When a large amount of data is written in and a large number of index shards exist, excessive internal request forwarding may trigger bulk rejection. Additionally, in a large-scale cluster, the long tail effect causes a high latency for bulk requests.</p>
|
|
<div class="p" id="css_01_0458__css_01_0397_p314231014571">You can specify the <span class="parmname" id="css_01_0458__css_01_0397_parmname6944736753"><b>index.bulk_routing</b></span> configuration item to enable bulk route optimization. You can use it to reduce the number of requests that need to be internally forwarded. For a cluster that has a large number of shards, this setting can improve write performance and reduce bulk rejection.<div class="note" id="css_01_0458__css_01_0397_note954015215555"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="css_01_0458__css_01_0397_p3915164616522">After bulk route optimization is enabled (that is, <span class="parmname" id="css_01_0458__css_01_0397_parmname1845417241533"><b>index.bulk_routing</b></span> is set to <span class="parmvalue" id="css_01_0458__css_01_0397_parmvalue358712381532"><b>pack</b></span> or <span class="parmvalue" id="css_01_0458__css_01_0397_parmvalue226524711532"><b>local_pack</b></span>), data writes are no longer routed based on <span class="parmvalue" id="css_01_0458__css_01_0397_parmvalue164152047105215"><b>_id</b></span>, and routing-related functionality may be affected. For example, ID-based GET requests may fail.</p>
|
|
</div></div>
|
|
</div>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.1.3.2.4.1.3 "><p id="css_01_0458__css_01_0397_p1553665675613"><a href="#css_01_0458__css_01_0397_section9414161115511">Bulk Route Optimization</a></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="css_01_0458__css_01_0397_row175361356205616"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.1.3.2.4.1.1 "><p id="css_01_0458__css_01_0397_p1753675675613">Bulk aggregation optimization</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60%" headers="mcps1.3.1.3.2.4.1.2 "><p id="css_01_0458__css_01_0397_p12380138319">You can specify the <strong id="css_01_0458__css_01_0397_b0926327193">index.aggr_perf_batch_size</strong> configuration item to enable or disable bulk aggregation optimization. When bulk aggregation optimization is enabled, documents in a bulk request are written in batches, rather than individually. This helps to reduce the overheads of memory requests, lock requests, and other calls, improving data import performance.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.1.3.2.4.1.3 "><p id="css_01_0458__css_01_0397_p20536175685615"><a href="#css_01_0458__css_01_0397_section1443152195614">Bulk Aggregation Optimization</a></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="css_01_0458__css_01_0397_row135366561569"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.1.3.2.4.1.1 "><p id="css_01_0458__css_01_0397_p95367566561">Text index acceleration</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60%" headers="mcps1.3.1.3.2.4.1.2 "><div class="p" id="css_01_0458__css_01_0397_p1495320741918">You can configure <strong id="css_01_0458__css_01_0397_b16598211172319">index.native_speed_up</strong> to enable or disable text index acceleration. This setting optimizes the indexing process and memory access to accelerate index building for text fields (text and keyword). When text index acceleration is enabled, you can configure <strong id="css_01_0458__css_01_0397_b331812346262">index.native_analyzer</strong> to enable word segmentation acceleration as well. For texts that need common word segmentation, you can use the analyzer to accelerate word segmentation.<div class="note" id="css_01_0458__css_01_0397_note123054283551"><span class="notetitle"> NOTE: </span><div class="notebody"><ul id="css_01_0458__css_01_0397_ul17269631162818"><li id="css_01_0458__css_01_0397_li183741656162814">Word segmentation acceleration can be enabled (set <strong id="css_01_0458__css_01_0397_b6591145782719">index.native_analyzer</strong> to <strong id="css_01_0458__css_01_0397_b39151101287">true</strong>) only when text index acceleration is enabled (set <strong id="css_01_0458__css_01_0397_b1268112219282">index.native_speed_up</strong> to <strong id="css_01_0458__css_01_0397_b206601830102819">true</strong>). Otherwise, word segmentation acceleration will not take effect.</li><li id="css_01_0458__css_01_0397_li10270103192813">Text index acceleration cannot be enabled for indexes that contain nested fields.</li></ul>
|
|
</div></div>
|
|
</div>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.1.3.2.4.1.3 "><p id="css_01_0458__css_01_0397_p45361756175613"><a href="#css_01_0458__css_01_0397_section15513121414583">Text Index Acceleration</a></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="css_01_0458__css_01_0397_row12295115135817"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.1.3.2.4.1.1 "><p id="css_01_0458__css_01_0397_p2029517519586">Index merge task optimization</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60%" headers="mcps1.3.1.3.2.4.1.2 "><p id="css_01_0458__css_01_0397_p1072614154718">Enabling the three optimization options above will increase the number of index merge tasks in the cluster. You can configure <span class="parmname" id="css_01_0458__css_01_0397_parmname1437333184820"><b>index.merge.scheduler.max_thread_count</b></span> to reduce the impact of index merge tasks on data import performance. The optimization is about increasing the number of shard merging threads and thereby alleviate the throttling of data import.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.1.3.2.4.1.3 "><p id="css_01_0458__css_01_0397_p129665185812"><a href="#css_01_0458__css_01_0397_section124450517595">Index Merge Task Optimization</a></p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="css_01_0458__css_01_0397_en-us_topic_0000001498504010_section10368166125"><h4 class="sectiontitle">Constraints</h4><p id="css_01_0458__p82222398467">Only OpenSearch 1.3.6 clusters support data import performance enhancement.</p>
|
|
</div>
|
|
<div class="section" id="css_01_0458__css_01_0397_en-us_topic_0000001498504010_section1342011569319"><h4 class="sectiontitle">Prerequisites</h4><p id="css_01_0458__css_01_0397_p7570529103216">The cluster whose data import performance you want to enhance is in the Available state.</p>
|
|
</div>
|
|
<div class="section" id="css_01_0458__css_01_0397_section9414161115511"><a name="css_01_0458__css_01_0397_section9414161115511"></a><a name="css_01_0397_section9414161115511"></a><h4 class="sectiontitle">Bulk Route Optimization</h4><div class="p" id="css_01_0458__en-us_topic_0000001549463585_p99041110285">You can specify the <span class="parmname" id="css_01_0458__css_01_0397_parmname6944736753_1"><b>index.bulk_routing</b></span> configuration item to enable bulk route optimization. You can use it to reduce the number of requests that need to be internally forwarded. For a cluster that has a large number of shards, this setting can improve write performance and reduce bulk rejection.<div class="note" id="css_01_0458__css_01_0397_note954015215555_1"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="css_01_0458__css_01_0397_p3915164616522_1">After bulk route optimization is enabled (that is, <span class="parmname" id="css_01_0458__css_01_0397_parmname1845417241533_1"><b>index.bulk_routing</b></span> is set to <span class="parmvalue" id="css_01_0458__css_01_0397_parmvalue358712381532_1"><b>pack</b></span> or <span class="parmvalue" id="css_01_0458__css_01_0397_parmvalue226524711532_1"><b>local_pack</b></span>), data writes are no longer routed based on <span class="parmvalue" id="css_01_0458__css_01_0397_parmvalue164152047105215_1"><b>_id</b></span>, and routing-related functionality may be affected. For example, ID-based GET requests may fail.</p>
|
|
</div></div>
|
|
</div>
|
|
<ol id="css_01_0458__css_01_0397_en-us_topic_0000001549463585_ol1549211439294"><li id="css_01_0458__css_01_0397_en-us_topic_0000001549463585_li12377205073817">Choose <strong id="css_01_0458__b124818916418">Clusters</strong> from the navigation pane. On the <strong id="css_01_0458__b5486917415">Clusters</strong> page, select an available cluster, and click <strong id="css_01_0458__b1748397419">Access Kibana</strong> in the <strong id="css_01_0458__b174811915419">Operation</strong> column to log in to OpenSearch Dashboards.</li><li id="css_01_0458__css_01_0397_en-us_topic_0000001549463585_li1668815383811">In the left navigation tree on the OpenSearch Dashboards console, choose <strong id="css_01_0458__b163983764113">Dev Tools</strong>.</li><li id="css_01_0458__css_01_0397_en-us_topic_0000001549463585_li14923433297">On the <strong id="css_01_0458__css_01_0397_b107456851310">Dev Tools</strong> page, run the following command to enable bulk route optimization:<pre class="screen" id="css_01_0458__css_01_0397_en-us_topic_0000001549463585_screen444475117597">PUT my_index
|
|
{
|
|
"settings": {
|
|
"index.bulk_routing": "local_pack"
|
|
}
|
|
}</pre>
|
|
<div class="p" id="css_01_0458__css_01_0397_p1175710136810">The options for the <strong id="css_01_0458__css_01_0397_b18878199131419">index.bulk_routing</strong> configuration item are as follows:<ul id="css_01_0458__css_01_0397_ul891810571812"><li id="css_01_0458__css_01_0397_li291875718817"><span class="parmvalue" id="css_01_0458__css_01_0397_parmvalue101051951494"><b>default</b></span>: Use the default routing mechanism. Records in a bulk request are split and then routed separately.</li><li id="css_01_0458__css_01_0397_li159189578810"><span class="parmvalue" id="css_01_0458__css_01_0397_parmvalue74881971392"><b>pack</b></span>: All data of a single bulk request is randomly routed to the same shard.</li><li id="css_01_0458__css_01_0397_li59187571089"><span class="parmvalue" id="css_01_0458__css_01_0397_parmvalue120614112919"><b>local_pack</b></span>: The data of a single bulk request is routed to a local shard of the data node that has received the bulk request. If the node does not contain the corresponding index shard, the data is randomly routed to another node that contains the index shard. This solution depends on the random distribution of client bulk requests and the balanced distribution of primary shards.</li></ul>
|
|
</div>
|
|
</li></ol>
|
|
</div>
|
|
<div class="section" id="css_01_0458__css_01_0397_section1443152195614"><a name="css_01_0458__css_01_0397_section1443152195614"></a><a name="css_01_0397_section1443152195614"></a><h4 class="sectiontitle">Bulk Aggregation Optimization</h4><p id="css_01_0458__en-us_topic_0000001549503741_p725354043716">You can specify the <strong id="css_01_0458__css_01_0397_b0926327193_1">index.aggr_perf_batch_size</strong> configuration item to enable or disable bulk aggregation optimization. When bulk aggregation optimization is enabled, documents in a bulk request are written in batches, rather than individually. This helps to reduce the overheads of memory requests, lock requests, and other calls, improving data import performance.</p>
|
|
<ol id="css_01_0458__css_01_0397_en-us_topic_0000001549503741_ol1549211439294"><li id="css_01_0458__css_01_0397_li155601062614">Choose <strong id="css_01_0458__b13501996415">Clusters</strong> from the navigation pane. On the <strong id="css_01_0458__b25012914114">Clusters</strong> page, select an available cluster, and click <strong id="css_01_0458__b155013914119">Access Kibana</strong> in the <strong id="css_01_0458__b750093415">Operation</strong> column to log in to OpenSearch Dashboards.</li><li id="css_01_0458__css_01_0397_li55612014267">In the left navigation tree on the OpenSearch Dashboards console, choose <strong id="css_01_0458__b106401437104110">Dev Tools</strong>.</li><li id="css_01_0458__css_01_0397_en-us_topic_0000001549503741_li14923433297">On the <strong id="css_01_0458__css_01_0397_b116391691918">Dev Tools</strong> page, run the following command to enable bulk aggregation optimization:<pre class="screen" id="css_01_0458__css_01_0397_en-us_topic_0000001549503741_screen1467804014239">PUT my_index
|
|
{
|
|
"settings": {
|
|
"index.aggr_perf_batch_size": "128"
|
|
}
|
|
}</pre>
|
|
<p id="css_01_0458__css_01_0397_p17395730201018">The value range of the <span class="parmname" id="css_01_0458__css_01_0397_parmname163951930111013"><b>index.aggr_perf_batch_size</b></span> configuration item is [1, Integer.MAX_VALUE]. The default value is 1, indicating that bulk aggregation optimization is disabled. If the value is greater than 1, bulk aggregation optimization is enabled, the value of <strong id="css_01_0458__css_01_0397_b66421432172012">MIN(bulk_doc_size, aggr_perf_batch_size)</strong> indicates the bulk size.</p>
|
|
</li></ol>
|
|
</div>
|
|
<div class="section" id="css_01_0458__css_01_0397_section15513121414583"><a name="css_01_0458__css_01_0397_section15513121414583"></a><a name="css_01_0397_section15513121414583"></a><h4 class="sectiontitle">Text Index Acceleration</h4><div class="p" id="css_01_0458__css_01_0397_p1936614335224">You can configure <strong id="css_01_0458__css_01_0397_css_01_0397_b16598211172319">index.native_speed_up</strong> to enable or disable text index acceleration. This setting optimizes the indexing process and memory access to accelerate index building for text fields (text and keyword). When text index acceleration is enabled, you can configure <strong id="css_01_0458__css_01_0397_css_01_0397_b331812346262">index.native_analyzer</strong> to enable word segmentation acceleration as well. For texts that need common word segmentation, you can use the analyzer to accelerate word segmentation.<div class="note" id="css_01_0458__css_01_0397_css_01_0397_note123054283551"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="css_01_0458__css_01_0397_css_01_0397_ul17269631162818"><li id="css_01_0458__css_01_0397_css_01_0397_li183741656162814">Word segmentation acceleration can be enabled (set <strong id="css_01_0458__css_01_0397_css_01_0397_b6591145782719">index.native_analyzer</strong> to <strong id="css_01_0458__css_01_0397_css_01_0397_b39151101287">true</strong>) only when text index acceleration is enabled (set <strong id="css_01_0458__css_01_0397_css_01_0397_b1268112219282">index.native_speed_up</strong> to <strong id="css_01_0458__css_01_0397_css_01_0397_b206601830102819">true</strong>). Otherwise, word segmentation acceleration will not take effect.</li><li id="css_01_0458__css_01_0397_css_01_0397_li10270103192813">Text index acceleration cannot be enabled for indexes that contain nested fields.</li></ul>
|
|
</div></div>
|
|
</div>
|
|
<ol id="css_01_0458__css_01_0397_en-us_topic_0000001549303621_ol1549211439294"><li id="css_01_0458__css_01_0397_li1251418618263">Choose <strong id="css_01_0458__b65311984118">Clusters</strong> from the navigation pane. On the <strong id="css_01_0458__b3531699414">Clusters</strong> page, select an available cluster, and click <strong id="css_01_0458__b753892419">Access Kibana</strong> in the <strong id="css_01_0458__b7531192413">Operation</strong> column to log in to OpenSearch Dashboards.</li><li id="css_01_0458__css_01_0397_li05149610262">In the left navigation tree on the OpenSearch Dashboards console, choose <strong id="css_01_0458__b4642163715414">Dev Tools</strong>.</li><li id="css_01_0458__css_01_0397_en-us_topic_0000001549303621_li14923433297">On the <strong id="css_01_0458__css_01_0397_b18280145422119">Dev Tools</strong> page, run the following command to enable text index acceleration:<pre class="screen" id="css_01_0458__css_01_0397_en-us_topic_0000001549303621_screen1467804014239">PUT my_index
|
|
{
|
|
"settings": {
|
|
"index.native_speed_up": true,
|
|
"index.native_analyzer": true
|
|
}
|
|
}</pre>
|
|
<p id="css_01_0458__css_01_0397_p2029183142310">The values of <span class="parmname" id="css_01_0458__parmname1751911171248"><b>index.native_speed_up</b></span> and <span class="parmname" id="css_01_0458__parmname15519121712411"><b>index.native_analyzer</b></span> are <strong id="css_01_0458__b10519417246">true</strong> or <strong id="css_01_0458__b351918176414">false</strong>. The default value is <strong id="css_01_0458__b1351941715419">false</strong>.</p>
|
|
</li></ol>
|
|
</div>
|
|
<div class="section" id="css_01_0458__css_01_0397_section124450517595"><a name="css_01_0458__css_01_0397_section124450517595"></a><a name="css_01_0397_section124450517595"></a><h4 class="sectiontitle">Index Merge Task Optimization</h4><p id="css_01_0458__css_01_0397_p11566721163213">Enabling the three optimization options above will increase the number of index merge tasks in the cluster. You can configure <span class="parmname" id="css_01_0458__css_01_0397_css_01_0397_parmname1437333184820"><b>index.merge.scheduler.max_thread_count</b></span> to reduce the impact of index merge tasks on data import performance. The optimization is about increasing the number of shard merging threads and thereby alleviate the throttling of data import.</p>
|
|
<ol id="css_01_0458__css_01_0397_ol1277941985919"><li id="css_01_0458__css_01_0397_li427591018269">Choose <strong id="css_01_0458__b47891047164014">Clusters</strong> from the navigation pane. On the <strong id="css_01_0458__b1579020479405">Clusters</strong> page, select an available cluster, and click <strong id="css_01_0458__b1479034715400">Access Kibana</strong> in the <strong id="css_01_0458__b8790147184012">Operation</strong> column to log in to OpenSearch Dashboards.</li><li id="css_01_0458__css_01_0397_li18275191015264">In the left navigation tree on the OpenSearch Dashboards console, choose <strong id="css_01_0458__b564191518417">Dev Tools</strong>.</li><li id="css_01_0458__css_01_0397_li6779819145915">On the <strong id="css_01_0458__css_01_0397_b1785011492411">Dev Tools</strong> page, run the following command to start index merge task optimization:<pre class="screen" id="css_01_0458__css_01_0397_screen1778014196590">PUT my_index
|
|
{
|
|
"settings": {
|
|
"index.merge.scheduler.max_thread_count": 8
|
|
}
|
|
}</pre>
|
|
<p id="css_01_0458__css_01_0397_p162576497495">The value range of <strong id="css_01_0458__css_01_0397_b1373213992419">index.merge.scheduler.max_thread_count</strong> is [1, node.processors/2]. The default value is 4, and the recommended value is 8.</p>
|
|
</li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="css_01_0453.html">Importing Data to an OpenSearch Cluster</a></div>
|
|
</div>
|
|
</div>
|
|
|