forked from docs/doc-exports
zhengxiu
93d856d5c5
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: zhengxiu <zhengxiu@huawei.com> Co-committed-by: zhengxiu <zhengxiu@huawei.com>
131 lines
11 KiB
HTML
131 lines
11 KiB
HTML
<a name="EN-US_TOPIC_0000001945223382"></a><a name="EN-US_TOPIC_0000001945223382"></a>
|
|
|
|
<h1 class="topictitle1">Decoupling Index Storage and Compute in an Elasticsearch Cluster Through Index Lifecycle Management</h1>
|
|
<div id="body8662426"><div class="section" id="EN-US_TOPIC_0000001945223382__en-us_topic_0000001934020326_section766512014019"><h4 class="sectiontitle">Overview</h4><p id="EN-US_TOPIC_0000001945223382__en-us_topic_0000001934020326_p547911812417">CSS supports decoupled storage and compute. That is, indexes can be frozen and their data can be dumped to OBS to reduce the storage cost of cold data. This document describes how to use index lifecycle management to automatically freeze indexes at a specific time to decouple storage and compute.</p>
|
|
<p id="EN-US_TOPIC_0000001945223382__en-us_topic_0000001934020326_p383294312412">In this example, a lifecycle policy is created for an Elasticsearch 7.10.2 cluster. According to this policy, a newly created index is automatically frozen in three days, with data dumped to OBS; and deleted in seven days.</p>
|
|
<div class="fignone" id="EN-US_TOPIC_0000001945223382__fig1632802910912"><span class="figcap"><b>Figure 1 </b>Storage-compute decoupling</span><br><span><img id="EN-US_TOPIC_0000001945223382__en-us_topic_0000001992205869_image7316936171810" src="figure/en-us_image_0000002061403354.png"></span></div>
|
|
</div>
|
|
<div class="section" id="EN-US_TOPIC_0000001945223382__section19541161817252"><h4 class="sectiontitle">Constraints</h4><p id="EN-US_TOPIC_0000001945223382__p188109457258">Only Elasticsearch 7.6.2 and 7.10.2 clusters support decoupled storage and compute. </p>
|
|
</div>
|
|
<div class="section" id="EN-US_TOPIC_0000001945223382__en-us_topic_0000001934020326_section1453615515018"><h4 class="sectiontitle">Prerequisites</h4><p id="EN-US_TOPIC_0000001945223382__p97501551142618">The Elasticsearch cluster is available.</p>
|
|
</div>
|
|
<div class="section" id="EN-US_TOPIC_0000001945223382__en-us_topic_0000001934020326_section1068981011015"><h4 class="sectiontitle">Decoupling Index Storage and Compute Through Index Lifecycle Management</h4><ol id="EN-US_TOPIC_0000001945223382__en-us_topic_0000001934020326_ol579517241688"><li id="EN-US_TOPIC_0000001945223382__li3529193555517">Log in to Kibana.<ol type="a" id="EN-US_TOPIC_0000001945223382__ol1949254165519"><li id="EN-US_TOPIC_0000001945223382__en-us_topic_0000002338102386_en-us_topic_0000002271391393_en-us_topic_0000001268594549_li5509181942419"><span id="EN-US_TOPIC_0000001945223382__en-us_topic_0000002338102386_ph152259618315">Log in to the CSS management console.</span></li><li id="EN-US_TOPIC_0000001945223382__en-us_topic_0000002338102386_en-us_topic_0000002271391393_li4250043448">In the navigation pane on the left, choose <span class="uicontrol" id="EN-US_TOPIC_0000001945223382__en-us_topic_0000002338102386_uicontrol192689888185117"><b>Clusters > Elasticsearch</b></span>.</li><li id="EN-US_TOPIC_0000001945223382__en-us_topic_0000002338102386_en-us_topic_0000002271391393_en-us_topic_0000001268154521_li6012554340">In the cluster list, find the target cluster, and click <strong id="EN-US_TOPIC_0000001945223382__en-us_topic_0000002338102386_b18928803044129">Kibana</strong> in the <strong id="EN-US_TOPIC_0000001945223382__en-us_topic_0000002338102386_b5255083364129">Operation</strong> column to log in to the Kibana console.</li><li id="EN-US_TOPIC_0000001945223382__en-us_topic_0000002338102386_en-us_topic_0000002271391393_li270075935613">In the left navigation pane, choose <strong id="EN-US_TOPIC_0000001945223382__en-us_topic_0000002338102386_b3927845824129">Dev Tools</strong>.<p id="EN-US_TOPIC_0000001945223382__en-us_topic_0000002338102386_p1810995434510">The left part of the console is the command input box, and the triangle icon in its upper-right corner is the execution button. The right part shows the execution result.</p>
|
|
</li></ol>
|
|
</li><li id="EN-US_TOPIC_0000001945223382__en-us_topic_0000001934020326_li10320113081711">Create a lifecycle policy named <strong id="EN-US_TOPIC_0000001945223382__en-us_topic_0000001934020326_b1998114012310">hot_warm_policy</strong>.<p id="EN-US_TOPIC_0000001945223382__en-us_topic_0000001934020326_p192937719172">Policy description: Three days after an index is created, the API for freezing indexes is automatically called to dump data to OBS. Seven days after index creation, the index is deleted.</p>
|
|
<pre class="screen" id="EN-US_TOPIC_0000001945223382__screen1083116171617">PUT _opendistro/_ism/policies/hot_warm_policy
|
|
{
|
|
"policy": {
|
|
"description": "hot warm archive delete workflow",
|
|
"error_notification": null,
|
|
"default_state": "hot",
|
|
"states": [
|
|
{
|
|
"name": "hot",
|
|
"actions": [],
|
|
"transitions": [
|
|
{
|
|
"state_name": "warm",
|
|
"conditions": {
|
|
"min_index_age": "3d"
|
|
}
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"name": "warm",
|
|
"actions": [
|
|
{
|
|
"freeze_low_cost": {}
|
|
}
|
|
],
|
|
"transitions": [
|
|
{
|
|
"state_name": "delete",
|
|
"conditions": {
|
|
"min_index_age": "7d"
|
|
}
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"name": "delete",
|
|
"actions": [
|
|
{
|
|
"delete": {}
|
|
}
|
|
],
|
|
"transitions": []
|
|
}
|
|
]
|
|
}
|
|
}</pre>
|
|
</li><li id="EN-US_TOPIC_0000001945223382__en-us_topic_0000001934020326_li115385520168">Create the index template <strong id="EN-US_TOPIC_0000001945223382__en-us_topic_0000001934020326_b11468134016309">template_hot_warm</strong>.<p id="EN-US_TOPIC_0000001945223382__en-us_topic_0000001934020326_p432502415202">Template description: All the new indexes starting with <strong id="EN-US_TOPIC_0000001945223382__en-us_topic_0000001934020326_b23861882314">data</strong> are automatically associated with the lifecycle policy <strong id="EN-US_TOPIC_0000001945223382__en-us_topic_0000001934020326_b105016538305">hot_warm_policy</strong>.</p>
|
|
<pre class="screen" id="EN-US_TOPIC_0000001945223382__en-us_topic_0000001934020326_screen1532014017297">PUT _template/template_hot_warm
|
|
{
|
|
"index_patterns": "data*",
|
|
"settings": {
|
|
"number_of_replicas": 5,
|
|
"number_of_shards": 1,
|
|
"opendistro.index_state_management.policy_id": "hot_warm_policy"
|
|
},
|
|
"mappings": {
|
|
"properties": {
|
|
"name": {
|
|
"type": "text"
|
|
}
|
|
}
|
|
}
|
|
}</pre>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="EN-US_TOPIC_0000001945223382__en-us_topic_0000001934020326_table1969461617545" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="EN-US_TOPIC_0000001945223382__en-us_topic_0000001934020326_row369411165549"><th align="left" class="cellrowborder" valign="top" width="60%" id="mcps1.3.4.2.3.4.2.3.1.1"><p id="EN-US_TOPIC_0000001945223382__en-us_topic_0000001934020326_p069431635417">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="40%" id="mcps1.3.4.2.3.4.2.3.1.2"><p id="EN-US_TOPIC_0000001945223382__en-us_topic_0000001934020326_p5694816135416">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="EN-US_TOPIC_0000001945223382__en-us_topic_0000001934020326_row1169441645413"><td class="cellrowborder" valign="top" width="60%" headers="mcps1.3.4.2.3.4.2.3.1.1 "><p id="EN-US_TOPIC_0000001945223382__en-us_topic_0000001934020326_p12694151605412">number_of_shards</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40%" headers="mcps1.3.4.2.3.4.2.3.1.2 "><p id="EN-US_TOPIC_0000001945223382__en-us_topic_0000001934020326_p769411169541">Number of index shards</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="EN-US_TOPIC_0000001945223382__en-us_topic_0000001934020326_row1769481685420"><td class="cellrowborder" valign="top" width="60%" headers="mcps1.3.4.2.3.4.2.3.1.1 "><p id="EN-US_TOPIC_0000001945223382__en-us_topic_0000001934020326_p66941716135418">number_of_replicas</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40%" headers="mcps1.3.4.2.3.4.2.3.1.2 "><p id="EN-US_TOPIC_0000001945223382__en-us_topic_0000001934020326_p126949169540">Number of index shard replicas</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="EN-US_TOPIC_0000001945223382__en-us_topic_0000001934020326_row1694161611544"><td class="cellrowborder" valign="top" width="60%" headers="mcps1.3.4.2.3.4.2.3.1.1 "><p id="EN-US_TOPIC_0000001945223382__en-us_topic_0000001934020326_p1369411620549">opendistro.index_state_management.policy_id</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40%" headers="mcps1.3.4.2.3.4.2.3.1.2 "><p id="EN-US_TOPIC_0000001945223382__en-us_topic_0000001934020326_p4694141616548">Lifecycle policy name</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</li><li id="EN-US_TOPIC_0000001945223382__en-us_topic_0000001934020326_li1812715651911">Create the <strong id="EN-US_TOPIC_0000001945223382__b197265595711">data-2022-06-06</strong> index. The index automatically uses the <strong id="EN-US_TOPIC_0000001945223382__b572685914718">template_hot_warm</strong> template and associates the index template with the lifecycle policy <strong id="EN-US_TOPIC_0000001945223382__b11727105920710">hot_warm_policy</strong>. As such, the index is automatically frozen in three days, and deleted in seven days after creation.<pre class="screen" id="EN-US_TOPIC_0000001945223382__en-us_topic_0000001934020326_screen0172542193514">POST data-2022-06-06/_bulk
|
|
{"index":{}}
|
|
{"name":"name1"}
|
|
{"index":{}}
|
|
{"name":"name2"}
|
|
{"index":{}}
|
|
{"name":"name3"}
|
|
{"index":{}}
|
|
{"name":"name4"}
|
|
{"index":{}}
|
|
{"name":"name5"}
|
|
{"index":{}}
|
|
{"name":"name6"}</pre>
|
|
</li><li id="EN-US_TOPIC_0000001945223382__en-us_topic_0000001934020326_li1097520318466">Query data and check whether storage and compute is automatically decoupled.<ul id="EN-US_TOPIC_0000001945223382__en-us_topic_0000001934020326_ul8362175954713"><li id="EN-US_TOPIC_0000001945223382__en-us_topic_0000001934020326_li1036255920479">Three days after the index is created, check the frozen index.<pre class="screen" id="EN-US_TOPIC_0000001945223382__en-us_topic_0000001934020326_screen19429203814233">GET _cat/freeze_indices?s=i&v</pre>
|
|
<p id="EN-US_TOPIC_0000001945223382__en-us_topic_0000001934020326_p19852165814220">The index generated three days ago is expected to be frozen.</p>
|
|
<pre class="screen" id="EN-US_TOPIC_0000001945223382__en-us_topic_0000001934020326_screen114951851172411">health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
|
|
green open <strong id="EN-US_TOPIC_0000001945223382__en-us_topic_0000001934020326_b19520181484710">data-2022-06-06</strong> x8ab5NX6T3Ox_xoGUanogQ 1 1 6 0 7.6kb 3.8kb</pre>
|
|
</li><li id="EN-US_TOPIC_0000001945223382__en-us_topic_0000001934020326_li1436295964717">Seven days after index creation, check the index. The index is expected to be deleted.</li></ul>
|
|
</li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="css_01_0091.html">Managing Index Policies for an Elasticsearch Cluster</a></div>
|
|
</div>
|
|
</div>
|
|
|