forked from docs/doc-exports
zhengxiu
93d856d5c5
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: zhengxiu <zhengxiu@huawei.com> Co-committed-by: zhengxiu <zhengxiu@huawei.com>
182 lines
20 KiB
HTML
182 lines
20 KiB
HTML
<a name="EN-US_TOPIC_0000001955726554"></a><a name="EN-US_TOPIC_0000001955726554"></a>
|
|
|
|
<h1 class="topictitle1">Automating Index Rollover in an OpenSearch Cluster Through Index Lifecycle Management</h1>
|
|
<div id="body0000001955726554"><p id="EN-US_TOPIC_0000001955726554__p12161714133918">As time series data is continuously written into OpenSearch and the index size keeps growing, you can configure Index Lifecycle Management (ISM) to periodically roll over data to new indexes and delete old indexes.</p>
|
|
<p id="EN-US_TOPIC_0000001955726554__p10161101453919">In this example, a lifecycle policy is created for an OpenSearch cluster. According to this policy, when the size of an index reaches 1 TB or the index is one day old, a new index will be automatically generated; seven days after the index is created, data duplicates will be disabled; 30 days after the index is created, the index will be deleted.</p>
|
|
<p id="EN-US_TOPIC_0000001955726554__p18162111413395">Assume that an index generates approximately 2.4 TB of data each day. The index alias is <strong id="EN-US_TOPIC_0000001955726554__b158353666951213">log-alias</strong>. The following figure illustrates the data format in the cluster. During reads, it points to all indexes whose name starts with <strong id="EN-US_TOPIC_0000001955726554__b128333640951213">test</strong>. During writes, it points to the latest index.</p>
|
|
<div class="fignone" id="EN-US_TOPIC_0000001955726554__fig1016215149398"><span class="figcap"><b>Figure 1 </b>log-alias format</span><br><span><img id="EN-US_TOPIC_0000001955726554__image116291418393" src="figure/en-us_image_0000002209248269.png"></span></div>
|
|
<div class="note" id="EN-US_TOPIC_0000001955726554__note216281419394"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="EN-US_TOPIC_0000001955726554__p9162191413395">The one day in the rollover time refers to 24 hours following the index creation time, not a calendar day.</p>
|
|
</div></div>
|
|
<div class="section" id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_section860661817529"><h4 class="sectiontitle">Prerequisites</h4><p id="EN-US_TOPIC_0000001955726554__p18305753413">The target OpenSearch cluster is available.</p>
|
|
</div>
|
|
<div class="section" id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_section165662212527"><h4 class="sectiontitle">Automating Index Rollover Through Index Lifecycle Management</h4><ol id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_ol579517241688"><li id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001938218544_li3529193555517">Log in to the OpenSearch Dashboards.<ol type="a" id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001938218544_ol1949254165519"><li id="EN-US_TOPIC_0000001955726554__en-us_topic_0000002338102386_li111185453113"><span id="EN-US_TOPIC_0000001955726554__en-us_topic_0000002338102386_ph134648105316">Log in to the CSS management console.</span></li><li id="EN-US_TOPIC_0000001955726554__en-us_topic_0000002338102386_li1711804571114">In the navigation pane on the left, choose <span class="uicontrol" id="EN-US_TOPIC_0000001955726554__en-us_topic_0000002338102386_uicontrol1511834581115"><b>Clusters > OpenSearch</b></span>.</li><li id="EN-US_TOPIC_0000001955726554__en-us_topic_0000002338102386_li201181045151113">In the cluster list, find the target cluster, and click <strong id="EN-US_TOPIC_0000001955726554__en-us_topic_0000002338102386_b8963520674129">Dashboards</strong> in the <strong id="EN-US_TOPIC_0000001955726554__en-us_topic_0000002338102386_b1363282744129">Operation</strong> column to log in to OpenSearch Dashboards.</li><li id="EN-US_TOPIC_0000001955726554__en-us_topic_0000002338102386_li811824512111">In the left navigation pane, choose <strong id="EN-US_TOPIC_0000001955726554__en-us_topic_0000002338102386_b45261424710">Dev Tools</strong>.<p id="EN-US_TOPIC_0000001955726554__en-us_topic_0000002338102386_p539424154519">The left part of the console is the command input box, and the triangle icon in its upper-right corner is the execution button. The right part shows the execution result.</p>
|
|
</li></ol>
|
|
</li><li id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_li10320113081711">Create a rollover lifecycle policy named <strong id="EN-US_TOPIC_0000001955726554__b591769749105020">rollover_workflow</strong>.<p id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_p4629103417915">Policy description: When the size of an index reaches 1 TB or its age exceeds one day, index rollover is performed automatically. When the index age reaches seven days, data replicas are disabled. When it reaches 30 days, the index is deleted.</p>
|
|
<pre class="screen" id="EN-US_TOPIC_0000001955726554__screen106751241271">PUT _plugins/_ism/policies/rollover_workflow
|
|
{
|
|
"policy": {
|
|
"description": "rollover test",
|
|
"default_state": "hot",
|
|
"states": [
|
|
{
|
|
"name": "hot",
|
|
"actions": [
|
|
{
|
|
"rollover": {
|
|
"min_size": "1tb",
|
|
"min_index_age": "1d"
|
|
}
|
|
}
|
|
],
|
|
"transitions": [
|
|
{
|
|
"state_name": "warm",
|
|
"conditions": {
|
|
"min_index_age": "7d"
|
|
}
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"name": "warm",
|
|
"actions": [
|
|
{
|
|
"replica_count": {
|
|
"number_of_replicas": 0
|
|
}
|
|
}
|
|
],
|
|
"transitions": [
|
|
{
|
|
"state_name": "delete",
|
|
"conditions": {
|
|
"min_index_age": "30d"
|
|
}
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"name": "delete",
|
|
"actions": [
|
|
{
|
|
"delete": {}
|
|
}
|
|
]
|
|
}
|
|
],
|
|
"ism_template": {
|
|
"index_patterns": [
|
|
"test*"
|
|
],
|
|
"priority": 100
|
|
}
|
|
}
|
|
}</pre>
|
|
<p id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_p65629318127">After a lifecycle policy is created, run the following command to query the policy details:</p>
|
|
<pre class="screen" id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_screen5607153217121">GET _plugins/_ism/policies/rollover_workflow</pre>
|
|
</li><li id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_li5883193171813">Create the index template <strong id="EN-US_TOPIC_0000001955726554__b676563070105020">template_test</strong>.<p id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_p166221125141410">Template description: All the new indexes starting with <strong id="EN-US_TOPIC_0000001955726554__b2139159208105020">test</strong> are automatically associated with the rollover lifecycle policy <strong id="EN-US_TOPIC_0000001955726554__b1903680684105020">rollover_workflow</strong>. The alias <strong id="EN-US_TOPIC_0000001955726554__b1049168827105020">log_alias</strong> is used during rollover.</p>
|
|
<pre class="screen" id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_screen780113131714">PUT _template/template_test
|
|
{
|
|
"index_patterns": "<em id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_i2610131291720">test*</em>",
|
|
"settings": {
|
|
"number_of_replicas": 1,
|
|
"number_of_shards": 1,
|
|
"index.opendistro.index_state_management.rollover_alias": "<em id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_i510710477187">log_alias</em>"
|
|
},
|
|
"mappings": {
|
|
"properties": {
|
|
"name": {
|
|
"type": "text"
|
|
}
|
|
}
|
|
}
|
|
}</pre>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_table1784915717199" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_row138492718197"><th align="left" class="cellrowborder" valign="top" width="60%" id="mcps1.3.7.2.3.4.2.3.1.1"><p id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_p178491177191">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="40%" id="mcps1.3.7.2.3.4.2.3.1.2"><p id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_p1784987101913">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_row17849873194"><td class="cellrowborder" valign="top" width="60%" headers="mcps1.3.7.2.3.4.2.3.1.1 "><p id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_p13849174193">number_of_shards</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40%" headers="mcps1.3.7.2.3.4.2.3.1.2 "><p id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_p78498713193">Number of index shards</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_row1484927151916"><td class="cellrowborder" valign="top" width="60%" headers="mcps1.3.7.2.3.4.2.3.1.1 "><p id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_p18498761910">number_of_replicas</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40%" headers="mcps1.3.7.2.3.4.2.3.1.2 "><p id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_p58491375194">Number of index shard replicas</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_row16849173199"><td class="cellrowborder" valign="top" width="60%" headers="mcps1.3.7.2.3.4.2.3.1.1 "><p id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_p1184915781910">index.opendistro.index_state_management.rollover_alias</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40%" headers="mcps1.3.7.2.3.4.2.3.1.2 "><p id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_p11849187171911">Index alias for rollover</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
<p id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_p64871119131913">After an index template is created, you can run the following command to query the template details:</p>
|
|
<pre class="screen" id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_screen848716193195">GET _template/template_test</pre>
|
|
</li><li id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_li5274918111415"><a name="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_li5274918111415"></a><a name="en-us_topic_0000001945382698_en-us_topic_0000001961259029_li5274918111415"></a>Create an index, specify <span class="parmname" id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_parmname94951233164014"><b>aliases</b></span>, and set <span class="parmname" id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_parmname20836173713406"><b>is_write_index</b></span> to <span class="parmvalue" id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_parmvalue84341040154017"><b>true</b></span>. The index template <strong id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_b26564516410">template_test</strong> is automatically used for the index and is associated with the lifecycle policy <strong id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_b144641256543">rollover_workflow</strong> based on the index template configuration. In this way, when the index size reaches 1 TB or the index is created for more than one day, the rollover automatically starts. After an index is created for seven days, the data copy is disabled. After an index is created for 30 days, the index is deleted.<div class="p" id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_p76581151326">The following index is the URL code of <strong id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_b133851433359"><test-{now/d}-000001></strong>. By default, an index name contains the creation date. For example, if an index is created on 2022-06-02, the index name is <strong id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_b147141511618">test-2022.06.02-000001</strong>.<pre class="screen" id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_screen1433020481821">PUT %3Ctest-%7Bnow%2Fd%7D-000001%3E
|
|
{
|
|
"aliases": {
|
|
"log_alias": {
|
|
"is_write_index": true
|
|
}
|
|
}
|
|
}</pre>
|
|
</div>
|
|
</li><li id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_li18611837202017">The alias <strong id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_b412013409713">log_alias</strong> is used to during data write, and <strong id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_b452610211685">log_alias</strong> always points to the last index.<pre class="screen" id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_screen201821127135511">POST log_alias/_bulk
|
|
{"index":{}}
|
|
{"name":"name1"}
|
|
{"index":{}}
|
|
{"name":"name2"}
|
|
{"index":{}}
|
|
{"name":"name3"}
|
|
{"index":{}}
|
|
{"name":"name4"}
|
|
{"index":{}}
|
|
{"name":"name5"}
|
|
{"index":{}}
|
|
{"name":"name6"}</pre>
|
|
</li><li id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_li34031933195512">Query data and check whether the rollover has taken effect.<ul id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_ul172254014561"><li id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_li11401933415">One day after the indexes are created, check the indexes starting with <strong id="EN-US_TOPIC_0000001955726554__b19983263718">test</strong>.<pre class="screen" id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_screen2058713081019">GET _cat/indices/<em id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_i963041112376">test</em>*?s=i</pre>
|
|
<p id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_p677117151674">There are supposed to be at least two indexes, for example:</p>
|
|
<pre class="screen" id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_screen351814712117">green open test-<Date>-000001 r8ab5NX6T3Ox_hoGUanogQ 1 1 6 0 416b 208b
|
|
green open test-<Date>-000002 sfwkVgy8RSSEw7W-xYjM2Q 1 1 0 0 209b 209b</pre>
|
|
<p id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_p4672144131015">In the preceding information, <strong id="EN-US_TOPIC_0000001955726554__b1027155032105021">test-<Date>-000001</strong> is the index created in <a href="#EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_li5274918111415">4</a>, and <strong id="EN-US_TOPIC_0000001955726554__b993548890105021">test-<Date>-000002</strong> is the index generated through rollover.</p>
|
|
</li><li id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_li101498401729">To query the index associated with the alias <strong id="EN-US_TOPIC_0000001955726554__b662655583105021">log_alias</strong>, run the following command:<pre class="screen" id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_screen1211423418159">GET _cat/aliases/log_alias?v</pre>
|
|
<p id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_p1478018532148">The alias is supposed to point to multiple indexes, for example:</p>
|
|
<pre class="screen" id="EN-US_TOPIC_0000001955726554__en-us_topic_0000001945382698_en-us_topic_0000001961259029_screen4664354161511">alias index filter routing.index routing.search is_write_index
|
|
log_alias test-<Date>-000001 - - - false
|
|
log_alias test-<Date>-000002 - - - true</pre>
|
|
</li></ul>
|
|
</li></ol>
|
|
</div>
|
|
<div class="section" id="EN-US_TOPIC_0000001955726554__section8379144943118"><h4 class="sectiontitle">FAQ: How Do I Skip Rollover for an Index?</h4><p id="EN-US_TOPIC_0000001955726554__p11132734510"><strong id="EN-US_TOPIC_0000001955726554__b144861321135213">Scenarios</strong></p>
|
|
<ul id="EN-US_TOPIC_0000001955726554__ul15648122515526"><li id="EN-US_TOPIC_0000001955726554__li146488251525">A manual rollover is performed: A time-based ISM policy was configured to automate index rollover. Before the rollover criteria were met, however, a manual rollover was performed. When the auto rollover time configured by ISM was reached, an automatic rollover was attempted but failed because the index had already been rolled over manually. As a result, the ISM task stopped. To solve this problem, you need to skip the index rollover and then retry the ISM task to resume automatic index rollover.</li><li id="EN-US_TOPIC_0000001955726554__li27141328145213">For an index that has already been rolled over automatically, a remove or add policy operation is performed: After the rollover policy is modified (through the remove policy or add policy operation), the ISM task restarts from the beginning. When rollover starts, the task fails. To solve this, you also need to skip index rollover and then retry the ISM task to resume automatic index rollover.</li></ul>
|
|
<div class="warning" id="EN-US_TOPIC_0000001955726554__note135721048154916"><span class="warningtitle"><img src="public_sys-resources/warning_3.0-en-us.png"> </span><div class="warningbody"><p id="EN-US_TOPIC_0000001955726554__p165721548124913">After rollover is skipped for an index, ISM will no longer attempt rollover or generate rolled-over indexes. This means skipping index rollover may lead to data loss. Please exercise caution.</p>
|
|
</div></div>
|
|
<p id="EN-US_TOPIC_0000001955726554__p83331521183615"><strong id="EN-US_TOPIC_0000001955726554__b549120322085">Procedure</strong></p>
|
|
<ol id="EN-US_TOPIC_0000001955726554__ol844901412108"><li id="EN-US_TOPIC_0000001955726554__li24495142101">If an ISM task stops due to an index rollover failure, run the following command to skip index rollover:<pre class="screen" id="EN-US_TOPIC_0000001955726554__screen99042916477">PUT <em id="EN-US_TOPIC_0000001955726554__i118956515111">index_name</em>/_settings
|
|
{
|
|
"index.plugins.index_state_management.rollover_skip": true
|
|
}</pre>
|
|
<p id="EN-US_TOPIC_0000001955726554__p1220865916137">If <span class="parmvalue" id="EN-US_TOPIC_0000001955726554__parmvalue1929214851420"><b>true</b></span> is returned, the configuration is successful.</p>
|
|
</li><li id="EN-US_TOPIC_0000001955726554__li1882122011112">After index rollover is skipped, run the following command to retry the ISM task:<pre class="screen" id="EN-US_TOPIC_0000001955726554__screen197151304486">POST _plugins/_ism/retry/<em id="EN-US_TOPIC_0000001955726554__i119917207139">index_name</em></pre>
|
|
<p id="EN-US_TOPIC_0000001955726554__p7985155984516">If the following information is returned, the retry is successful:</p>
|
|
<pre class="screen" id="EN-US_TOPIC_0000001955726554__screen26608274198">{
|
|
"updated_indices": 1,
|
|
"failures": false,
|
|
"failed_indices": []
|
|
}</pre>
|
|
</li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="css_01_0320.html">Managing Index Policies for OpenSearch Clusters</a></div>
|
|
</div>
|
|
</div>
|
|
|