forked from docs/doc-exports
chenxiaoxiong
f9e2808b7c
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: chenxiaoxiong <chenxiaoxiong@huawei.com> Co-committed-by: chenxiaoxiong <chenxiaoxiong@huawei.com>
85 lines
13 KiB
HTML
85 lines
13 KiB
HTML
<a name="dataartsstudio_01_0071"></a><a name="dataartsstudio_01_0071"></a>
|
|
|
|
<h1 class="topictitle1">To Elasticsearch/CSS</h1>
|
|
<div id="body8662426"><p id="dataartsstudio_01_0071__en-us_topic_0108275347_p32842430161426">If the destination link of a job is a link described in <a href="dataartsstudio_01_1380.html">Elasticsearch Link Parameters</a> or <a href="dataartsstudio_01_0035.html">CSS Link Parameters</a>, configure the destination job parameters based on <a href="#dataartsstudio_01_0071__en-us_topic_0108275347_table5046103815165">Table 1</a>.</p>
|
|
<div class="notice" id="dataartsstudio_01_0071__en-us_topic_0108275347_note16186446106"><span class="noticetitle"><img src="public_sys-resources/notice_3.0-en-us.png"> </span><div class="noticebody"><p id="dataartsstudio_01_0071__en-us_topic_0108275347_p018704141018">The parameters required for table/file migration are different from those for entire DB migration. The following table lists the parameters for table/file migration. The actual parameters are subject to those displayed on the console.</p>
|
|
</div></div>
|
|
|
|
<div class="tablenoborder"><a name="dataartsstudio_01_0071__en-us_topic_0108275347_table5046103815165"></a><a name="en-us_topic_0108275347_table5046103815165"></a><table cellpadding="4" cellspacing="0" summary="" id="dataartsstudio_01_0071__en-us_topic_0108275347_table5046103815165" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Job parameters when Elasticsearch/CSS is the destination</caption><thead align="left"><tr id="dataartsstudio_01_0071__en-us_topic_0108275347_row585315215165"><th align="left" class="cellrowborder" valign="top" width="17.169999999999998%" id="mcps1.3.3.2.4.1.1"><p id="dataartsstudio_01_0071__en-us_topic_0108275347_p1626397215165">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="63.99%" id="mcps1.3.3.2.4.1.2"><p id="dataartsstudio_01_0071__en-us_topic_0108275347_p4231334915165">Description</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="18.84%" id="mcps1.3.3.2.4.1.3"><p id="dataartsstudio_01_0071__en-us_topic_0108275347_p482921015165">Example Value</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="dataartsstudio_01_0071__en-us_topic_0108275347_row4012116315165"><td class="cellrowborder" valign="top" width="17.169999999999998%" headers="mcps1.3.3.2.4.1.1 "><p id="dataartsstudio_01_0071__en-us_topic_0108275347_p2858877215165">Index</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="63.99%" headers="mcps1.3.3.2.4.1.2 "><p id="dataartsstudio_01_0071__en-us_topic_0108275347_p15492661577">Elasticsearch index, which is similar to the name of a relational database. <span id="dataartsstudio_01_0071__en-us_topic_0108275347_ph1997374045914">CDM supports automatic creation of indexes and field types. The index and field type names can contain only lowercase letters.</span></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="18.84%" headers="mcps1.3.3.2.4.1.3 "><p id="dataartsstudio_01_0071__en-us_topic_0108275347_p166427315165">index</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0071__en-us_topic_0108275347_row1497845915165"><td class="cellrowborder" valign="top" width="17.169999999999998%" headers="mcps1.3.3.2.4.1.1 "><p id="dataartsstudio_01_0071__en-us_topic_0108275347_p529563715165">Type</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="63.99%" headers="mcps1.3.3.2.4.1.2 "><p id="dataartsstudio_01_0071__en-us_topic_0108275347_p527763715824">Elasticsearch type, which is similar to the table name of a relational database. The type name can contain only lowercase letters.</p>
|
|
<div class="note" id="dataartsstudio_01_0071__en-us_topic_0108275347_note13864143991413"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dataartsstudio_01_0071__en-us_topic_0108275347_en-us_topic_0108275408_p47952026161812">Elasticsearch 7.x and later versions do not support custom types. Instead, only the <strong id="dataartsstudio_01_0071__en-us_topic_0108275347_en-us_topic_0108275408_b44102315505">_doc</strong> type can be used. In this case, this parameter does not take effect even if it is set.</p>
|
|
</div></div>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="18.84%" headers="mcps1.3.3.2.4.1.3 "><p id="dataartsstudio_01_0071__en-us_topic_0108275347_p3753014815165">type</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0071__en-us_topic_0108275347_row575714722812"><td class="cellrowborder" valign="top" width="17.169999999999998%" headers="mcps1.3.3.2.4.1.1 "><p id="dataartsstudio_01_0071__en-us_topic_0108275347_p1275744711282">Operation</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="63.99%" headers="mcps1.3.3.2.4.1.2 "><p id="dataartsstudio_01_0071__en-us_topic_0108275347_p1436017293019">Operation type</p>
|
|
<ul id="dataartsstudio_01_0071__en-us_topic_0108275347_ul1853857133619"><li id="dataartsstudio_01_0071__en-us_topic_0108275347_li1585317571365"><strong id="dataartsstudio_01_0071__en-us_topic_0108275347_b1264505319291">INDEX</strong>: No primary key is required. Elasticsearch generates IDs so that data is written to a new file with a unique ID for each write operation.</li><li id="dataartsstudio_01_0071__en-us_topic_0108275347_li4522311103715"><strong id="dataartsstudio_01_0071__en-us_topic_0108275347_b128092021134416">CREATE</strong>: A primary key needs to be specified. If the primary key already exists, the write operation fails.</li><li id="dataartsstudio_01_0071__en-us_topic_0108275347_li385314577369"><strong id="dataartsstudio_01_0071__en-us_topic_0108275347_b92884577447">UPDATE</strong>: A primary key needs to be specified. If the primary key already exists, the original data is overwritten.</li><li id="dataartsstudio_01_0071__en-us_topic_0108275347_li148535579364"><strong id="dataartsstudio_01_0071__en-us_topic_0108275347_b1941510153456">UPSERT</strong>: A primary key needs to be specified. If a primary key already exists, the existing data is overwritten. If there is no primary key, a new document is created for writing data.</li></ul>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="18.84%" headers="mcps1.3.3.2.4.1.3 "><p id="dataartsstudio_01_0071__en-us_topic_0108275347_p57575478285">INDEX</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0071__en-us_topic_0108275347_row1271559105513"><td class="cellrowborder" valign="top" width="17.169999999999998%" headers="mcps1.3.3.2.4.1.1 "><p id="dataartsstudio_01_0071__en-us_topic_0108275347_p57115925517">Pipeline ID</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="63.99%" headers="mcps1.3.3.2.4.1.2 "><p id="dataartsstudio_01_0071__en-us_topic_0108275347_p8755914553">ID of the pipeline used to convert the format of the data transferred to Elasticsearch.</p>
|
|
<p id="dataartsstudio_01_0071__en-us_topic_0108275347_p025918918317">If the destination is Elasticsearch, you need to create a pipeline ID in Kibana first.</p>
|
|
<p id="dataartsstudio_01_0071__en-us_topic_0108275347_p825999193116">If the destination is CSS, you do not need to create a pipeline ID. Instead, enter the name of the configuration file, which is <strong id="dataartsstudio_01_0071__en-us_topic_0108275347_b7528331314">name</strong> by default.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="18.84%" headers="mcps1.3.3.2.4.1.3 "><p id="dataartsstudio_01_0071__en-us_topic_0108275347_p370855117719">If the destination is Elasticsearch: pipeline_id</p>
|
|
<p id="dataartsstudio_01_0071__en-us_topic_0108275347_p66081252474">If the destination is CSS: <strong id="dataartsstudio_01_0071__en-us_topic_0108275347_b134927916335">name</strong> (name of the configuration file)</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0071__en-us_topic_0108275347_row19345155172412"><td class="cellrowborder" valign="top" width="17.169999999999998%" headers="mcps1.3.3.2.4.1.1 "><p id="dataartsstudio_01_0071__en-us_topic_0108275347_p23455514249">Write ES with Routing</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="63.99%" headers="mcps1.3.3.2.4.1.2 "><p id="dataartsstudio_01_0071__en-us_topic_0108275347_p310612415262">If you enable this function, a column can be written to Elasticsearch as a route.</p>
|
|
<div class="note" id="dataartsstudio_01_0071__en-us_topic_0108275347_note87558281261"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dataartsstudio_01_0071__en-us_topic_0108275347_p5345159244">Before enabling this function, create indexes at the destination to improve the query efficiency.</p>
|
|
</div></div>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="18.84%" headers="mcps1.3.3.2.4.1.3 "><p id="dataartsstudio_01_0071__en-us_topic_0108275347_p1934518532410">No</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0071__en-us_topic_0108275347_row9742198122412"><td class="cellrowborder" valign="top" width="17.169999999999998%" headers="mcps1.3.3.2.4.1.1 "><p id="dataartsstudio_01_0071__en-us_topic_0108275347_p20743198112417">Route Column</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="63.99%" headers="mcps1.3.3.2.4.1.2 "><p id="dataartsstudio_01_0071__en-us_topic_0108275347_p274368112418">This parameter is available when <strong id="dataartsstudio_01_0071__en-us_topic_0108275347_b1211113811573">Write ES with Routing</strong> is set to <strong id="dataartsstudio_01_0071__en-us_topic_0108275347_b2420344165712">Yes</strong>. It specifies the destination routing column. If the destination index exists but the column information cannot be obtained, you can manually enter the column. The route column can be empty. If it is empty, no routing value is specified for the data written to Elasticsearch.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="18.84%" headers="mcps1.3.3.2.4.1.3 "><p id="dataartsstudio_01_0071__en-us_topic_0108275347_p207431187248">value1</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0071__en-us_topic_0108275347_row1435112005414"><td class="cellrowborder" valign="top" width="17.169999999999998%" headers="mcps1.3.3.2.4.1.1 "><p id="dataartsstudio_01_0071__en-us_topic_0108275347_p935120165411">Periodically Create Index</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="63.99%" headers="mcps1.3.3.2.4.1.2 "><div class="p" id="dataartsstudio_01_0071__en-us_topic_0108275347_p53511015420">For streaming jobs that continuously write data to Elasticsearch, CDM periodically creates indexes and writes data to the indexes, which helps you delete expired data. The indexes can be created based on the following periods:<ul id="dataartsstudio_01_0071__en-us_topic_0108275347_ul112848823812"><li id="dataartsstudio_01_0071__en-us_topic_0108275347_li1228468113814"><strong id="dataartsstudio_01_0071__en-us_topic_0108275347_b11669615145316">Every hour</strong>: CDM creates indexes on the hour. The new indexes are named in the format of <em id="dataartsstudio_01_0071__en-us_topic_0108275347_i196691115165312">Index name+Year+Month+Day+Hour</em>, for example, <span class="parmvalue" id="dataartsstudio_01_0071__en-us_topic_0108275347_parmvalue1966916158537"><b>index2018121709</b></span>.</li><li id="dataartsstudio_01_0071__en-us_topic_0108275347_li163712311494"><strong id="dataartsstudio_01_0071__en-us_topic_0108275347_b146257316556">Every day</strong>: CDM creates indexes at 00:00 every day. The new indexes are named in the format of <em id="dataartsstudio_01_0071__en-us_topic_0108275347_i46251538554">Index name+Year+Month+Day</em>, for example, <span class="parmvalue" id="dataartsstudio_01_0071__en-us_topic_0108275347_parmvalue12625123175511"><b>index20181217</b></span>.</li><li id="dataartsstudio_01_0071__en-us_topic_0108275347_li118381542104917"><strong id="dataartsstudio_01_0071__en-us_topic_0108275347_b13625108175514">Every week</strong>: CDM creates indexes at 00:00 every Monday. The new indexes are named in the format of <em id="dataartsstudio_01_0071__en-us_topic_0108275347_i862517811552">Index name+Year+Week</em>, for example, <span class="parmvalue" id="dataartsstudio_01_0071__en-us_topic_0108275347_parmvalue0625208115513"><b>index201842</b></span>.</li><li id="dataartsstudio_01_0071__en-us_topic_0108275347_li879918301535"><strong id="dataartsstudio_01_0071__en-us_topic_0108275347_b44681114185517">Every month</strong>: CDM creates indexes at 00:00 on the first day of each month. The new indexes are named in the format of <em id="dataartsstudio_01_0071__en-us_topic_0108275347_i134681814115511">Index name+Year+Month</em>, for example, <span class="parmvalue" id="dataartsstudio_01_0071__en-us_topic_0108275347_parmvalue1146841475518"><b>index201812</b></span>.</li><li id="dataartsstudio_01_0071__en-us_topic_0108275347_li96271445145414"><strong id="dataartsstudio_01_0071__en-us_topic_0108275347_b168691831404">Do not create</strong>: Do not create indexes periodically.</li></ul>
|
|
</div>
|
|
<p id="dataartsstudio_01_0071__en-us_topic_0108275347_p837316411590">When extracting data from a file, you must configure a single extractor, which means setting <span class="parmname" id="dataartsstudio_01_0071__en-us_topic_0108275347_parmname25931019135516"><b>Concurrent Extractors</b></span> to <strong id="dataartsstudio_01_0071__en-us_topic_0108275347_b75931019185513">1</strong>. Otherwise, this parameter is invalid.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="18.84%" headers="mcps1.3.3.2.4.1.3 "><p id="dataartsstudio_01_0071__en-us_topic_0108275347_p19351806548">Every hour</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="dataartsstudio_01_0061.html">Configuring CDM Destination Job Parameters</a></div>
|
|
</div>
|
|
</div>
|
|
|