forked from docs/doc-exports
zhengxiu
2125539080
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: zhengxiu <zhengxiu@huawei.com> Co-committed-by: zhengxiu <zhengxiu@huawei.com>
99 lines
9.5 KiB
HTML
99 lines
9.5 KiB
HTML
<a name="css_01_0394"></a><a name="css_01_0394"></a>
|
|
|
|
<h1 class="topictitle1">Different Ways to Import Data to an Elasticsearch Cluster</h1>
|
|
<div id="body0000001945265112"><div class="section" id="css_01_0394__section1218985910454"><h4 class="sectiontitle">Introduction</h4><p id="css_01_0394__en-us_topic_0000001961259049_p5636738181213">Elasticsearch clusters support multiple data ingestion methods, as listed in <a href="#css_01_0394__en-us_topic_0000001961259049_table114913297713">Table 1</a>. Select one that fits your needs the best. Before starting to import data, determine whether to enhance the data import performance of Elasticsearch clusters first. For details, see <a href="css_01_0397.html">Enhancing the Data Import Performance of Elasticsearch Clusters</a>.</p>
|
|
|
|
<div class="tablenoborder"><a name="css_01_0394__en-us_topic_0000001961259049_table114913297713"></a><a name="en-us_topic_0000001961259049_table114913297713"></a><table cellpadding="4" cellspacing="0" summary="" id="css_01_0394__en-us_topic_0000001961259049_table114913297713" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Different ways to import data to an Elasticsearch cluster</caption><thead align="left"><tr id="css_01_0394__en-us_topic_0000001961259049_row914911291275"><th align="left" class="cellrowborder" valign="top" width="24.072407240724072%" id="mcps1.3.1.3.2.5.1.1"><p id="css_01_0394__en-us_topic_0000001961259049_p10149152919711">Data Import Method</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="28.992899289928992%" id="mcps1.3.1.3.2.5.1.2"><p id="css_01_0394__en-us_topic_0000001961259049_p41497296713">Scenario</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="12.73127312731273%" id="mcps1.3.1.3.2.5.1.3"><p id="css_01_0394__p10123334115516">Supported Data Formats</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="34.20342034203421%" id="mcps1.3.1.3.2.5.1.4"><p id="css_01_0394__en-us_topic_0000001961259049_p181496292716">Details</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="css_01_0394__en-us_topic_0000001961259049_row214982915714"><td class="cellrowborder" valign="top" width="24.072407240724072%" headers="mcps1.3.1.3.2.5.1.1 "><p id="css_01_0394__p11792181416384">Logstash data processing pipeline</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="28.992899289928992%" headers="mcps1.3.1.3.2.5.1.2 "><p id="css_01_0394__p1194151715616">Open-source Logstash offers a server-side, real-time data processing pipeline, which supports data ingestion from multiple sources. It can be used to collect various data, such as logs, monitoring data, and metrics.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="12.73127312731273%" headers="mcps1.3.1.3.2.5.1.3 "><p id="css_01_0394__p12123334205518">JSON, CSV, and text</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="34.20342034203421%" headers="mcps1.3.1.3.2.5.1.4 "><p id="css_01_0394__p6138135455019"><a href="css_01_0048.html">Using In-house Built Logstash to Import Data to Elasticsearch</a></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="css_01_0394__en-us_topic_0000001961259049_row123514502113"><td class="cellrowborder" valign="top" width="24.072407240724072%" headers="mcps1.3.1.3.2.5.1.1 "><p id="css_01_0394__p1378911419381">Open-source Elasticsearch API</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="28.992899289928992%" headers="mcps1.3.1.3.2.5.1.2 "><p id="css_01_0394__p26761016183811">Open-source Elasticsearch APIs can be used to import data. This method is flexible, as you can write your own application code.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="12.73127312731273%" headers="mcps1.3.1.3.2.5.1.3 "><p id="css_01_0394__p3123934165511">JSON</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="34.20342034203421%" headers="mcps1.3.1.3.2.5.1.4 "><p id="css_01_0394__p513811544509"><a href="css_01_0024.html">Using Open Source Elasticsearch APIs to Import Data to Elasticsearch</a></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="css_01_0394__en-us_topic_0000001961259049_row814919291979"><td class="cellrowborder" valign="top" width="24.072407240724072%" headers="mcps1.3.1.3.2.5.1.1 "><p id="css_01_0394__p1478961411380">Cloud Data Migration (CDM)</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="28.992899289928992%" headers="mcps1.3.1.3.2.5.1.2 "><p id="css_01_0394__p1678831418380">You can use CDM for batch data migration. For example, if data is stored in OBS or an Oracle database, CDM is recommended.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="12.73127312731273%" headers="mcps1.3.1.3.2.5.1.3 "><p id="css_01_0394__p81232034205515">JSON</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="34.20342034203421%" headers="mcps1.3.1.3.2.5.1.4 "><p id="css_01_0394__p3137754165015"><a href="css_01_0396.html">Using CDM to Import Data to Elasticsearch</a></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="css_01_0394__en-us_topic_0000001961259049_row61491729276"><td class="cellrowborder" valign="top" width="24.072407240724072%" headers="mcps1.3.1.3.2.5.1.1 "><p id="css_01_0394__p578751412380">Data Replication Service (DRS)</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="28.992899289928992%" headers="mcps1.3.1.3.2.5.1.2 "><p id="css_01_0394__p14859123412563">DRS can be used for online database migration and real-time data synchronization.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="12.73127312731273%" headers="mcps1.3.1.3.2.5.1.3 "><p id="css_01_0394__p1012317348559">Relational Database Service (RDS)</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="34.20342034203421%" headers="mcps1.3.1.3.2.5.1.4 "><p id="css_01_0394__p12136165419509"><a href="#css_01_0394__section1896852904516">Using DRS to Import Data from a Database to Elasticsearch</a></p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="css_01_0394__section1896852904516"><a name="css_01_0394__section1896852904516"></a><a name="section1896852904516"></a><h4 class="sectiontitle">Using DRS to Import Data from a Database to Elasticsearch</h4><p id="css_01_0394__p010114705211">DRS is an easy-to-use, stable, and efficient cloud service for online database migration and real-time database synchronization. Real-time data synchronization refers to the real-time replication of data from one source to another while ensuring data consistency.</p>
|
|
<p id="css_01_0394__p276917415521">DRS can be used to import data from multiple types of relational databases to Elasticsearch clusters. For details about the supported software versions for source databases and destination clusters, see <a href="#css_01_0394__table1099912411111">Table 2</a>.</p>
|
|
|
|
<div class="tablenoborder"><a name="css_01_0394__table1099912411111"></a><a name="table1099912411111"></a><table cellpadding="4" cellspacing="0" summary="" id="css_01_0394__table1099912411111" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Supported versions for using DRS to import data from a database to Elasticsearch</caption><thead align="left"><tr id="css_01_0394__row69991141101116"><th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.2.4.2.4.1.1"><p id="css_01_0394__p14999341161110">Scenario</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.2.4.2.4.1.2"><p id="css_01_0394__p1899916413113">Source DB</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.2.4.2.4.1.3"><p id="css_01_0394__p79991741191119">Destination Elasticsearch Cluster</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="css_01_0394__row7044291113"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.4.2.4.1.1 "><p id="css_01_0394__p146841547131217">Importing data from an RDS for MySQL database to a CSS Elasticsearch cluster</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.4.2.4.1.2 "><p id="css_01_0394__p40042171115">RDS for MySQL 5.5, 5.6, 5.7, or 8.0</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.4.2.4.1.3 "><p id="css_01_0394__p2802274138">Elasticsearch 5.5, 6.2, 6.5, 7.1, 7.6, 7.9, or 7.10</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="css_01_0394__row140042111110"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.4.2.4.1.1 "><p id="css_01_0394__p1956072019156">Importing data from a TaurusDB database to a CSS Elasticsearch cluster</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.4.2.4.1.2 "><p id="css_01_0394__p403429113">Primary/standby TaurusDB instances</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.4.2.4.1.3 "><p id="css_01_0394__p0084251117">Elasticsearch 5.5, 6.2, 6.5, 7.1, 7.6, 7.9, or 7.10</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="css_01_0394__row12064271119"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.4.2.4.1.1 "><p id="css_01_0394__p2015428116">Importing data from an in-house built MySQL database to a CSS Elasticsearch cluster</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.4.2.4.1.2 "><p id="css_01_0394__p182141424154314">MySQL database 5.5, 5.6, 5.7, or 8.0 created on a local server or ECS</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.4.2.4.1.3 "><p id="css_01_0394__p120842101117">Elasticsearch 5.5, 6.2, 6.5, 7.1, 7.6, 7.9, or 7.10</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="css_01_0045.html">Importing Data to an Elasticsearch Cluster</a></div>
|
|
</div>
|
|
</div>
|
|
|