doc-exports/docs/css/umn/css_01_0024.html

<a name="EN-US_TOPIC_0000001938377776"></a><a name="EN-US_TOPIC_0000001938377776"></a>

<h1 class="topictitle1">Using Open Source Elasticsearch APIs to Import Data to Elasticsearch</h1>
<div id="body1526032894158"><p id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_p6252043134616">With <span id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_text182485785910">CSS</span>, you can use open-source Elasticsearch APIs on Kibana or an ECS server to import data to an Elasticsearch cluster. JSON files are supported.</p>
<ul id="EN-US_TOPIC_0000001938377776__ul195013127811"><li id="EN-US_TOPIC_0000001938377776__li195011612585"><a href="#EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_section1430231820400">Using an Open-Source Elasticsearch API to Import Data on Kibana</a>: Run POST commands to import data.</li><li id="EN-US_TOPIC_0000001938377776__li175011122080"><a href="#EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_section239718062912">Using an Open-Source Elasticsearch API to Import Data on an ECS Server</a>: Run cURL commands to import data.</li></ul>
<div class="section" id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_section1430231820400"><a name="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_section1430231820400"></a><a name="en-us_topic_0000001223594412_section1430231820400"></a><h4 class="sectiontitle">Using an Open-Source Elasticsearch API to Import Data on Kibana</h4><div class="p" id="EN-US_TOPIC_0000001938377776__p7201111917100">On Kibana, you can run POST commands to import single pieces of data using an open-source Elasticsearch API.<ol id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_ol2068013371496"><li id="EN-US_TOPIC_0000001938377776__li1313723121510">Log in to Kibana.<ol type="a" id="EN-US_TOPIC_0000001938377776__ol694442481517"><li id="EN-US_TOPIC_0000001938377776__en-us_topic_0000002338102386_en-us_topic_0000002271391393_en-us_topic_0000001268594549_li5509181942419"><span id="EN-US_TOPIC_0000001938377776__en-us_topic_0000002338102386_ph152259618315">Log in to the CSS management console.</span></li><li id="EN-US_TOPIC_0000001938377776__en-us_topic_0000002338102386_en-us_topic_0000002271391393_li4250043448">In the navigation pane on the left, choose <span class="uicontrol" id="EN-US_TOPIC_0000001938377776__en-us_topic_0000002338102386_uicontrol192689888185117"><b>Clusters &gt; Elasticsearch</b></span>.</li><li id="EN-US_TOPIC_0000001938377776__en-us_topic_0000002338102386_en-us_topic_0000002271391393_en-us_topic_0000001268154521_li6012554340">In the cluster list, find the target cluster, and click <strong id="EN-US_TOPIC_0000001938377776__en-us_topic_0000002338102386_b18928803044129">Kibana</strong> in the <strong id="EN-US_TOPIC_0000001938377776__en-us_topic_0000002338102386_b5255083364129">Operation</strong> column to log in to the Kibana console.</li><li id="EN-US_TOPIC_0000001938377776__en-us_topic_0000002338102386_en-us_topic_0000002271391393_li270075935613">In the left navigation pane, choose <strong id="EN-US_TOPIC_0000001938377776__en-us_topic_0000002338102386_b3927845824129">Dev Tools</strong>.<p id="EN-US_TOPIC_0000001938377776__en-us_topic_0000002338102386_p1810995434510">The left part of the console is the command input box, and the triangle icon in its upper-right corner is the execution button. The right part shows the execution result.</p>
</li></ol>
</li><li id="EN-US_TOPIC_0000001938377776__li167981434143611">Run the following command on the console to check whether the cluster has indexes:<pre class="screen" id="EN-US_TOPIC_0000001938377776__screen1778816833813">GET _cat/indices?v</pre>
<ul id="EN-US_TOPIC_0000001938377776__ul585911504127"><li id="EN-US_TOPIC_0000001938377776__li78591650181211">If indexes are available in the cluster to which you want to import data, you do not need to create an index. Go to <a href="#EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_li599201141619">7</a>.</li><li id="EN-US_TOPIC_0000001938377776__li78591550181212">If no indexes are available in the cluster, go to the next step to create an index.</li></ul>
</li><li id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_li1718421717426">Run the following command to create an index for storing imported data and create a custom mapping to define the data type.<p id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_p366432292117"><a name="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_li1718421717426"></a><a name="en-us_topic_0000001223594412_li1718421717426"></a>For example, run the following command to create index <span class="parmname" id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_parmname3407144082617"><b>my_store</b></span>:</p>
<p id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_p118001898274">Sample code for Elasticsearch 7.x or later:</p>
<pre class="screen" id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_screen1181163818278">PUT /<em id="EN-US_TOPIC_0000001938377776__i1754713447233">my_store</em>
{
    "settings": {
        "number_of_shards": 1
    },
    "mappings": {
        "properties": {
            "productName": {
                "type": "text"
            },
            "size": {
                "type": "keyword"
            }
        }
    }
}</pre>
<div class="p" id="EN-US_TOPIC_0000001938377776__p18170161691311">Sample code for Elasticsearch earlier than 7.x:<pre class="screen" id="EN-US_TOPIC_0000001938377776__screen1817051661313">PUT /<em id="EN-US_TOPIC_0000001938377776__i81701168138">my_store</em>
{
    "settings": {
        "number_of_shards": 1
    },
    "mappings": {
        "products": {
            "properties": {
                "productName": {
                    "type": "text"
                },
                "size": {
                    "type": "keyword"
                }
            }
        }
    }
}</pre>
</div>
</li><li id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_li599201141619"><a name="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_li599201141619"></a><a name="en-us_topic_0000001223594412_li599201141619"></a>Run commands on Kibana to import data. In the example below, only a single piece of data is imported:<p id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_p8754193443011"><a name="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_li599201141619"></a><a name="en-us_topic_0000001223594412_li599201141619"></a>Sample code for Elasticsearch 7.x or later:</p>
<pre class="screen" id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_screen184976293110">POST /my_store/_bulk  
{"index":{}}  
{"productName":"Latest art shirts for women in 2017 autumn","size":"L"}</pre>
<div class="p" id="EN-US_TOPIC_0000001938377776__p645144211135">Sample code for Elasticsearch earlier than 7.x:<pre class="screen" id="EN-US_TOPIC_0000001938377776__screen845254261317">POST /my_store/products/_bulk 
{"index":{}} 
{"productName":"Latest art shirts for women in 2017 autumn","size":"L"}</pre>
</div>
<p id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_p116351525183015">The command output is similar to that shown in <a href="#EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_fig86351225133018">Figure 1</a>. If the value of the <span class="parmname" id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_parmname19219201981614"><b>errors</b></span> field in the result is <span class="parmvalue" id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_parmvalue1321951912167"><b>false</b></span>, the data is successfully imported.</p>
<div class="fignone" id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_fig86351225133018"><a name="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_fig86351225133018"></a><a name="en-us_topic_0000001223594412_fig86351225133018"></a><span class="figcap"><b>Figure 1 </b>Response message</span><br><span><img id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_image19635132563018" src="figure/en-us_image_0000001938378084.png"></span></div>
</li></ol>
</div>
</div>
<div class="section" id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_section239718062912"><a name="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_section239718062912"></a><a name="en-us_topic_0000001223594412_section239718062912"></a><h4 class="sectiontitle">Using an Open-Source Elasticsearch API to Import Data on an ECS Server</h4><p id="EN-US_TOPIC_0000001938377776__p202850191256">On an ECS server, you can run cURL commands to use an open-source Elasticsearch API to import JSON files.</p>
<p id="EN-US_TOPIC_0000001938377776__p893852815257">In the example below, a cluster in non-security mode is used to describe how to import data using cURL commands. For the commands for a security cluster, see <a href="css_01_0063.html">Accessing an Elasticsearch Cluster Using cURL Commands</a>.</p>
<ol id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_ol4678613573"><li id="EN-US_TOPIC_0000001938377776__li17339257131514">Obtain the private network address of the cluster.<ol type="a" id="EN-US_TOPIC_0000001938377776__ol1368351111167"><li id="EN-US_TOPIC_0000001938377776__en-us_topic_0000002338102386_li962471018233"><span id="EN-US_TOPIC_0000001938377776__en-us_topic_0000002338102386_ph1162461016234">Log in to the CSS management console.</span></li><li id="EN-US_TOPIC_0000001938377776__en-us_topic_0000002338102386_li562441032314">In the navigation pane on the left, choose <span class="uicontrol" id="EN-US_TOPIC_0000001938377776__en-us_topic_0000002338102386_uicontrol206034223985117"><b>Clusters &gt; Elasticsearch</b></span>.</li><li id="EN-US_TOPIC_0000001938377776__en-us_topic_0000002338102386_en-us_topic_0000001961178833_li392216231538">In the cluster list, obtain the target cluster's private IP address from the <strong id="EN-US_TOPIC_0000001938377776__en-us_topic_0000002338102386_b7217183618175">Private IP Address</strong> column. Generally, the IP address format is <em id="EN-US_TOPIC_0000001938377776__en-us_topic_0000002338102386_i8377769238175">&lt;host&gt;</em>:<em id="EN-US_TOPIC_0000001938377776__en-us_topic_0000002338102386_i16969141938175">&lt;port&gt;</em> or <em id="EN-US_TOPIC_0000001938377776__en-us_topic_0000002338102386_i18383433328175">&lt;host&gt;</em>:<em id="EN-US_TOPIC_0000001938377776__en-us_topic_0000002338102386_i16522841718175">&lt;port&gt;</em>,<em id="EN-US_TOPIC_0000001938377776__en-us_topic_0000002338102386_i6783075698175">&lt;host&gt;</em>:<em id="EN-US_TOPIC_0000001938377776__en-us_topic_0000002338102386_i18005065588175">&lt;port&gt;</em>.<p id="EN-US_TOPIC_0000001938377776__en-us_topic_0000002338102386_en-us_topic_0000001961178833_p18422185125018">If the cluster has only one node, the IP address and port number of this single node are displayed, for example, <span class="parmvalue" id="EN-US_TOPIC_0000001938377776__en-us_topic_0000002338102386_en-us_topic_0000001961178833_parmvalue1916781025515"><b>10.62.179.32:9200</b></span>. If the cluster has multiple nodes and all of them are data nodes, the IP addresses and port numbers of all these nodes are displayed; if some of them are client nodes, only the IP addresses and port numbers of these client nodes are displayed; for example, <span class="parmvalue" id="EN-US_TOPIC_0000001938377776__en-us_topic_0000002338102386_en-us_topic_0000001961178833_parmvalue1425141583119"><b>10.62.179.32:9200,10.62.179.33:9200</b></span>.</p>
</li></ol>
</li><li id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_li1681812204612">Log in to the ECS that you are going to use to import data.</li><li id="EN-US_TOPIC_0000001938377776__li1241183810523">Upload a JSON file to the ECS.<p id="EN-US_TOPIC_0000001938377776__p796375616261"><a name="EN-US_TOPIC_0000001938377776__li1241183810523"></a><a name="li1241183810523"></a>For example, save the following data as a JSON file and upload the file to the ECS:</p>
<div class="p" id="EN-US_TOPIC_0000001938377776__p1244692118177">For Elasticsearch 7.x or later:<pre class="screen" id="EN-US_TOPIC_0000001938377776__screen144613217174">{"index": {"_index":"my_store"}}
{"productName":"Autumn new woman blouses 2019","size":"M"}
{"index": {"_index":"my_store"}}
{"productName":"Autumn new woman blouses 2019","size":"L"}</pre>
</div>
<div class="p" id="EN-US_TOPIC_0000001938377776__p1068371672712">For Elasticsearch earlier than 7.x:<pre class="screen" id="EN-US_TOPIC_0000001938377776__screen168311619278">{"index": {"_index":"my_store","_type":"products"}}
{"productName":"Autumn new woman blouses 2019","size":"M"}
{"index": {"_index":"my_store","_type":"products"}}
{"productName":"Autumn new woman blouses 2019","size":"L"}</pre>
</div>
</li><li id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_li74710325013">Run the following command in the path where the JSON file is stored in the ECS to import the JSON file to an Elasticsearch cluster.<pre class="screen" id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_screen14946155711418">curl -X PUT "http://{<em id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_i9475182012386">Private network address and port number of the node</em>}/_bulk" -H 'Content-Type: application/json' --data-binary @test.json</pre>
<p id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_p1126010231663">If communication encryption is enabled for the cluster where you are importing data, you need to send HTTPS requests by add <strong id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_b48412413319">-k</strong> to the cURL commands.</p>
<pre class="screen" id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_screen950417302617">curl -X PUT -k "https://{<em id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_i15935126102415">Private network address and port number of the node</em>}/_bulk" -H 'Content-Type: application/json' --data-binary @test.json</pre>
<p id="EN-US_TOPIC_0000001938377776__p7146183171910">Replace <em id="EN-US_TOPIC_0000001938377776__i7654202945910"><strong id="EN-US_TOPIC_0000001938377776__b18653152910590">{Private network address and port number of the node}</strong></em> with the private network address and port number of a node in the cluster. The value of the <strong id="EN-US_TOPIC_0000001938377776__b368983815596">-X</strong> parameter is a command and that of the <strong id="EN-US_TOPIC_0000001938377776__b368918384597">-H</strong> parameter is a message header. In the preceding command, <span class="parmvalue" id="EN-US_TOPIC_0000001938377776__parmvalue11689113815595"><b>PUT</b></span> is the value of the <strong id="EN-US_TOPIC_0000001938377776__b19690113819595">-X</strong> parameter and <span class="parmvalue" id="EN-US_TOPIC_0000001938377776__parmvalue869043895911"><b>'Content-Type: application/json' --data-binary @test.json</b></span> is the value of the <strong id="EN-US_TOPIC_0000001938377776__b569093811590">-H</strong> parameter. Do not add <strong id="EN-US_TOPIC_0000001938377776__b6872355145910">-k</strong> between a parameter and its value. <span class="filepath" id="EN-US_TOPIC_0000001938377776__filepath296616019020"><b>test.json</b></span> indicates the JSON file to be ingested.</p>
<div class="note" id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_note3364193816247"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="EN-US_TOPIC_0000001938377776__p1590019544184">In the case of the failure of a cluster node, if the cluster contains multiple nodes, you can replace {<em id="EN-US_TOPIC_0000001938377776__i653719121403">Private network address and port number of the node</em>} with the private network address and port number of any available node in the cluster; if the cluster contains only one node, restore this node and execute the command again.</p>
</div></div>
<p id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_p1441510225910"><strong id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_b460119369412">Example 1:</strong> In this example, assume that you need to import data in the <span class="filepath" id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_filepath175795113105214"><b>test.json</b></span> file to an Elasticsearch cluster, where communication encryption is disabled and the private network address and port number of one node are <span class="parmvalue" id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_parmvalue2112850572165410"><b>192.168.0.90</b></span> and <span class="parmvalue" id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_parmvalue1903804345105237"><b>9200</b></span> respectively.</p>
<ol type="a" id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_ol4472337122815"><li id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_li1992511339283">Run the following command to create an index named <strong id="EN-US_TOPIC_0000001938377776__b104201355250">my_store</strong>:<p id="EN-US_TOPIC_0000001938377776__p3566113842019">Sample code for Elasticsearch 7.x or later:</p>
<pre class="screen" id="EN-US_TOPIC_0000001938377776__screen11566183862015">curl -X PUT http://192.168.0.90:9200/my_store -H 'Content-Type: application/json' -d '
{
    "settings": {
        "number_of_shards": 1
    },
    "mappings": {
        "properties": {
            "productName": {
                "type": "text"
            },
            "size": {
                "type": "keyword"
            }
        }
    }
}'</pre>
<div class="p" id="EN-US_TOPIC_0000001938377776__p15456122115328">Sample code for Elasticsearch earlier than 7.x:<pre class="screen" id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_screen20693154120324">curl -X PUT http://192.168.0.90:9200/my_store -H 'Content-Type: application/json' -d '
 { 
   "settings": { 
     "number_of_shards": 1 
   }, 
   "mappings": { 
     "products": { 
       "properties": { 
         "productName": { 
           "type": "text" 
           }, 
         "size": { 
           "type": "keyword" 
         } 
       } 
     } 
   } 
 }'</pre>
</div>
</li><li id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_li63151641202810">Run the following command to import the <strong id="EN-US_TOPIC_0000001938377776__b610817211617">test.json</strong> file:<pre class="screen" id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_screen12455428121611">curl -X PUT "http://192.168.0.90:9200/_bulk" -H 'Content-Type: application/json' --data-binary @test.json</pre>
<p id="EN-US_TOPIC_0000001938377776__p1718013477396">In this case, if the following information is displayed, the data is successfully imported:</p>
<pre class="screen" id="EN-US_TOPIC_0000001938377776__screen492214294018">{"took":204,"errors":false,"items":[{"index":{"_index":"my_store","_type":"_doc","_id":"DJQkBIwBbJvUd2769Wi-","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1,"status":201}},{"index":{"_index":"my_store","_type":"_doc","_id":"DZQkBIwBbJvUd2769Wi_","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":1,"_primary_term":1,"status":201}}]}</pre>
</li></ol>
<div class="p" id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_p89581835184219"><strong id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_b209581335114214">Example 2:</strong> In this example, assume that you need to import data in the <span class="filepath" id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_filepath5959183510422"><b>test.json</b></span> file to an Elasticsearch cluster, where communication encryption has been enabled The private network address and port of one node are <strong id="EN-US_TOPIC_0000001938377776__b51926141478">192.168.0.90</strong> and <strong id="EN-US_TOPIC_0000001938377776__b0904122213718">9200</strong>, respectively.<ol type="a" id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_ol59631635184211"><li id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_li15965163514220">Run the following command to create an index named <strong id="EN-US_TOPIC_0000001938377776__b295933112714">my_store</strong>:<pre class="screen" id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_screen129651350429">curl -X PUT -k https://192.168.0.90:9200/my_store -H 'Content-Type: application/json' -d '
 { 
   "settings": { 
     "number_of_shards": 1 
   }, 
   "mappings": { 
     "products": { 
       "properties": { 
         "productName": { 
           "type": "text" 
           }, 
         "size": { 
           "type": "keyword" 
         } 
       } 
     } 
   } 
 }'</pre>
</li><li id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_li169761035204215">Run the following command to import the <strong id="EN-US_TOPIC_0000001938377776__b4927174012716">test.json</strong> file:<pre class="screen" id="EN-US_TOPIC_0000001938377776__en-us_topic_0000001223594412_screen1095155018491">curl -X PUT -k "https://192.168.0.90:9200/_bulk" -H 'Content-Type: application/json' --data-binary @test.json</pre>
<p id="EN-US_TOPIC_0000001938377776__p812654134010">In this case, if the following information is displayed, the data is successfully imported:</p>
<pre class="screen" id="EN-US_TOPIC_0000001938377776__screen1455933713459">{"took":204,"errors":false,"items":[{"index":{"_index":"my_store","_type":"_doc","_id":"DJQkBIwBbJvUd2769Wi-","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1,"status":201}},{"index":{"_index":"my_store","_type":"_doc","_id":"DZQkBIwBbJvUd2769Wi_","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":1,"_primary_term":1,"status":201}}]}</pre>
</li></ol>
</div>
</li></ol>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="css_01_0004.html">Importing Data to an Elasticsearch Cluster</a></div>
</div>
</div>