forked from docs/doc-exports
Hasko, Vladimir
cfc48b3aed
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-committed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
210 lines
25 KiB
HTML
210 lines
25 KiB
HTML
<a name="dli_09_0122"></a><a name="dli_09_0122"></a>
|
|
|
|
<h1 class="topictitle1">Using Spark-submit to Submit a Spark Jar Job</h1>
|
|
<div id="body8662426"><div class="section" id="dli_09_0122__en-us_topic_0192429929_section159401845155918"><h4 class="sectiontitle">Introduction to DLI Spark-submit</h4><p id="dli_09_0122__en-us_topic_0192429929_p155950531597">DLI Spark-submit is a command line tool used to submit Spark jobs to the DLI server. This tool provides command lines compatible with open-source Spark.</p>
|
|
</div>
|
|
<div class="section" id="dli_09_0122__section12581831432"><h4 class="sectiontitle">Preparations</h4><ol id="dli_09_0122__dli_09_0126_ol189969321298"><li id="dli_09_0122__dli_09_0126_li149961532192916">Getting authorized.<p id="dli_09_0122__dli_09_0126_p16317291479"><a name="dli_09_0122__dli_09_0126_li149961532192916"></a><a name="dli_09_0126_li149961532192916"></a>DLI uses the Identity and Access Management (IAM) to implement fine-grained permissions for your enterprise-level tenants. IAM provides identity authentication, permissions management, and access control, helping you securely access your resources.</p>
|
|
<p id="dli_09_0122__dli_09_0126_p188991750134717">With IAM, you can use your account to create IAM users for your employees, and assign permissions to the users to control their access to specific resource types. </p>
|
|
<p id="dli_09_0122__dli_09_0126_p13200194518338">Currently, roles (coarse-grained authorization) and policies (fine-grained authorization) are supported. </p>
|
|
</li><li id="dli_09_0122__dli_09_0126_li714994042910">Create a queue. Choose <strong id="dli_09_0122__dli_09_0126_b544120341765">Resources</strong> > <strong id="dli_09_0122__dli_09_0126_b14660163713613">Queue Management</strong>. On the page displayed, click <strong id="dli_09_0122__dli_09_0126_b132671161576">Buy Queue</strong> in the upper right corner. On the <strong id="dli_09_0122__dli_09_0126_b1754622216715">Buy Queue</strong> page displayed, select <strong id="dli_09_0122__dli_09_0126_b131631011983">For general purpose</strong> for <strong id="dli_09_0122__dli_09_0126_b82061574716">Type</strong>, that is, the compute resources of the Spark job.<div class="note" id="dli_09_0122__dli_09_0126_note1970895364011"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_09_0122__dli_09_0126_p1717716210156">If the user who creates the queue is not an administrator, the queue can be used only after being authorized by the administrator. For details about how to assign permissions, see .</p>
|
|
</div></div>
|
|
</li></ol>
|
|
</div>
|
|
<div class="section" id="dli_09_0122__en-us_topic_0192429929_section22631635783"><h4 class="sectiontitle">Downloading the DLI Client Tool</h4><p id="dli_09_0122__en-us_topic_0192429929_p152998521105">You can download the DLI client tool from the DLI management console.</p>
|
|
<ol id="dli_09_0122__en-us_topic_0192429929_ol11426753162256"><li id="dli_09_0122__en-us_topic_0192429929_li19719768162256"><span>Log in to the DLI management console.</span></li><li id="dli_09_0122__en-us_topic_0192429929_li230712530303"><span>Obtain the SDK download address from the administrator.</span></li><li id="dli_09_0122__en-us_topic_0192429929_li168171617259"><span>On the <strong id="dli_09_0122__b15974152216381">DLI SDK DOWNLOAD</strong> page, click <strong id="dli_09_0122__b20975922123816">dli-clientkit-<</strong><em id="dli_09_0122__i1897572218388">version</em><strong id="dli_09_0122__b16976822173812">></strong> to download the DLI client tool.</span><p><div class="note" id="dli_09_0122__en-us_topic_0192429929_note618619117343"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_09_0122__en-us_topic_0192429929_p91861214349">The Beeline client is named <strong id="dli_09_0122__b11511172553812">dli-clientkit-<</strong><em id="dli_09_0122__i8511192593816">version</em><strong id="dli_09_0122__b35119252386">>-bin.tar.gz</strong>, which can be used in Linux and depends on JDK 1.8 or later.</p>
|
|
</div></div>
|
|
</p></li></ol>
|
|
</div>
|
|
<div class="section" id="dli_09_0122__en-us_topic_0192429929_section203255161411"><h4 class="sectiontitle">Configuring DLI Spark-submit</h4><p id="dli_09_0122__en-us_topic_0192429929_p569384916116">Ensure that you have installed JDK 1.8 or later and configured environment variables on the computer where spark-submit is installed. You are advised to use spark-submit on the computer running Linux.</p>
|
|
</div>
|
|
<ol id="dli_09_0122__en-us_topic_0192429929_ol36912088143945"><li id="dli_09_0122__en-us_topic_0192429929_li19848140144118"><span>Download and decompress <strong id="dli_09_0122__b1966004812404">dli-clientkit-<</strong><em id="dli_09_0122__i26604489408">version</em><strong id="dli_09_0122__b6660134824019">>-bin.tar.gz</strong>. In this step, set <em id="dli_09_0122__i3661174810401">version</em> to the actual version.</span></li><li id="dli_09_0122__en-us_topic_0192429929_li47233974144230"><span>Go to the directory where <strong id="dli_09_0122__b7908104520243">dli-clientkit-<version>-bin.tar.gz</strong> is decompressed. In the directory, there are three subdirectories <strong id="dli_09_0122__b490812459241">bin</strong>, <strong id="dli_09_0122__b1090816451242">conf</strong>, and <strong id="dli_09_0122__b12908184514248">lib</strong>, which respectively store the execution scripts, configuration files, and dependency packages related to <strong id="dli_09_0122__b89081645162418">Spark-submit</strong>.</span></li><li id="dli_09_0122__en-us_topic_0192429929_li64195479144118"><span>Go to the <strong id="dli_09_0122__b797382616101">conf</strong> directory and modify the configuration items in the <strong id="dli_09_0122__b1197316268107">client.properties</strong> file. For details about the configuration items, see <a href="#dli_09_0122__en-us_topic_0192429929_table571552873114">Table 1</a>.</span><p>
|
|
<div class="tablenoborder"><a name="dli_09_0122__en-us_topic_0192429929_table571552873114"></a><a name="en-us_topic_0192429929_table571552873114"></a><table cellpadding="4" cellspacing="0" summary="" id="dli_09_0122__en-us_topic_0192429929_table571552873114" frame="border" border="1" rules="all"><caption><b>Table 1 </b>DLI client parameters</caption><thead align="left"><tr id="dli_09_0122__en-us_topic_0192429929_row10713182823115"><th align="left" class="cellrowborder" valign="top" width="15.02%" id="mcps1.3.5.3.2.1.2.5.1.1"><p id="dli_09_0122__en-us_topic_0192429929_p27119155155349">Item</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="11.35%" id="mcps1.3.5.3.2.1.2.5.1.2"><p id="dli_09_0122__en-us_topic_0192429929_p49167942155349">Mandatory</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="22.18%" id="mcps1.3.5.3.2.1.2.5.1.3"><p id="dli_09_0122__en-us_topic_0192429929_p23180374155349">Default Value</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="51.449999999999996%" id="mcps1.3.5.3.2.1.2.5.1.4"><p id="dli_09_0122__en-us_topic_0192429929_p65670986155349">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="dli_09_0122__en-us_topic_0192429929_row765513114239"><td class="cellrowborder" valign="top" width="15.02%" headers="mcps1.3.5.3.2.1.2.5.1.1 "><p id="dli_09_0122__en-us_topic_0192429929_p865661182312">dliEndPont</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="11.35%" headers="mcps1.3.5.3.2.1.2.5.1.2 "><p id="dli_09_0122__en-us_topic_0192429929_p106578110234">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="22.18%" headers="mcps1.3.5.3.2.1.2.5.1.3 "><p id="dli_09_0122__en-us_topic_0192429929_p136572116239">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="51.449999999999996%" headers="mcps1.3.5.3.2.1.2.5.1.4 "><p id="dli_09_0122__p18210539189">Domain name of DLI</p>
|
|
<p id="dli_09_0122__en-us_topic_0192429929_p312510316239">If you lef this parameter empty, the program determines the domain name based on <strong id="dli_09_0122__b6633622123119">region</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_09_0122__en-us_topic_0192429929_row27131428163110"><td class="cellrowborder" valign="top" width="15.02%" headers="mcps1.3.5.3.2.1.2.5.1.1 "><p id="dli_09_0122__en-us_topic_0192429929_p371342817318">obsEndPoint</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="11.35%" headers="mcps1.3.5.3.2.1.2.5.1.2 "><p id="dli_09_0122__en-us_topic_0192429929_p971342818315">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="22.18%" headers="mcps1.3.5.3.2.1.2.5.1.3 "><p id="dli_09_0122__p163183171019">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="51.449999999999996%" headers="mcps1.3.5.3.2.1.2.5.1.4 "><p id="dli_09_0122__en-us_topic_0192429929_p47131028203117">OBS service domain name.</p>
|
|
<p id="dli_09_0122__p14442163782513">Obtain the OBS domain name from the administrator.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_09_0122__en-us_topic_0192429929_row1771562811317"><td class="cellrowborder" valign="top" width="15.02%" headers="mcps1.3.5.3.2.1.2.5.1.1 "><p id="dli_09_0122__en-us_topic_0192429929_p1071316281314">bucketName</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="11.35%" headers="mcps1.3.5.3.2.1.2.5.1.2 "><p id="dli_09_0122__en-us_topic_0192429929_p107151228113118">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="22.18%" headers="mcps1.3.5.3.2.1.2.5.1.3 "><p id="dli_09_0122__en-us_topic_0192429929_p127151228103110">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="51.449999999999996%" headers="mcps1.3.5.3.2.1.2.5.1.4 "><p id="dli_09_0122__en-us_topic_0192429929_p3715152803112">Name of a bucket on OBS. This bucket is used to store JAR files, Python program files, and configuration files used in Spark programs.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_09_0122__en-us_topic_0192429929_row5715128163118"><td class="cellrowborder" valign="top" width="15.02%" headers="mcps1.3.5.3.2.1.2.5.1.1 "><p id="dli_09_0122__en-us_topic_0192429929_p5715132883110">obsPath</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="11.35%" headers="mcps1.3.5.3.2.1.2.5.1.2 "><p id="dli_09_0122__en-us_topic_0192429929_p13715102811312">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="22.18%" headers="mcps1.3.5.3.2.1.2.5.1.3 "><p id="dli_09_0122__en-us_topic_0192429929_p67151281317">dli-spark-submit-resources</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="51.449999999999996%" headers="mcps1.3.5.3.2.1.2.5.1.4 "><p id="dli_09_0122__en-us_topic_0192429929_p1771572819316">Directory for storing JAR files, Python program files, and configuration files on OBS. The directory is in the bucket specified by <strong id="dli_09_0122__b38561343402">Bucket Name</strong>. If the directory does not exist, the program automatically creates it.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_09_0122__en-us_topic_0192429929_row147502589331"><td class="cellrowborder" valign="top" width="15.02%" headers="mcps1.3.5.3.2.1.2.5.1.1 "><p id="dli_09_0122__en-us_topic_0192429929_p20751858133310">localFilePath</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="11.35%" headers="mcps1.3.5.3.2.1.2.5.1.2 "><p id="dli_09_0122__en-us_topic_0192429929_p97517581338">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="22.18%" headers="mcps1.3.5.3.2.1.2.5.1.3 "><p id="dli_09_0122__en-us_topic_0192429929_p137511558123313">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="51.449999999999996%" headers="mcps1.3.5.3.2.1.2.5.1.4 "><p id="dli_09_0122__en-us_topic_0192429929_p1947213712368">The local directory for storing JAR files, Python program files, and configuration files used in Spark programs.</p>
|
|
<p id="dli_09_0122__en-us_topic_0192429929_p47524581335">The program automatically uploads the files on which Spark depends to the OBS path and loads them to the resource package on the DLI server.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_09_0122__en-us_topic_0192429929_row4715182812316"><td class="cellrowborder" valign="top" width="15.02%" headers="mcps1.3.5.3.2.1.2.5.1.1 "><p id="dli_09_0122__en-us_topic_0192429929_p1715182816312">ak</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="11.35%" headers="mcps1.3.5.3.2.1.2.5.1.2 "><p id="dli_09_0122__en-us_topic_0192429929_p9715182863112">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="22.18%" headers="mcps1.3.5.3.2.1.2.5.1.3 "><p id="dli_09_0122__en-us_topic_0192429929_p9715528123112">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="51.449999999999996%" headers="mcps1.3.5.3.2.1.2.5.1.4 "><p id="dli_09_0122__en-us_topic_0192429929_p2715192883114">User's Access Key (AK)</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_09_0122__en-us_topic_0192429929_row12280456104419"><td class="cellrowborder" valign="top" width="15.02%" headers="mcps1.3.5.3.2.1.2.5.1.1 "><p id="dli_09_0122__en-us_topic_0192429929_p7281125694415">sk</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="11.35%" headers="mcps1.3.5.3.2.1.2.5.1.2 "><p id="dli_09_0122__en-us_topic_0192429929_p628145604410">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="22.18%" headers="mcps1.3.5.3.2.1.2.5.1.3 "><p id="dli_09_0122__en-us_topic_0192429929_p6281185644413">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="51.449999999999996%" headers="mcps1.3.5.3.2.1.2.5.1.4 "><p id="dli_09_0122__en-us_topic_0192429929_p22811756124410">User's Secret Key (SK)</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_09_0122__en-us_topic_0192429929_row596810814520"><td class="cellrowborder" valign="top" width="15.02%" headers="mcps1.3.5.3.2.1.2.5.1.1 "><p id="dli_09_0122__en-us_topic_0192429929_p14968589452">projectId</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="11.35%" headers="mcps1.3.5.3.2.1.2.5.1.2 "><p id="dli_09_0122__en-us_topic_0192429929_p1696910854519">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="22.18%" headers="mcps1.3.5.3.2.1.2.5.1.3 "><p id="dli_09_0122__en-us_topic_0192429929_p7969381452">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="51.449999999999996%" headers="mcps1.3.5.3.2.1.2.5.1.4 "><p id="dli_09_0122__en-us_topic_0192429929_p996918154519">Project ID used by a user to access DLI.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_09_0122__en-us_topic_0192429929_row10516594514"><td class="cellrowborder" valign="top" width="15.02%" headers="mcps1.3.5.3.2.1.2.5.1.1 "><p id="dli_09_0122__en-us_topic_0192429929_p16545104517">region</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="11.35%" headers="mcps1.3.5.3.2.1.2.5.1.2 "><p id="dli_09_0122__en-us_topic_0192429929_p2611518456">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="22.18%" headers="mcps1.3.5.3.2.1.2.5.1.3 "><p id="dli_09_0122__en-us_topic_0192429929_p2619534518">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="51.449999999999996%" headers="mcps1.3.5.3.2.1.2.5.1.4 "><p id="dli_09_0122__en-us_topic_0192429929_p10645104510">Region of interconnected DLI.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
<p id="dli_09_0122__en-us_topic_0192429929_p497673619291">Modify the configuration items in the <strong id="dli_09_0122__b993374511247">spark-defaults.conf</strong> file based on the Spark application requirements. The configuration items are compatible with the open-source Spark configuration items. For details, see the open-source Spark configuration item description.</p>
|
|
</p></li></ol>
|
|
<div class="section" id="dli_09_0122__en-us_topic_0192429929_section223712146331"><h4 class="sectiontitle">Using Spark-submit to Submit a Spark Job</h4><ol id="dli_09_0122__en-us_topic_0192429929_ol11256733143711"><li id="dli_09_0122__en-us_topic_0192429929_li49999422419"><span>Go to the <strong id="dli_09_0122__b199341545162410">bin</strong> directory of the tool file, run the <strong id="dli_09_0122__b19341745102419">spark-submit</strong> command, and carry related parameters.</span><p><div class="p" id="dli_09_0122__en-us_topic_0192429929_p140134484116">The command format is as follows:<pre class="screen" id="dli_09_0122__en-us_topic_0192429929_screen34136240151350">spark-submit [options] <app jar | python file> [app arguments]</pre>
|
|
</div>
|
|
<div class="p" id="dli_09_0122__en-us_topic_0192429929_p1181134424216">
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_09_0122__en-us_topic_0192429929_table7810194416426" frame="border" border="1" rules="all"><caption><b>Table 2 </b>DLI Spark-submit parameters</caption><thead align="left"><tr id="dli_09_0122__en-us_topic_0192429929_row180874434217"><th align="left" class="cellrowborder" valign="top" width="17.59%" id="mcps1.3.6.2.1.2.2.1.2.4.1.1"><p id="dli_09_0122__en-us_topic_0192429929_p128085447429"><strong id="dli_09_0122__b188603464012">Parameter</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="21.83%" id="mcps1.3.6.2.1.2.2.1.2.4.1.2"><p id="dli_09_0122__en-us_topic_0192429929_p20808174410425"><strong id="dli_09_0122__b986113413401">Value</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="60.58%" id="mcps1.3.6.2.1.2.2.1.2.4.1.3"><p id="dli_09_0122__en-us_topic_0192429929_p14808194434219"><strong id="dli_09_0122__en-us_topic_0106891196_b84235270693349">Description</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="dli_09_0122__en-us_topic_0192429929_row6809124404213"><td class="cellrowborder" valign="top" width="17.59%" headers="mcps1.3.6.2.1.2.2.1.2.4.1.1 "><p id="dli_09_0122__en-us_topic_0192429929_p480814454218">--class</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="21.83%" headers="mcps1.3.6.2.1.2.2.1.2.4.1.2 "><p id="dli_09_0122__en-us_topic_0192429929_p19808134424215"><CLASS_NAME></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.58%" headers="mcps1.3.6.2.1.2.2.1.2.4.1.3 "><p id="dli_09_0122__en-us_topic_0192429929_p857611954917">Name of the main class of the submitted Java or Scala application.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_09_0122__en-us_topic_0192429929_row181017443420"><td class="cellrowborder" valign="top" width="17.59%" headers="mcps1.3.6.2.1.2.2.1.2.4.1.1 "><p id="dli_09_0122__en-us_topic_0192429929_p180954484210">--conf</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="21.83%" headers="mcps1.3.6.2.1.2.2.1.2.4.1.2 "><p id="dli_09_0122__en-us_topic_0192429929_p128091544154216"><PROP=VALUE></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.58%" headers="mcps1.3.6.2.1.2.2.1.2.4.1.3 "><p id="dli_09_0122__en-us_topic_0192429929_p18573919184914">Spark program parameters can be configured in the <strong id="dli_09_0122__b1786114418409">spark-defaults.conf</strong> file in the <strong id="dli_09_0122__b1786115474015">conf</strong> directory. If both the command and the configuration file are configured, the parameter value specified in the command is preferentially used.</p>
|
|
<div class="note" id="dli_09_0122__en-us_topic_0192429929_note81016135509"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dli_09_0122__en-us_topic_0192429929_p311113145012">If there are multiple <strong id="dli_09_0122__b1286219416404">conf</strong> files, the format is <strong id="dli_09_0122__b486220444016">--conf key1=value1 --conf key2=value2</strong>.</p>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_09_0122__en-us_topic_0192429929_row1081094419421"><td class="cellrowborder" valign="top" width="17.59%" headers="mcps1.3.6.2.1.2.2.1.2.4.1.1 "><p id="dli_09_0122__en-us_topic_0192429929_p2810164444211">--jars</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="21.83%" headers="mcps1.3.6.2.1.2.2.1.2.4.1.2 "><p id="dli_09_0122__en-us_topic_0192429929_p7810124416429"><JARS></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.58%" headers="mcps1.3.6.2.1.2.2.1.2.4.1.3 "><p id="dli_09_0122__en-us_topic_0192429929_p2571111934914">Name of the JAR file on which the Spark application depends. Use commas (,) to separate multiple names. The JAR file must be stored in the local path specified by <strong id="dli_09_0122__b842352706131735">localFilePath</strong> in the <strong id="dli_09_0122__b842352706131725">client.properties</strong> file in advance.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_09_0122__en-us_topic_0192429929_row1681064410424"><td class="cellrowborder" valign="top" width="17.59%" headers="mcps1.3.6.2.1.2.2.1.2.4.1.1 "><p id="dli_09_0122__en-us_topic_0192429929_p20810134417428">--name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="21.83%" headers="mcps1.3.6.2.1.2.2.1.2.4.1.2 "><p id="dli_09_0122__en-us_topic_0192429929_p081094419427"><NAME></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.58%" headers="mcps1.3.6.2.1.2.2.1.2.4.1.3 "><p id="dli_09_0122__en-us_topic_0192429929_p1254641904913">Name of a Spark application.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_09_0122__en-us_topic_0192429929_row77075745011"><td class="cellrowborder" valign="top" width="17.59%" headers="mcps1.3.6.2.1.2.2.1.2.4.1.1 "><p id="dli_09_0122__en-us_topic_0192429929_p1707127105020">--queue</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="21.83%" headers="mcps1.3.6.2.1.2.2.1.2.4.1.2 "><p id="dli_09_0122__en-us_topic_0192429929_p1370720717507"><QUEUE_NAME></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.58%" headers="mcps1.3.6.2.1.2.2.1.2.4.1.3 "><p id="dli_09_0122__en-us_topic_0192429929_p37074719505">Name of the Spark queue on the DLI server. Jobs are submitted to the queue for execution.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_09_0122__en-us_topic_0192429929_row739012085011"><td class="cellrowborder" valign="top" width="17.59%" headers="mcps1.3.6.2.1.2.2.1.2.4.1.1 "><p id="dli_09_0122__en-us_topic_0192429929_p93916015502">--py-files</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="21.83%" headers="mcps1.3.6.2.1.2.2.1.2.4.1.2 "><p id="dli_09_0122__en-us_topic_0192429929_p7391160165018"><PY_FILES></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.58%" headers="mcps1.3.6.2.1.2.2.1.2.4.1.3 "><p id="dli_09_0122__en-us_topic_0192429929_p439118085013">Name of the Python program file on which the Spark application depends. Use commas (,) to separate multiple file names. The Python program file must be saved in the local path specified by <strong id="dli_09_0122__b84235270613187">localFilePath</strong> in the <strong id="dli_09_0122__b842352706131812">client.properties</strong> file in advance.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_09_0122__en-us_topic_0192429929_row03121526155019"><td class="cellrowborder" valign="top" width="17.59%" headers="mcps1.3.6.2.1.2.2.1.2.4.1.1 "><p id="dli_09_0122__en-us_topic_0192429929_p2312162612504">-s,--skip-upload-resources</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="21.83%" headers="mcps1.3.6.2.1.2.2.1.2.4.1.2 "><p id="dli_09_0122__en-us_topic_0192429929_p331272625012"><all | app | deps></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.58%" headers="mcps1.3.6.2.1.2.2.1.2.4.1.3 "><p id="dli_09_0122__en-us_topic_0192429929_p123125269502">Specifies whether to skip. Upload the JAR file, Python program file, and configuration file to OBS and load them to the resource list on the DLI server. If related resource files have been loaded to the DLI resource list, skip this step.</p>
|
|
<p id="dli_09_0122__en-us_topic_0192429929_p988914164412">If this parameter is not specified, all resource files in the command are uploaded and loaded to DLI by default.</p>
|
|
<ul id="dli_09_0122__en-us_topic_0192429929_ul15146455105118"><li id="dli_09_0122__en-us_topic_0192429929_li1914635516511"><strong id="dli_09_0122__b2863194114016">all</strong>: Skips the upload and loading all resource files.</li><li id="dli_09_0122__en-us_topic_0192429929_li114635513519"><strong id="dli_09_0122__b842352706131847">app</strong>: Skips the upload and loading of Spark application files.</li><li id="dli_09_0122__en-us_topic_0192429929_li81463559516"><strong id="dli_09_0122__b84235270613190">deps</strong>: skips the upload and loading of all dependent files.</li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_09_0122__en-us_topic_0192429929_row783621125114"><td class="cellrowborder" valign="top" width="17.59%" headers="mcps1.3.6.2.1.2.2.1.2.4.1.1 "><p id="dli_09_0122__en-us_topic_0192429929_p68369145118">-h,--help</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="21.83%" headers="mcps1.3.6.2.1.2.2.1.2.4.1.2 "><p id="dli_09_0122__en-us_topic_0192429929_p88361011518">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.58%" headers="mcps1.3.6.2.1.2.2.1.2.4.1.3 "><p id="dli_09_0122__en-us_topic_0192429929_p3836141115117">Displays command help information.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
<div class="p" id="dli_09_0122__en-us_topic_0192429929_p157851724114511">Command example:<pre class="screen" id="dli_09_0122__en-us_topic_0192429929_screen11786624174517">./spark-submit --name <name> --queue <queue_name> --class org.apache.spark.examples.SparkPi spark-examples_2.11-2.1.0.luxor.jar 10
|
|
./spark-submit --name <name> --queue <queue_name> word_count.py</pre>
|
|
<div class="note" id="dli_09_0122__note146027341339"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_09_0122__p46037341313">To use the DLI queue rather than the existing Spark environment, use <strong id="dli_09_0122__b14996486572">./spark-submit</strong> instead of <strong id="dli_09_0122__b68275513576">spark-submit</strong>.</p>
|
|
</div></div>
|
|
</div>
|
|
</p></li></ol>
|
|
<p id="dli_09_0122__en-us_topic_0192429929_p19291520183816"></p>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_09_0203.html">Spark Jar Jobs</a></div>
|
|
</div>
|
|
</div>
|
|
|