Adding and Executing a Job

Function

This API is used to add and submit a job in an MRS cluster.

• If Kerberos authentication is enabled for a cluster, on the Dashboard tab page of the cluster details page, click Click to synchronize on the right side of IAM User Sync to synchronize IAM users, and then submit a job through this API.

URI

• Format

  POST /v2/{project_id}/clusters/{cluster_id}/job-executions

• Parameter description

  Table 1 URI parameter description

  Parameter	Mandatory	Description
  project_id	Yes	Project ID. For details on how to obtain the project ID, see Obtaining a Project ID.
  cluster_id	Yes	Cluster ID. For details on how to obtain the cluster ID, see Obtaining a Cluster ID.

Request

Response

Example

• Example request

  The following is an example of an MapReduce job request:

  {
      "job_name":"MapReduceTest",
      "job_type":"MapReduce",
      "arguments":[
          "obs://obs-test/program/hadoop-mapreduce-examples-x.x.x.jar",
          "wordcount",
          "obs://obs-test/input/",
          "obs://obs-test/job/mapreduce/output"
      ],
      "properties":{
          "fs.obs.endpoint":"obs endpoint",
          "fs.obs.access.key":"xxx",
          "fs.obs.secret.key":"yyy"
      }
  }

  The following is an example of a SparkSubmit job request:

  {
      "job_name":"SparkSubmitTest",
      "job_type":"SparkSubmit",
      "arguments":[
          "--master",
          "yarn",
          "--deploy-mode",
          "cluster",
          "--py-files",
          "obs://obs-test/a.py",
          "--conf",
          "spark.yarn.appMasterEnv.PYTHONPATH=/tmp:$PYTHONPATH",
          "--conf",
          "spark.yarn.appMasterEnv.aaa=aaaa",
          "--conf",
          "spark.executorEnv.aaa=executoraaa",
          "--properties-file",
          "obs://obs-test/test-spark.conf",
          "obs://obs-test/pi.py",
          "100000"
      ],
      "properties":{
          "fs.obs.access.key":"xxx",
          "fs.obs.secret.key":"yyy"
      }
  }

  The following is an example of a HiveScript job request:

  {
      "job_name":"HiveScriptTest",
      "job_type":"HiveScript",
      "arguments":[
          "obs://obs-test/sql/test_script.sql"
      ],
      "properties":{
          "fs.obs.endpoint":"obs endpoint",
          "fs.obs.access.key":"xxx",
          "fs.obs.secret.key":"yyy"
      }
  }

  The following is an example of a HiveSQL job request:

  {
      "job_name":"HiveSqlTest",
      "job_type":"HiveSql",
      "arguments":[
          "DROP TABLE IF EXISTS src_wordcount;create external table src_wordcount(line string);insert into src_wordcount values('v1')"
      ],
      "properties":{
          "fs.obs.endpoint":"obs endpoint",
          "fs.obs.access.key":"xxx",
          "fs.obs.secret.key":"yyy"
      }
  }

  The following is an example of a DistCp job request:

  {
      "job_name":"DistCpTest",
      "job_type":"DistCp",
      "arguments":[
          "obs://obs-test/DistcpJob/",
          "/user/test/sparksql/"
      ],
      "properties":{
          "fs.obs.endpoint":"obs endpoint",
          "fs.obs.access.key":"xxx",
          "fs.obs.secret.key":"yyy"
      }
  }

  The following is an example of a SparkScript job request:

  {
      "job_name":"SparkScriptTest",
      "job_type":"SparkScript",
      "arguments":[
          "op-key1",
          "op-value1",
          "op-key2",
          "op-value2",
          "obs://obs-test/sql/test_script.sql"
      ],
      "properties":{
          "fs.obs.access.key":"xxx",
          "fs.obs.secret.key":"yyy"
      }
  } 

  The following is an example of a SparkSQL job request:

  {
      "job_name":"SparkSqlTest",
      "job_type":"SparkSql",
      "arguments":[
          "op-key1",
          "op-value1",
          "op-key2",
          "op-value2",
          "create table student_info3 (id string,name string,gender string,age int,addr string);"
      ],
      "properties":{
          "fs.obs.access.key":"xxx",
          "fs.obs.secret.key":"yyy"
      }
  } 

  The following is an example of a Flink job request:

  {
      "job_name":"FlinkTest",
      "job_type":"Flink",
      "arguments":[
          "run",
          "-d",
          "-ynm",
          "testExcutorejobhdfsbatch",
          "-m",
          "yarn-cluster",
          "hdfs://test/examples/batch/WordCount.jar"
      ],
      "properties":{
          "fs.obs.endpoint":"obs endpoint",
          "fs.obs.access.key":"xxx",
          "fs.obs.secret.key":"yyy"
      }
  }

• Example response
  • Example of a successful response

    {
      "job_submit_result":{
          "job_id":"44b37a20-ffe8-42b1-b42b-78a5978d7e40",
          "state":"COMPLETE"
      }
    }

  • Example of a failed response

    {
    "error_msg": Hive jobs cannot be submitted.
    "error_code":"0168"
    }

Status Code

For details about status codes, see Status Codes.