Running a Kafka Job

You can submit programs developed by yourself to MRS to execute them, and obtain the results. This topic describes how to generate and consume messages in a Kafka topic.

Currently, Kafka jobs cannot be submitted on the GUI. You can submit them in the background.

Submitting a Job in the Background

Query the instance addresses of ZooKeeper and Kafka, and then run the Kafka job.

Querying the Instance Address (3.x)

  1. Log in to the MRS console.
  2. Choose Clusters > Active Clusters, select a running cluster, and click its name to switch to the cluster details page.
  3. Go to the FusionInsight Manager page. For details, see Accessing FusionInsight Manager (MRS 3.x or Later). On MRS Manager, choose Services > ZooKeeper > Instance to query the IP addresses of ZooKeeper instances. Record any IP address of a ZooKeeper instance.
  4. Choose Services > Kafka > Instance to query the IP addresses of Kafka instances. Record any IP address of a Kafka instance.

Querying the Instance Address (Versions Earlier Than 3.x)

  1. Log in to the MRS console.
  2. Choose Clusters > Active Clusters, select a running cluster, and click its name to switch to the cluster details page.
  3. On the MRS cluster details page, choose Components > ZooKeeper > Instance to query the IP addresses of ZooKeeper instances. Record any IP address of a ZooKeeper instance.

    For MRS 1.7.2 or earlier, log in to MRS Manager. For details, see Accessing MRS Manager MRS 2.1.0 or Earlier). Choose Services > ZooKeeper > Instance to query the IP addresses of ZooKeeper instances. Record any IP address of a ZooKeeper instance.

  4. Choose Components > Kafka > Instance to query the IP addresses of Kafka instances. Record any IP address of a Kafka instance.

Running a Kafka Job

In MRS 3.x and later versions, the default installation path of the client is /opt/Bigdata/client. In MRS 3.x and earlier versions, the default installation path is /opt/client. For details, see the actual situation.

  1. Log in to the Master2 node. For details, see Logging In to an ECS.
  2. Run the following command to initialize environment variables:

    source /opt/Bigdata/client/bigdata_env

  3. If the Kerberos authentication is enabled for the current cluster, run the following command to authenticate the user. If the Kerberos authentication is disabled for the current cluster, skip this step.

    kinit MRS cluster user

    Example: kinit admin

  4. Run the following command to create a Kafka topic:

    kafka-topics.sh --create --zookeeper <IP address of the ZooKeeper role instance:2181/kafka> --partitions 2 --replication-factor 2 --topic <Topic name>

  5. Produce messages in a topic test.

    Run the following command: kafka-console-producer.sh --broker-list <IP address of the Kafka role instance:9092> --topic <Topic name> --producer.config /opt/Bigdata/client/Kafka/kafka/config/producer.properties.

    Input specified information as the messages produced by the producer and then press Enter to send the messages. To end message production, press Ctrl+C to exit.

  6. Consume messages in the topic test.

    kafka-console-consumer.sh --topic <Topic name> --bootstrap-server <Kafka role instance IP:210079092> --consumer.config /opt/Bigdata/client/Kafka/kafka/config/consumer.properties

    If Kerberos authentication is enabled in the cluster, change the port number 9092 to 21007 when running the preceding two commands. For details, see List of Open Source Component Ports.

Parent topic: Job Management