:original_name: mrs_01_1066.html

.. _mrs_01_1066:

Typical Scenario: Collecting Logs from Kafka and Uploading Them to HDFS Through the Flume Client
================================================================================================

Scenario
--------

This section describes how to use the Flume client to collect logs from the topic list (test1) of the Kafka client and save them to the **/flume/test** directory on HDFS.

This section applies to MRS 3.\ *x* or later clusters.

.. note::

   By default, the cluster network environment is secure and the SSL authentication is not enabled during the data transmission process. For details about how to use the encryption mode, see :ref:`Configuring the Encrypted Transmission <mrs_01_1069>`.

Prerequisites
-------------

-  The cluster has been installed, including the HDFS, Kafka, and Flume services.
-  The Flume client has been installed. For details, see `Installing the Flume Client <https://docs.otc.t-systems.com/cmpntguide/mrs/mrs_01_0392.html>`__.
-  You have created user **flume_hdfs** and authorized the HDFS directory and data to be operated during log verification.
-  The network environment of the cluster is secure.

Procedure
---------

#. On FusionInsight Manager, choose **System > User** and choose **More > Download Authentication Credential** to download the Kerberos certificate file of user **flume_hdfs** and save it to the local host.
#. Configure the client parameters of the Flume role.

   a. Use the Flume configuration tool on FusionInsight Manager to configure the Flume role client parameters and generate a configuration file.

      #. Log in to FusionInsight Manager and choose **Cluster** > **Services**. On the page that is displayed, choose **Flume**. On the displayed page, click the **Configuration Tool** tab.

      #. Set **Agent Name** to **client**. Select the source, channel, and sink to be used, drag them to the GUI on the right, and connect them.

         For example, use Kafka Source, File Channel, and HDFS Sink.

      #. Double-click the source, channel, and sink. Set corresponding configuration parameters by referring to :ref:`Table 1 <mrs_01_1066__table15799134418507>` based on the actual environment.

         .. note::

            -  If you want to continue using the **properties.propretites** file by modifying it, log in to FusionInsight Manager, choose **Cluster** > *Name of the desired cluster* > **Services**. On the page that is displayed, choose **Flume**. On the displayed page, click the **Configuration Tool** tab, click **Import**, import the file, and modify the configuration items related to non-encrypted transmission.
            -  It is recommended that the numbers of Sources, Channels, and Sinks do not exceed 40 during configuration file import. Otherwise, the response time may be very long.

         .. _mrs_01_1066__table15799134418507:

         .. table:: **Table 1** Parameters to be modified for the Flume role client

             
            | Parameter               | Description                                                                                                                                                                                                                                                                                                                                                                                                                                             | Example Value                                                                                                                                                                                                                              |
             
            | Name                    | The value must be unique and cannot be left blank.                                                                                                                                                                                                                                                                                                                                                                                                      | test                                                                                                                                                                                                                                       |
             
            | kafka.topics            | Specifies the subscribed Kafka topic list, in which topics are separated by commas (,). This parameter cannot be left blank.                                                                                                                                                                                                                                                                                                                            | test1                                                                                                                                                                                                                                      |
             
            | kafka.consumer.group.id | Specifies the data group ID obtained from Kafka. This parameter cannot be left blank.                                                                                                                                                                                                                                                                                                                                                                   | flume                                                                                                                                                                                                                                      |
             
            | kafka.bootstrap.servers | Specifies the bootstrap IP address and port list of Kafka. The default value is all Kafka lists in a Kafka cluster. If Kafka has been installed in the cluster and its configurations have been synchronized, this parameter can be left blank.                                                                                                                                                                                                         | 192.168.101.10:21007                                                                                                                                                                                                                       |
             
            | batchSize               | Specifies the number of events that Flume sends in a batch (number of data pieces).                                                                                                                                                                                                                                                                                                                                                                     | 61200                                                                                                                                                                                                                                      |
             
            | dataDirs                | Specifies the directory for storing buffer data. The run directory is used by default. Configuring multiple directories on disks can improve transmission efficiency. Use commas (,) to separate multiple directories. If the directory is inside the cluster, the **/srv/BigData/hadoop/dataX/flume/data** directory can be used. **dataX** ranges from data1 to dataN. If the directory is outside the cluster, it needs to be independently planned. | /srv/BigData/hadoop/data1/flume/data                                                                                                                                                                                                       |
             
            | checkpointDir           | Specifies the directory for storing the checkpoint information, which is under the run directory by default. If the directory is inside the cluster, the **/srv/BigData/hadoop/dataX/flume/checkpoint** directory can be used. **dataX** ranges from data1 to dataN. If the directory is outside the cluster, it needs to be independently planned.                                                                                                     | /srv/BigData/hadoop/data1/flume/checkpoint                                                                                                                                                                                                 |
             
            | transactionCapacity     | Specifies the transaction size, that is, the number of events in a transaction that can be processed by the current Channel. The size cannot be smaller than the batchSize of Source. Setting the same size as batchSize is recommended.                                                                                                                                                                                                                | 61200                                                                                                                                                                                                                                      |
             
            | hdfs.path               | Specifies the HDFS data write directory. This parameter cannot be left blank.                                                                                                                                                                                                                                                                                                                                                                           | hdfs://hacluster/flume/test                                                                                                                                                                                                                |
             
            | hdfs.filePrefix         | Specifies the file name prefix after data is written to HDFS.                                                                                                                                                                                                                                                                                                                                                                                           | TMP\_                                                                                                                                                                                                                                      |
             
            | hdfs.batchSize          | Specifies the maximum number of events that can be written to HDFS once.                                                                                                                                                                                                                                                                                                                                                                                | 61200                                                                                                                                                                                                                                      |
             
            | hdfs.kerberosPrincipal  | Specifies the Kerberos authentication user, which is mandatory in security versions. This configuration is required only in security clusters.                                                                                                                                                                                                                                                                                                          | flume_hdfs                                                                                                                                                                                                                                 |
             
            | hdfs.kerberosKeytab     | Specifies the keytab file path for Kerberos authentication, which is mandatory in security versions. This configuration is required only in security clusters.                                                                                                                                                                                                                                                                                          | /opt/test/conf/user.keytab                                                                                                                                                                                                                 |
            |                         |                                                                                                                                                                                                                                                                                                                                                                                                                                                         |                                                                                                                                                                                                                                            |
            |                         |                                                                                                                                                                                                                                                                                                                                                                                                                                                         | .. note::                                                                                                                                                                                                                                  |
            |                         |                                                                                                                                                                                                                                                                                                                                                                                                                                                         |                                                                                                                                                                                                                                            |
            |                         |                                                                                                                                                                                                                                                                                                                                                                                                                                                         |    Obtain the **user.keytab** file from the Kerberos certificate file of the user **flume_hdfs**. In addition, ensure that the user who installs and runs the Flume client has the read and write permissions on the **user.keytab** file. |
             
            | hdfs.useLocalTimeStamp  | Specifies whether to use the local time. Possible values are **true** and **false**.                                                                                                                                                                                                                                                                                                                                                                    | true                                                                                                                                                                                                                                       |
             

      #. Click **Export** to save the **properties.properties** configuration file to the local.

   b. Upload the **properties.properties** file to **flume/conf/** under the installation directory of the Flume client.

   c. To connect the Flume client to the HDFS, you need to add the following configuration:

      #. Download the Kerberos certificate of account **flume_hdfs** and obtain the **krb5.conf** configuration file. Upload the configuration file to the **fusioninsight-flume-1.9.0/conf/** directory on the node where the client is installed.

      #. In **fusioninsight-flume-1.9.0/conf/**, create the **jaas.conf** configuration file.

         **vi jaas.conf**

         .. code-block::

            KafkaClient {
            com.sun.security.auth.module.Krb5LoginModule required
            useKeyTab=true
            keyTab="/opt/test/conf/user.keytab"
            principal="flume_hdfs@<System domain name>"
            useTicketCache=false
            storeKey=true
            debug=true;
            };

         Values of **keyTab** and **principal** vary depending on the actual situation.

      #. Obtain configuration files **core-site.xml** and **hdfs-site.xml** from **/opt/FusionInsight_Cluster\_\ <Cluster ID>\ \_Flume_ClientConfig/Flume/config** and upload them to **fusioninsight-flume-1.9.0/conf/**.

   d. Run the following command to restart the Flume process:

      **flume-manager.sh restart**

3. Verify log transmission.

   a. Log in to FusionInsight Manager as a user who has the management permission on HDFS. For details, see :ref:`Accessing FusionInsight Manager (MRS 3.x or Later) <mrs_01_2124>`. Choose **Cluster** > **Services** > **HDFS**. On the page that is displayed, click the **NameNode(**\ *Node name*\ **,Active)** link next to **NameNode WebUI** to go to the HDFS web UI. On the displayed page, choose **Utilities** > **Browse the file system**.
   b. Check whether the data is generated in the **/flume/test** directory on the HDFS.
