forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
35 lines
9.4 KiB
HTML
35 lines
9.4 KiB
HTML
<a name="mrs_01_0379"></a><a name="mrs_01_0379"></a>
|
|
|
|
<h1 class="topictitle1">Managing Messages in Kafka Topics</h1>
|
|
<div id="body1589421630459"><div class="section" id="mrs_01_0379__s97aa33d045f046bb9c1d9907172b3db2"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_0379__a5d6cb2522caf46b6ae5e00750c6f3d83">You can produce or consume messages in Kafka topics using the MRS cluster client. For clusters with Kerberos authentication enabled, you must have the permission to perform these operations.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_0379__sa5c652338d284a47ad73fc0e957155c5"><h4 class="sectiontitle">Prerequisites</h4><p id="mrs_01_0379__a1acbd534cd91469080e7578d5bafdc50">You have installed the Kafka client.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_0379__s87eef4ea94af42b7944cf554191823a4"><h4 class="sectiontitle">Procedure</h4><ol id="mrs_01_0379__o734dac617c7c4e8dadaf85650ff773b7"><li id="mrs_01_0379__li13806122116487"><span>Go to the Kafka service page.</span><p><ul id="mrs_01_0379__en-us_topic_0264265925_ul1692910323486"><li id="mrs_01_0379__en-us_topic_0264265925_li792916322486">For versions earlier than MRS 1.9.2, log in to MRS Manager and choose <strong id="mrs_01_0379__en-us_topic_0264265925_b699115497148">Services</strong> > <strong id="mrs_01_0379__en-us_topic_0264265925_b181218241519">Kafka</strong>.</li><li id="mrs_01_0379__en-us_topic_0264265925_li979413816488">For MRS 1.9.2 or later, click the cluster name on the MRS console and choose <strong id="mrs_01_0379__en-us_topic_0264265925_b97122019159">Components</strong> > <strong id="mrs_01_0379__en-us_topic_0264265925_b113132021514">Kafka</strong>.<div class="note" id="mrs_01_0379__en-us_topic_0264265925_note1103203791311"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_0379__en-us_topic_0264265925_p15841154717426">If the <strong id="mrs_01_0379__en-us_topic_0264265925_b430410223013">Components</strong> tab is unavailable, complete IAM user synchronization first. (On the <strong id="mrs_01_0379__en-us_topic_0264265925_b1830414223017">Dashboard</strong> page, click <strong id="mrs_01_0379__en-us_topic_0264265925_b0304142215011">Synchronize</strong> on the right side of <span class="parmname" id="mrs_01_0379__en-us_topic_0264265925_parmname73054228016"><b>IAM User Sync</b></span> to synchronize IAM users.)</p>
|
|
</div></div>
|
|
</li><li id="mrs_01_0379__en-us_topic_0264265925_li13471132141315">For MRS 3.<em id="mrs_01_0379__en-us_topic_0264265925_i2094017447543">x</em> or later, log in to FusionInsight Manager. For details, see <a href="mrs_01_2124.html">Accessing FusionInsight Manager (MRS 3.x or Later)</a>. Choose <strong id="mrs_01_0379__en-us_topic_0264265925_b77193404157">Cluster</strong> > <em id="mrs_01_0379__en-us_topic_0264265925_i197251400151">Name of the desired cluster</em> > <strong id="mrs_01_0379__en-us_topic_0264265925_b13726154031515">Services</strong> > <strong id="mrs_01_0379__en-us_topic_0264265925_b17726114041512">Kafka</strong>.</li></ul>
|
|
</p></li><li id="mrs_01_0379__l17ba03ec896f47d8a5c53b1e8f726441"><span>Click <span class="uicontrol" id="mrs_01_0379__uicontrol1334161117819"><b>instance</b></span>. Query the IP addresses of the Kafka instances.</span><p><p id="mrs_01_0379__ac99617dfc17b4cd89397cfb63d2de888">Record the IP address of any Kafka instance.</p>
|
|
</p></li><li id="mrs_01_0379__lc99c3810794b4305a8805127c0448528"><span>Prepare the client based on service requirements. Log in to the node where the client is installed.</span></li><li id="mrs_01_0379__l43ebcdf0488f43aa8a6dc5925fa6d624"><span>Run the following command to switch to the client directory, for example, /<strong id="mrs_01_0379__b177601388257">opt</strong><strong id="mrs_01_0379__b9760193814255"></strong><strong id="mrs_01_0379__b7760113862512">/client/Kafka/kafka/bin</strong>.</span><p><p id="mrs_01_0379__abf921a6902ab4d60bb2418a62dd04f71"><strong id="mrs_01_0379__aa2ffb20c51ca4d0bbf4035c8445d8c74">cd /opt/client/Kafka/kafka/bin</strong></p>
|
|
</p></li><li id="mrs_01_0379__l6c733670bf264983a44af845bf95d835"><span>Run the following command to configure environment variables:</span><p><p id="mrs_01_0379__afb1d4384127a43e1b162f119632db3bd"><strong id="mrs_01_0379__ab5d39d6fe1ff4198a45fa9cb49fd05c2">source /opt/client/bigdata_env</strong></p>
|
|
</p></li><li id="mrs_01_0379__l62c83e0a6fe7451db024ee5be8542c12"><span>For clusters with Kerberos authentication enabled, run the following command to authenticate the user. For clusters with Kerberos authentication disabled, skip this step.</span><p><p id="mrs_01_0379__a12b63597f2374ecfac281f9803f541e2"><strong id="mrs_01_0379__b12323201355115">kinit</strong> <em id="mrs_01_0379__i15328613205116">Kafka user</em></p>
|
|
<p id="mrs_01_0379__p1517912384815">Example:</p>
|
|
<p id="mrs_01_0379__ab973d2e6dc584ce996bd87ba46250f10"><strong id="mrs_01_0379__a310ad170e91146c897a1ba5f4e634d43">kinit admin</strong></p>
|
|
</p></li><li id="mrs_01_0379__l4998af2788284d209339aff58f21ad42"><span>Manage messages in Kafka topics using the following commands:</span><p><ul id="mrs_01_0379__u6117f8baf6e94bf4bab52f9a13b8867a"><li id="mrs_01_0379__lb4468efc44c4410db1fdd9bed4ed5742">Producing messages<p id="mrs_01_0379__a7b38507bd22d444abfaf96672ddff8b1"><a name="mrs_01_0379__lb4468efc44c4410db1fdd9bed4ed5742"></a><a name="lb4468efc44c4410db1fdd9bed4ed5742"></a><strong id="mrs_01_0379__b136457202402">sh kafka-console-producer.sh --broker-list</strong> <em id="mrs_01_0379__ad4cb407ba2c54021864adb859619ccd3">IP address of the node where the Kafka instance resides</em><strong id="mrs_01_0379__b9444153024010">:9092 --topic</strong> <em id="mrs_01_0379__a41e4d250d37c4f70becacf0e227b9c5b">Topic name</em> <strong id="mrs_01_0379__b18571133914015">--producer.config /opt/client/Kafka/kafka/config/producer.properties</strong></p>
|
|
<p id="mrs_01_0379__adbfec31492e84dec85245866152e21be">You can input specified information as the messages produced by the producer and then press <strong id="mrs_01_0379__en-us_topic_0054328610_b842352706145827">Enter</strong> to send the messages. To end message producing, press <span class="parmname" id="mrs_01_0379__en-us_topic_0054328610_parmname53594428115310"><b>Ctrl + C</b></span> to exit.</p>
|
|
</li><li id="mrs_01_0379__lf644862db08e49a3a0220751b224e3f5">Consuming messages<p id="mrs_01_0379__a70e2de5e4e8f4fb1ab3d7729f99ae5b2"><a name="mrs_01_0379__lf644862db08e49a3a0220751b224e3f5"></a><a name="lf644862db08e49a3a0220751b224e3f5"></a><strong id="mrs_01_0379__b762345014018">sh kafka-console-consumer.sh --topic</strong> <em id="mrs_01_0379__a35f63752e3e94aa7abc4c0d5b680607b">Topic name</em> <strong id="mrs_01_0379__b44661734419">--bootstrap-server</strong> <em id="mrs_01_0379__a91c908b9ff4e4c9487134f1292b65532">IP address of the node where the Kafka instance resides</em><strong id="mrs_01_0379__add1bb9f278eb479288c746d3fd6ac33f">:9092 --consumer.config /opt/client/Kafka/kafka/config/consumer.properties</strong></p>
|
|
<p id="mrs_01_0379__a1ed7692cb70141c3be5396648a495f01">In the configuration file, <span class="parmname" id="mrs_01_0379__en-us_topic_0054328610_parmname131291742291820"><b>group.id</b></span> (indicating the consumer group) is set to <span class="parmvalue" id="mrs_01_0379__en-us_topic_0054328610_parmvalue3068481691826"><b>example-group1</b></span> by default. Users can change the value as required. The value takes effect each time consumption occurs.</p>
|
|
<p id="mrs_01_0379__aa70ea7c8ccec40f7862182db36e3d103">By default, the system reads unprocessed messages in the current consumer group when the command is executed. If a new consumer group is specified in the configuration file and the <span class="parmname" id="mrs_01_0379__en-us_topic_0054328610_parmname4937532169517"><b>--from-beginning</b></span> parameter is added to the command, the system reads all messages that have not been automatically deleted in Kafka.</p>
|
|
</li></ul>
|
|
<div class="note" id="mrs_01_0379__n423e2322dc654e578f61b5c2eb04c0ff"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_0379__u504276addcca4640900fe37d9f5a21ce"><li id="mrs_01_0379__laae5cccb4fce48399c17d97f9c67c6ac">For the IP address of the node where the Kafka instance locates, use the IP address of any broker instance.</li><li id="mrs_01_0379__l0dbcf512675a44c8abfce63d72b0c036">If Kerberos authentication is enabled, change the port to <span class="parmvalue" id="mrs_01_0379__en-us_topic_0054328610_parmvalue37116123175139"><b>21007</b></span>.</li><li id="mrs_01_0379__li3741142312283">By default, the ZooKeeper's <span class="parmname" id="mrs_01_0379__en-us_topic_0244909890_pd3e452bfa0d24adda216f431d1bcebc0"><b>clientPort</b></span> value is <span class="parmvalue" id="mrs_01_0379__en-us_topic_0244909890_p67aea0e3f02b49829ae8bf8dea1fc2d7"><b>2<span id="mrs_01_0379__en-us_topic_0244909890_text59469239252">181</span></b></span>.<p id="mrs_01_0379__en-us_topic_0244909890_p1052717376567">For MRS 1.6.2 or earlier, the value of ZooKeeper's <strong id="mrs_01_0379__b175294223752849">clientPort</strong> defaults to <strong id="mrs_01_0379__b55170413352849">24002</strong>. For details, see <a href="https://docs.otc.t-systems.com/usermanual/mrs/mrs_01_0504.html" target="_blank" rel="noopener noreferrer">List of Open Source Component Ports</a>.</p>
|
|
</li></ul>
|
|
</div></div>
|
|
</p></li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0375.html">Using Kafka</a></div>
|
|
</div>
|
|
</div>
|
|
|