forked from docs/doc-exports
chenxiaoxiong
f9e2808b7c
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: chenxiaoxiong <chenxiaoxiong@huawei.com> Co-committed-by: chenxiaoxiong <chenxiaoxiong@huawei.com>
152 lines
18 KiB
HTML
152 lines
18 KiB
HTML
<a name="dataartsstudio_01_0058"></a><a name="dataartsstudio_01_0058"></a>
|
|
|
|
<h1 class="topictitle1">From Kafka/DMS Kafka</h1>
|
|
<div id="body8662426"><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p46135468113529">If the source link of a job is a <a href="dataartsstudio_01_0033.html">Kafka link</a> or <a href="dataartsstudio_01_0038.html">DMS Kafka link</a>, configure the source job parameters based on <a href="#dataartsstudio_01_0058__en-us_topic_0108275337_table45985388113529">Table 1</a>.</p>
|
|
|
|
<div class="tablenoborder"><a name="dataartsstudio_01_0058__en-us_topic_0108275337_table45985388113529"></a><a name="en-us_topic_0108275337_table45985388113529"></a><table cellpadding="4" cellspacing="0" summary="" id="dataartsstudio_01_0058__en-us_topic_0108275337_table45985388113529" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="dataartsstudio_01_0058__en-us_topic_0108275337_row1499348113529"><th align="left" class="cellrowborder" valign="top" width="7.5200000000000005%" id="mcps1.3.2.2.5.1.1"><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p03651454146">Type</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="14.469999999999999%" id="mcps1.3.2.2.5.1.2"><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p54338376113529"><strong id="dataartsstudio_01_0058__en-us_topic_0108275337_b842352706103559">Parameter</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="60.46%" id="mcps1.3.2.2.5.1.3"><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p39332299113529"><strong id="dataartsstudio_01_0058__en-us_topic_0108275337_b842352706193020">Description</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="17.549999999999997%" id="mcps1.3.2.2.5.1.4"><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p31799688113529"><strong id="dataartsstudio_01_0058__en-us_topic_0108275337_b84235270611522">Example Value</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="dataartsstudio_01_0058__en-us_topic_0108275337_row29414735113529"><td class="cellrowborder" rowspan="10" valign="top" width="7.5200000000000005%" headers="mcps1.3.2.2.5.1.1 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p4365194513143">Basic parameters</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.469999999999999%" headers="mcps1.3.2.2.5.1.2 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p33783328113529">Topics</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.46%" headers="mcps1.3.2.2.5.1.3 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p52095013113529">One or more topics can be entered.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="17.549999999999997%" headers="mcps1.3.2.2.5.1.4 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p58946502113529">est1,est2</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0058__en-us_topic_0108275337_row1757114317498"><td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.1 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p72289311596">Data Format</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.2 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p111321138622">Format used for parsing data. The options are as follows:</p>
|
|
<ul id="dataartsstudio_01_0058__en-us_topic_0108275337_ul59573391929"><li id="dataartsstudio_01_0058__en-us_topic_0108275337_li169571839927"><strong id="dataartsstudio_01_0058__en-us_topic_0108275337_b177561537113415">Binary</strong>: Data is transferred directly. It is not converted to another format. This setting is suitable for file migration.</li><li id="dataartsstudio_01_0058__en-us_topic_0108275337_li1085014450210"><strong id="dataartsstudio_01_0058__en-us_topic_0108275337_b694354212345">CSV</strong>: Source data will be migrated after being converted in CSV format.</li><li id="dataartsstudio_01_0058__en-us_topic_0108275337_li1691575895019"><strong id="dataartsstudio_01_0058__en-us_topic_0108275337_b1282873945419">JSON</strong>: Source data will be migrated after being converted in JSON format.</li><li id="dataartsstudio_01_0058__en-us_topic_0108275337_li9788195055514"><strong id="dataartsstudio_01_0058__en-us_topic_0108275337_b8545125511191">CDC (DRS)</strong>: Source data will be migrated after being converted in DRS format.</li><li id="dataartsstudio_01_0058__en-us_topic_0108275337_li989620424512"><strong id="dataartsstudio_01_0058__en-us_topic_0108275337_b167351528112110">CDC (JSON)</strong>: Source data will be migrated after being converted in JSON format.</li><li id="dataartsstudio_01_0058__en-us_topic_0108275337_li15719144115520"><strong id="dataartsstudio_01_0058__en-us_topic_0108275337_b14326205713217">CDC (DRS_AVRO)</strong>: Source data will be migrated after being converted in DRS_AVRO format.</li><li id="dataartsstudio_01_0058__en-us_topic_0108275337_li123380675510"><strong id="dataartsstudio_01_0058__en-us_topic_0108275337_b1274091982218">CDC (DRS_JSON)</strong>: Source data will be migrated after being converted in DRS_JSON format.</li></ul>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.3 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p182286311598">Binary</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0058__en-us_topic_0108275337_row485219175014"><td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.1 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p22327637113529">Offset</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.2 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p63708157113529">Initial offset parameter</p>
|
|
<ul id="dataartsstudio_01_0058__en-us_topic_0108275337_ul36502502113529"><li id="dataartsstudio_01_0058__en-us_topic_0108275337_li60087069113529"><strong id="dataartsstudio_01_0058__en-us_topic_0108275337_b84235270616244">Latest</strong>: Maximum offset, indicating that the latest data will be extracted.</li><li id="dataartsstudio_01_0058__en-us_topic_0108275337_li35214430113529"><strong id="dataartsstudio_01_0058__en-us_topic_0108275337_b842352706162735">Earliest</strong>: Minimum offset, indicating that the earliest data will be extracted.</li><li id="dataartsstudio_01_0058__en-us_topic_0108275337_li14236153105416"><strong id="dataartsstudio_01_0058__en-us_topic_0108275337_b83802560536">Submitted</strong>: data that has been submitted</li><li id="dataartsstudio_01_0058__en-us_topic_0108275337_li10217596524"><strong id="dataartsstudio_01_0058__en-us_topic_0108275337_b258692419541">Time Range</strong>: data within a specified time range</li></ul>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.3 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p33796583113529">Latest</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0058__en-us_topic_0108275337_row86165196577"><td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.1 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p7616161975718">Data Extraction Timeout Duration</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.2 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p206165191574">Maximum duration (minutes) of data extraction. For example, a job scheduled daily needs a sufficient duration to extract the data generated by the topic every day.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.3 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p14616719145716">60</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0058__en-us_topic_0108275337_row876374510598"><td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.1 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p1076474519596">Suspension Period</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.2 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p8764144520590">If the value is set to 60 and no data is returned within 60s after the consumer requests data extraction from Kafka (generally because all the data in the topic has been read or the network or Kafka cluster is unavailable), the task will stop immediately. Otherwise, the system will retry reading data.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.3 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p3764174519595">60</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0058__en-us_topic_0108275337_row15194006113529"><td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.1 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p22755010113529">Consumer Group ID</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.2 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p1638011885013">Consumer group ID</p>
|
|
<p id="dataartsstudio_01_0058__en-us_topic_0108275337_p1114017325017">If you export data from DMS Kafka, enter any value for Kafka Platinum but a valid consumer group ID for Kafka Basic.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.3 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p45507701113529">sumer-group</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0058__en-us_topic_0108275337_row143475371275"><td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.1 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p17348537678">Start Time</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.2 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p83481137474">This parameter is required when <strong id="dataartsstudio_01_0058__en-us_topic_0108275337_b1946024182513">Offset</strong> is set to <strong id="dataartsstudio_01_0058__en-us_topic_0108275337_b666795912412">Time Range</strong>. It specifies the start time for pulling data, including the data at the specified time point.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.3 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p1434816371674">2020-12-20 12:00:00</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0058__en-us_topic_0108275337_row104250407712"><td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.1 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p184261140779">End Time</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.2 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p14262401975">This parameter is required when <strong id="dataartsstudio_01_0058__en-us_topic_0108275337_b5879738172516">Offset</strong> is set to <strong id="dataartsstudio_01_0058__en-us_topic_0108275337_b19879138112514">Time Range</strong>. It specifies the end time for pulling data, excluding the data at the specified time point.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.3 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p134264404720">2020-12-20 20:00:00</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0058__en-us_topic_0108275337_row62245198113529"><td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.1 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p8696283113529">Field Delimiter</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.2 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p9802957173352">This parameter is required when <strong id="dataartsstudio_01_0058__en-us_topic_0108275337_b7333626112618">Data Format</strong> is set to <strong id="dataartsstudio_01_0058__en-us_topic_0108275337_b14923343269">CSV</strong>. The default value is space. To set the <strong id="dataartsstudio_01_0058__en-us_topic_0108275337_b842352706113647">Tab</strong> key as the delimiter, set this parameter to <span class="parmvalue" id="dataartsstudio_01_0058__en-us_topic_0108275337_parmvalue1470440010114039"><b>\t</b></span>.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.3 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p56938959113529">,</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0058__en-us_topic_0108275337_row137516216419"><td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.1 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p2751526417">Record Delimiter</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.2 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p875115214414">This parameter is required when <strong id="dataartsstudio_01_0058__en-us_topic_0108275337_b9926134152618">Data Format</strong> is set to <strong id="dataartsstudio_01_0058__en-us_topic_0108275337_b1392712413269">CSV</strong> or <strong id="dataartsstudio_01_0058__en-us_topic_0108275337_b181771118469">JSON</strong>. The default value is space. To set the <strong id="dataartsstudio_01_0058__en-us_topic_0108275337_b9380194492612">Tab</strong> key as the delimiter, set this parameter to <span class="parmvalue" id="dataartsstudio_01_0058__en-us_topic_0108275337_parmvalue18380184417267"><b>\t</b></span>.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.3 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p1175182947">,</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0058__en-us_topic_0108275337_row2081054918286"><td class="cellrowborder" rowspan="7" valign="top" width="7.5200000000000005%" headers="mcps1.3.2.2.5.1.1 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p28105490284">Advanced parameters</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.469999999999999%" headers="mcps1.3.2.2.5.1.2 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p19811174910286">UseConfigFile</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.46%" headers="mcps1.3.2.2.5.1.3 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p14811164918281">This parameter is required when <span class="parmname" id="dataartsstudio_01_0058__en-us_topic_0108275337_parmname527815952915"><b>Data Format</b></span> is set to <span class="parmvalue" id="dataartsstudio_01_0058__en-us_topic_0108275337_parmvalue1027820913290"><b>CDC</b></span>. It is used to configure OBS files.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="17.549999999999997%" headers="mcps1.3.2.2.5.1.4 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p68113493286">No</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0058__en-us_topic_0108275337_row1094742616161"><td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.1 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p4948326121618">OBS Link</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.2 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p1994862617161">Select an OBS link.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.3 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p1194811261164">obs_link</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0058__en-us_topic_0108275337_row1487143019164"><td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.1 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p10871143014165">OBS Bucket</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.2 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p730164911714">Select an OBS bucket.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.3 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p087163016166">obs_test</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0058__en-us_topic_0108275337_row13193543165"><td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.1 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p7241954151613">Config File</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.2 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p132485417164">Select the OBS configuration file.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.3 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p11241454121611">/obs/config.csv</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0058__en-us_topic_0108275337_row48652991113529"><td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.1 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p48578216113529">Max. Poll Records</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.2 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p42521393113529">(Optional) Maximum number of records per poll</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.3 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p21680770113529">100</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0058__en-us_topic_0108275337_row6670289111413"><td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.1 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p3422508911413">Max. Poll Interval</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.2 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p2076879511413">(Optional) Maximum interval between polls (seconds)</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.3 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p455081711413">100</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0058__en-us_topic_0108275337_row03991170307"><td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.1 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p339991753010">Notice Topic</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.2 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p103991178304">Topic for sending notification data. If the data format is CDC, the notification content is the names of the generated files.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.2.2.5.1.3 "><p id="dataartsstudio_01_0058__en-us_topic_0108275337_p12399717113013">notice</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="dataartsstudio_01_0047.html">Configuring CDM Source Job Parameters</a></div>
|
|
</div>
|
|
</div>
|
|
|