vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/dli/sqlreference/dli_08_0406.html
Su, Xiaomeng 76a5b1ee83 dli_sqlreference_20240227 
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
2024-03-27 22:02:33 +00:00

383 lines
41 KiB
HTML
Raw Blame History

<a name="dli_08_0406"></a><a name="dli_08_0406"></a>
<h1 class="topictitle1">Redis Dimension Table</h1>
<div id="body8662426"><div class="section" id="dli_08_0406__en-us_topic_0000001262655738_section656951413435"><h4 class="sectiontitle">Function</h4><p id="dli_08_0406__en-us_topic_0000001262655738_p07182170435">Create a Redis table to connect to source streams for wide table generation.</p>
</div>
<div class="section" id="dli_08_0406__en-us_topic_0000001262655738_section39013440442"><h4 class="sectiontitle">Prerequisites</h4><ul id="dli_08_0406__en-us_topic_0000001262655738_ul2994557141118"><li id="dli_08_0406__en-us_topic_0000001262655738_li4994205712112">An enhanced datasource connection with Redis has been established, so that you can configure security group rules as required.
</li></ul>
</div>
<div class="section" id="dli_08_0406__en-us_topic_0000001262655738_section17819158194516"><h4 class="sectiontitle">Precautions</h4><ul id="dli_08_0406__en-us_topic_0000001262655738_ul163752885110"><li id="dli_08_0406__en-us_topic_0000001262655738_li38267577259">When you create a Flink OpenSource SQL job, set <strong id="dli_08_0406__en-us_topic_0000001262655738_b155715225124">Flink Version</strong> to <strong id="dli_08_0406__en-us_topic_0000001262655738_b1057152271216">1.12</strong> in the <strong id="dli_08_0406__en-us_topic_0000001262655738_b135725223126">Running Parameters</strong> tab. Select <strong id="dli_08_0406__en-us_topic_0000001262655738_b7572162214123">Save Job Log</strong>, and specify the OBS bucket for saving job logs.</li><li id="dli_08_0406__en-us_topic_0000001262655738_li63742818515">To obtain the key values, you can set the primary key in Flink. The primary key maps to the Redis key.</li><li id="dli_08_0406__en-us_topic_0000001262655738_li20233539172616">If the primary key cannot be a composite primary key, and only can be one field.</li><li id="dli_08_0406__en-us_topic_0000001262655738_li1877444315214"><a name="dli_08_0406__en-us_topic_0000001262655738_li1877444315214"></a><a name="en-us_topic_0000001262655738_li1877444315214"></a>Constraints on <strong id="dli_08_0406__en-us_topic_0000001262655738_b19622533171411">schema-syntax</strong>:<ul id="dli_08_0406__en-us_topic_0000001262655738_ul950931104619"><li id="dli_08_0406__en-us_topic_0000001262655738_li136351521994">If <strong id="dli_08_0406__en-us_topic_0000001262655738_b06821049191611">schema-syntax</strong> is <strong id="dli_08_0406__en-us_topic_0000001262655738_b1626695620162">map</strong> or <strong id="dli_08_0406__en-us_topic_0000001262655738_b1415915819160">array</strong>, there can be only one non-primary key and it must be of the same <strong id="dli_08_0406__en-us_topic_0000001262655738_b144446569170">map</strong> or <strong id="dli_08_0406__en-us_topic_0000001262655738_b12928195811175">array</strong> type.</li><li id="dli_08_0406__en-us_topic_0000001262655738_li273823533910">If <strong id="dli_08_0406__en-us_topic_0000001262655738_b19251596182">schema-syntax</strong> is <strong id="dli_08_0406__en-us_topic_0000001262655738_b14578141181810">fields-scores</strong>, the number of non-primary keys must be an even number, and the second key of every two keys except the primary key must be of the <strong id="dli_08_0406__en-us_topic_0000001262655738_b1857659131912">double</strong> type. The <strong id="dli_08_0406__en-us_topic_0000001262655738_b1942471052019">double</strong> value is the score of the previous key. The following is an example:<pre class="screen" id="dli_08_0406__en-us_topic_0000001262655738_screen31671225143412">CREATE TABLE redisSource (
redisKey string,
order_id string,
score1 double,
order_channel string,
score2 double,
order_time string,
score3 double,
pay_amount double,
score4 double,
real_pay double,
score5 double,
pay_time string,
score6 double,
user_id string,
score7 double,
user_name string,
score8 double,
area_id string,
score9 double,
primary key (redisKey) not enforced
) WITH (
'connector' = 'redis',
'host' = '<em id="dli_08_0406__en-us_topic_0000001262655738_i19786121520470">RedisIP</em>',
'password' = '<em id="dli_08_0406__en-us_topic_0000001262655738_i11381115884314">Redis</em><em id="dli_08_0406__en-us_topic_0000001262655738_i9881358447">Password</em>',
'data-type' = 'sorted-set',
'deploy-mode' = 'master-replica',
'schema-syntax' = 'fields-scores'
);</pre>
</li></ul>
</li><li id="dli_08_0406__en-us_topic_0000001262655738_li817313914378"><a name="dli_08_0406__en-us_topic_0000001262655738_li817313914378"></a><a name="en-us_topic_0000001262655738_li817313914378"></a>Restrictions on <strong id="dli_08_0406__en-us_topic_0000001262655738_b858613113017">data-type</strong>:<ul id="dli_08_0406__en-us_topic_0000001262655738_ul298216248468"><li id="dli_08_0406__en-us_topic_0000001262655738_li1117215413230">When <strong id="dli_08_0406__en-us_topic_0000001262655738_b14223103723012">data-type</strong> is <strong id="dli_08_0406__en-us_topic_0000001262655738_b68791140203019">set</strong>, the types of non-primary keys defined in Flink must be the same.</li><li id="dli_08_0406__en-us_topic_0000001262655738_li10541130133010">If <strong id="dli_08_0406__en-us_topic_0000001262655738_b3786201116499">data-type</strong> is <strong id="dli_08_0406__en-us_topic_0000001262655738_b1280032234913">sorted-set</strong> and <strong id="dli_08_0406__en-us_topic_0000001262655738_b1854082614916">schema-syntax</strong> is <strong id="dli_08_0406__en-us_topic_0000001262655738_b5186153017491">fields</strong> or <strong id="dli_08_0406__en-us_topic_0000001262655738_b497463120495">array</strong>, only <strong id="dli_08_0406__en-us_topic_0000001262655738_b199621750204912">sorted set</strong> values can be read from Redis, and the <strong id="dli_08_0406__en-us_topic_0000001262655738_b74996239504">score</strong> value cannot be read.</li><li id="dli_08_0406__en-us_topic_0000001262655738_li1690199111318">If <strong id="dli_08_0406__en-us_topic_0000001262655738_b41131548105214">data-type</strong> is <strong id="dli_08_0406__en-us_topic_0000001262655738_b12548185120521">string</strong>, only one non-primary key field is allowed.</li><li id="dli_08_0406__en-us_topic_0000001262655738_li20784191104919">If <strong id="dli_08_0406__en-us_topic_0000001262655738_b13316203014531">data-type</strong> is <strong id="dli_08_0406__en-us_topic_0000001262655738_b1489383775312">sorted-set</strong> and <strong id="dli_08_0406__en-us_topic_0000001262655738_b56712429538">schema-syntax</strong> is <strong id="dli_08_0406__en-us_topic_0000001262655738_b1599224445316">map</strong>, there can be only one non-primary key in addition to the primary key and the non-primary key must be of the <strong id="dli_08_0406__en-us_topic_0000001262655738_b14639192665411">map</strong> type. The <strong id="dli_08_0406__en-us_topic_0000001262655738_b1347675719545">map</strong> values of the non-primary key must be of the <strong id="dli_08_0406__en-us_topic_0000001262655738_b9276423105515">double</strong> type, indicating the score. The keys in the map are the values in the Redis set.</li><li id="dli_08_0406__en-us_topic_0000001262655738_li183386254811">If <strong id="dli_08_0406__en-us_topic_0000001262655738_b649216111571">data-type</strong> is <strong id="dli_08_0406__en-us_topic_0000001262655738_b1643241620575">sorted-set</strong> and <strong id="dli_08_0406__en-us_topic_0000001262655738_b154327192574">schema-syntax</strong> is <strong id="dli_08_0406__en-us_topic_0000001262655738_b19414243574">array-scores</strong>, only two non-primary keys are allowed and must be of the <strong id="dli_08_0406__en-us_topic_0000001262655738_b31753220580">array</strong> type.<div class="p" id="dli_08_0406__en-us_topic_0000001262655738_p32723374810">The first key indicates values in the Redis set. The second key is of the <strong id="dli_08_0406__en-us_topic_0000001262655738_b2538061901">array&lt;double&gt;</strong> type, indicating index scores. The following is an example:<pre class="screen" id="dli_08_0406__en-us_topic_0000001262655738_screen529105693713">CREATE TABLE redisSink (
order_id string,
arrayField Array&lt;String&gt;,
arrayScore array&lt;double&gt;,
primary key (order_id) not enforced
) WITH (
'connector' = 'redis',
'host' = '<em id="dli_08_0406__en-us_topic_0000001262655738_i95731223184819">RedisIP</em>',
'password' = '<em id="dli_08_0406__en-us_topic_0000001262655738_i422813064810">Redis</em><em id="dli_08_0406__en-us_topic_0000001262655738_i18228130174815">Password</em>',
'data-type' = 'sorted-set',
"default-score" = '3',
'deploy-mode' = 'master-replica',
'schema-syntax' = 'array-scores'
);</pre>
</div>
</li></ul>
</li></ul>
</div>
<div class="section" id="dli_08_0406__en-us_topic_0000001262655738_section04688345452"><h4 class="sectiontitle">Syntax</h4><pre class="screen" id="dli_08_0406__en-us_topic_0000001262655738_screen680774084519">create table dwsSource (
attr_name attr_type
(',' attr_name attr_type)*
(',' watermark for rowtime_column_name as watermark-strategy_expression)
,PRIMARY KEY (attr_name, ...) NOT ENFORCED
)
with (
'connector' = 'redis',
'host' = ''
);</pre>
</div>
<div class="section" id="dli_08_0406__en-us_topic_0000001262655738_section125539417466"><h4 class="sectiontitle">Parameters</h4>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0406__en-us_topic_0000001262655738_dli_08_0238_en-us_topic_0113887276_en-us_topic_0111499973_table15955155035214" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="dli_08_0406__en-us_topic_0000001262655738_dli_08_0238_en-us_topic_0113887276_en-us_topic_0111499973_row29690509522"><th align="left" class="cellrowborder" valign="top" width="20.23%" id="mcps1.3.5.2.2.6.1.1"><p id="dli_08_0406__en-us_topic_0000001262655738_dli_08_0238_en-us_topic_0113887276_en-us_topic_0111499973_p697520507529">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="8.469999999999999%" id="mcps1.3.5.2.2.6.1.2"><p id="dli_08_0406__en-us_topic_0000001262655738_dli_08_0238_en-us_topic_0113887276_en-us_topic_0111499973_p997925010528">Mandatory</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="11.86%" id="mcps1.3.5.2.2.6.1.3"><p id="dli_08_0406__en-us_topic_0000001262655738_p16760165920121">Default Value</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="16.91%" id="mcps1.3.5.2.2.6.1.4"><p id="dli_08_0406__en-us_topic_0000001262655738_p115642046191210">Data Types</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="42.53%" id="mcps1.3.5.2.2.6.1.5"><p id="dli_08_0406__en-us_topic_0000001262655738_dli_08_0238_en-us_topic_0113887276_en-us_topic_0111499973_p9983150185215">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="dli_08_0406__en-us_topic_0000001262655738_dli_08_0238_en-us_topic_0113887276_en-us_topic_0111499973_row398835017524"><td class="cellrowborder" valign="top" width="20.23%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0406__en-us_topic_0000001262655738_en-us_topic_0000001080679435_p11571163194715">connector</p>
</td>
<td class="cellrowborder" valign="top" width="8.469999999999999%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0406__en-us_topic_0000001262655738_en-us_topic_0000001080679435_p457213319472">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="11.86%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0406__en-us_topic_0000001262655738_p10803204281212">None</p>
</td>
<td class="cellrowborder" valign="top" width="16.91%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0406__en-us_topic_0000001262655738_p056434631215">String</p>
</td>
<td class="cellrowborder" valign="top" width="42.53%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0406__en-us_topic_0000001262655738_en-us_topic_0000001080679435_p15721631124720">Connector type. Set this parameter to <strong id="dli_08_0406__en-us_topic_0000001262655738_b1536449308">redis</strong>.</p>
</td>
</tr>
<tr id="dli_08_0406__en-us_topic_0000001262655738_dli_08_0238_en-us_topic_0113887276_en-us_topic_0111499973_row210135125214"><td class="cellrowborder" valign="top" width="20.23%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0406__en-us_topic_0000001262655738_en-us_topic_0000001080679435_p5572731184713">host</p>
</td>
<td class="cellrowborder" valign="top" width="8.469999999999999%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0406__en-us_topic_0000001262655738_en-us_topic_0000001080679435_p3572133154715">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="11.86%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0406__en-us_topic_0000001262655738_p08030423126">None</p>
</td>
<td class="cellrowborder" valign="top" width="16.91%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0406__en-us_topic_0000001262655738_p10564846141213">String</p>
</td>
<td class="cellrowborder" valign="top" width="42.53%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0406__en-us_topic_0000001262655738_en-us_topic_0000001080679435_p1757263114475">Redis connector address</p>
</td>
</tr>
<tr id="dli_08_0406__en-us_topic_0000001262655738_dli_08_0238_en-us_topic_0113887276_en-us_topic_0111499973_row124195117528"><td class="cellrowborder" valign="top" width="20.23%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0406__en-us_topic_0000001262655738_en-us_topic_0000001080679435_p1057218314476">port</p>
</td>
<td class="cellrowborder" valign="top" width="8.469999999999999%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0406__en-us_topic_0000001262655738_en-us_topic_0000001080679435_p11572113114477">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.86%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0406__en-us_topic_0000001262655738_p1380319422125">6379</p>
</td>
<td class="cellrowborder" valign="top" width="16.91%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0406__en-us_topic_0000001262655738_p456424619121">Integer</p>
</td>
<td class="cellrowborder" valign="top" width="42.53%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0406__en-us_topic_0000001262655738_en-us_topic_0000001080679435_p657263154715">Redis connector port</p>
</td>
</tr>
<tr id="dli_08_0406__en-us_topic_0000001262655738_dli_08_0238_en-us_topic_0113887276_en-us_topic_0111499973_row15401751115211"><td class="cellrowborder" valign="top" width="20.23%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0406__en-us_topic_0000001262655738_en-us_topic_0000001080679435_p1457293194711">password</p>
</td>
<td class="cellrowborder" valign="top" width="8.469999999999999%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0406__en-us_topic_0000001262655738_en-us_topic_0000001080679435_p6572153120474">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.86%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0406__en-us_topic_0000001262655738_p23491624911">None</p>
</td>
<td class="cellrowborder" valign="top" width="16.91%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0406__en-us_topic_0000001262655738_p356417468127">String</p>
</td>
<td class="cellrowborder" valign="top" width="42.53%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0406__en-us_topic_0000001262655738_en-us_topic_0000001080679435_p1572331194712">Redis authentication password</p>
</td>
</tr>
<tr id="dli_08_0406__en-us_topic_0000001262655738_row15192195234419"><td class="cellrowborder" valign="top" width="20.23%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0406__en-us_topic_0000001262655738_p419212521448">namespace</p>
</td>
<td class="cellrowborder" valign="top" width="8.469999999999999%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0406__en-us_topic_0000001262655738_p1219295214413">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.86%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0406__en-us_topic_0000001262655738_p5571167104910">None</p>
</td>
<td class="cellrowborder" valign="top" width="16.91%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0406__en-us_topic_0000001262655738_p0192195264414">String</p>
</td>
<td class="cellrowborder" valign="top" width="42.53%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0406__en-us_topic_0000001262655738_p219255220440">Redis key namespace</p>
</td>
</tr>
<tr id="dli_08_0406__en-us_topic_0000001262655738_row1135235419445"><td class="cellrowborder" valign="top" width="20.23%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0406__en-us_topic_0000001262655738_p1435275418442">delimiter</p>
</td>
<td class="cellrowborder" valign="top" width="8.469999999999999%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0406__en-us_topic_0000001262655738_p23521554154418">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.86%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0406__en-us_topic_0000001262655738_p135265404414">:</p>
</td>
<td class="cellrowborder" valign="top" width="16.91%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0406__en-us_topic_0000001262655738_p1635219546445">String</p>
</td>
<td class="cellrowborder" valign="top" width="42.53%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0406__en-us_topic_0000001262655738_p5352454144413">Delimiter between the Redis key and namespace</p>
</td>
</tr>
<tr id="dli_08_0406__en-us_topic_0000001262655738_row665722864612"><td class="cellrowborder" valign="top" width="20.23%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0406__en-us_topic_0000001262655738_p5657182854620">data-type</p>
</td>
<td class="cellrowborder" valign="top" width="8.469999999999999%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0406__en-us_topic_0000001262655738_p06574284465">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.86%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0406__en-us_topic_0000001262655738_p6657192854619">hash</p>
</td>
<td class="cellrowborder" valign="top" width="16.91%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0406__en-us_topic_0000001262655738_p565752884615">String</p>
</td>
<td class="cellrowborder" valign="top" width="42.53%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0406__en-us_topic_0000001262655738_p1565702864615">Redis data type. Available values are as follows:</p>
<ul id="dli_08_0406__en-us_topic_0000001262655738_ul1995143910473"><li id="dli_08_0406__en-us_topic_0000001262655738_li6995193915475">hash</li><li id="dli_08_0406__en-us_topic_0000001262655738_li1727616434474">list</li><li id="dli_08_0406__en-us_topic_0000001262655738_li1966154411477">set</li><li id="dli_08_0406__en-us_topic_0000001262655738_li291412452478">sorted-set</li><li id="dli_08_0406__en-us_topic_0000001262655738_li107761449134720">string</li></ul>
<p id="dli_08_0406__en-us_topic_0000001262655738_p545113734915">For details about the constraints, see <a href="#dli_08_0406__en-us_topic_0000001262655738_li817313914378">Constraints on data-type</a>.</p>
</td>
</tr>
<tr id="dli_08_0406__en-us_topic_0000001262655738_row991541014813"><td class="cellrowborder" valign="top" width="20.23%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0406__en-us_topic_0000001262655738_p1091512109486">schema-syntax</p>
</td>
<td class="cellrowborder" valign="top" width="8.469999999999999%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0406__en-us_topic_0000001262655738_p491541024812">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.86%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0406__en-us_topic_0000001262655738_p3915131054817">fields</p>
</td>
<td class="cellrowborder" valign="top" width="16.91%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0406__en-us_topic_0000001262655738_p9915110124818">String</p>
</td>
<td class="cellrowborder" valign="top" width="42.53%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0406__en-us_topic_0000001262655738_p1319513182387">Redis schema semantics. Available values are as follows:</p>
<ul id="dli_08_0406__en-us_topic_0000001262655738_ul13496191015240"><li id="dli_08_0406__en-us_topic_0000001262655738_li64966101248"><strong id="dli_08_0406__en-us_topic_0000001262655738_b85831218513">fields</strong>: applicable to all data types</li><li id="dli_08_0406__en-us_topic_0000001262655738_li2627112019248"><strong id="dli_08_0406__en-us_topic_0000001262655738_b16942182116514">fields-scores</strong>: applicable to <strong id="dli_08_0406__en-us_topic_0000001262655738_b108942030451">sorted set</strong> data</li><li id="dli_08_0406__en-us_topic_0000001262655738_li16505143442414"><strong id="dli_08_0406__en-us_topic_0000001262655738_b1121418341853">array</strong>: applicable to <strong id="dli_08_0406__en-us_topic_0000001262655738_b09425371051">list</strong>, <strong id="dli_08_0406__en-us_topic_0000001262655738_b171051539554">set</strong>, and <strong id="dli_08_0406__en-us_topic_0000001262655738_b14386134117515">sorted set</strong> data</li><li id="dli_08_0406__en-us_topic_0000001262655738_li119731655102413"><strong id="dli_08_0406__en-us_topic_0000001262655738_b92781521518">array-scores</strong>: applicable to <strong id="dli_08_0406__en-us_topic_0000001262655738_b28148541155">sorted set</strong> data</li><li id="dli_08_0406__en-us_topic_0000001262655738_li1520510832516"><strong id="dli_08_0406__en-us_topic_0000001262655738_b25291959352">map</strong>: applicable to <strong id="dli_08_0406__en-us_topic_0000001262655738_b233818314618">hash</strong> and <strong id="dli_08_0406__en-us_topic_0000001262655738_b28861851867">sorted set</strong> data</li></ul>
<p id="dli_08_0406__en-us_topic_0000001262655738_p1884161010504">For details about the constraints, see <a href="#dli_08_0406__en-us_topic_0000001262655738_li1877444315214">Constraints on schema-syntax</a>.</p>
</td>
</tr>
<tr id="dli_08_0406__en-us_topic_0000001262655738_row5416114395"><td class="cellrowborder" valign="top" width="20.23%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0406__en-us_topic_0000001262655738_p104117118390">deploy-mode</p>
</td>
<td class="cellrowborder" valign="top" width="8.469999999999999%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0406__en-us_topic_0000001262655738_p641201111397">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.86%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0406__en-us_topic_0000001262655738_p680320427129">standalone</p>
</td>
<td class="cellrowborder" valign="top" width="16.91%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0406__en-us_topic_0000001262655738_p12564114631212">String</p>
</td>
<td class="cellrowborder" valign="top" width="42.53%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0406__en-us_topic_0000001262655738_p124161110399">Deployment mode of the Redis cluster. The value can be <strong id="dli_08_0406__en-us_topic_0000001262655738_b1956219529818">standalone</strong>, <strong id="dli_08_0406__en-us_topic_0000001262655738_b6647755686">master-replica</strong>, or <strong id="dli_08_0406__en-us_topic_0000001262655738_b157384588811">cluster</strong>. The default value is<strong id="dli_08_0406__en-us_topic_0000001262655738_b10695104917"> standalone</strong>.</p>
</td>
</tr>
<tr id="dli_08_0406__en-us_topic_0000001262655738_row11421243182419"><td class="cellrowborder" valign="top" width="20.23%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0406__en-us_topic_0000001262655738_p91429439241">retry-count</p>
</td>
<td class="cellrowborder" valign="top" width="8.469999999999999%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0406__en-us_topic_0000001262655738_p1214218431249">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="11.86%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0406__en-us_topic_0000001262655738_p185661846124714">5</p>
</td>
<td class="cellrowborder" valign="top" width="16.91%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0406__en-us_topic_0000001262655738_p16142343202411">Integer</p>
</td>
<td class="cellrowborder" valign="top" width="42.53%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0406__en-us_topic_0000001262655738_p8477733112317">Size of each connection request queue. If the number of connection requests in a queue exceeds the queue size, command calling will cause RedisException. Setting <strong id="dli_08_0406__en-us_topic_0000001262655738_b1699188115012">requestQueueSize</strong> to a small value will cause exceptions to occur earlier during overload or disconnection. A larger value indicates more time required to reach the boundary, but more requests may be queued and more heap space may be used. The default value is <strong id="dli_08_0406__en-us_topic_0000001262655738_b140111333490">2147483647</strong>.</p>
</td>
</tr>
<tr id="dli_08_0406__en-us_topic_0000001262655738_row35139459243"><td class="cellrowborder" valign="top" width="20.23%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0406__en-us_topic_0000001262655738_p5513174592413">connection-timeout-millis</p>
</td>
<td class="cellrowborder" valign="top" width="8.469999999999999%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0406__en-us_topic_0000001262655738_p12513164512415">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.86%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0406__en-us_topic_0000001262655738_p145134451249">10000</p>
</td>
<td class="cellrowborder" valign="top" width="16.91%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0406__en-us_topic_0000001262655738_p15131845162410">Integer</p>
</td>
<td class="cellrowborder" valign="top" width="42.53%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0406__en-us_topic_0000001262655738_p651384513248">Maximum timeout for connecting to the Redis cluster</p>
</td>
</tr>
<tr id="dli_08_0406__en-us_topic_0000001262655738_row129182132518"><td class="cellrowborder" valign="top" width="20.23%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0406__en-us_topic_0000001262655738_p10932162519">commands-timeout-millis</p>
</td>
<td class="cellrowborder" valign="top" width="8.469999999999999%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0406__en-us_topic_0000001262655738_p159122112518">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.86%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0406__en-us_topic_0000001262655738_p59320251">2000</p>
</td>
<td class="cellrowborder" valign="top" width="16.91%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0406__en-us_topic_0000001262655738_p139728259">Integer</p>
</td>
<td class="cellrowborder" valign="top" width="42.53%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0406__en-us_topic_0000001262655738_p181022202517">Maximum time for waiting for a completion response</p>
</td>
</tr>
<tr id="dli_08_0406__en-us_topic_0000001262655738_row145384872415"><td class="cellrowborder" valign="top" width="20.23%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0406__en-us_topic_0000001262655738_p7541748102418">rebalancing-timeout-millis</p>
</td>
<td class="cellrowborder" valign="top" width="8.469999999999999%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0406__en-us_topic_0000001262655738_p19541448152415">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.86%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0406__en-us_topic_0000001262655738_p5545483242">15000</p>
</td>
<td class="cellrowborder" valign="top" width="16.91%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0406__en-us_topic_0000001262655738_p13548489244">Integer</p>
</td>
<td class="cellrowborder" valign="top" width="42.53%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0406__en-us_topic_0000001262655738_p1854448102418">Sleep time when the Redis cluster fails</p>
</td>
</tr>
<tr id="dli_08_0406__en-us_topic_0000001262655738_row393516498245"><td class="cellrowborder" valign="top" width="20.23%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0406__en-us_topic_0000001262655738_p1493518491244">scan-keys-count</p>
</td>
<td class="cellrowborder" valign="top" width="8.469999999999999%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0406__en-us_topic_0000001262655738_p139351949172411">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.86%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0406__en-us_topic_0000001262655738_p5935194982413">1000</p>
</td>
<td class="cellrowborder" valign="top" width="16.91%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0406__en-us_topic_0000001262655738_p18935149162419">Integer</p>
</td>
<td class="cellrowborder" valign="top" width="42.53%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0406__en-us_topic_0000001262655738_p18935104910246">Number of data records read in each scan</p>
</td>
</tr>
<tr id="dli_08_0406__en-us_topic_0000001262655738_row15644042519"><td class="cellrowborder" valign="top" width="20.23%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0406__en-us_topic_0000001262655738_p1864110102520">default-score</p>
</td>
<td class="cellrowborder" valign="top" width="8.469999999999999%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0406__en-us_topic_0000001262655738_p186414012510">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.86%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0406__en-us_topic_0000001262655738_p106416014258">0</p>
</td>
<td class="cellrowborder" valign="top" width="16.91%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0406__en-us_topic_0000001262655738_p106410020257">Double</p>
</td>
<td class="cellrowborder" valign="top" width="42.53%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0406__en-us_topic_0000001262655738_p1464160172514">Default score when <strong id="dli_08_0406__en-us_topic_0000001262655738_b11533176191214">data-type</strong> is <strong id="dli_08_0406__en-us_topic_0000001262655738_b11865911101213">sorted-set</strong></p>
</td>
</tr>
<tr id="dli_08_0406__en-us_topic_0000001262655738_row170253422011"><td class="cellrowborder" valign="top" width="20.23%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0406__en-us_topic_0000001262655738_p157021934122019">deserialize-error-policy</p>
</td>
<td class="cellrowborder" valign="top" width="8.469999999999999%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0406__en-us_topic_0000001262655738_p14702183422010">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.86%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0406__en-us_topic_0000001262655738_p1670212343201">fail-job</p>
</td>
<td class="cellrowborder" valign="top" width="16.91%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0406__en-us_topic_0000001262655738_p1170217345202">Enum</p>
</td>
<td class="cellrowborder" valign="top" width="42.53%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0406__en-us_topic_0000001262655738_p61681019142213">How to process a data parsing failure</p>
<p id="dli_08_0406__en-us_topic_0000001262655738_p2702123411200">Available values are as follows:</p>
<ul id="dli_08_0406__en-us_topic_0000001262655738_ul399111416217"><li id="dli_08_0406__en-us_topic_0000001262655738_li4991144152111"><strong id="dli_08_0406__en-us_topic_0000001262655738_b162962021171412">fail-job</strong>: Fail the job</li><li id="dli_08_0406__en-us_topic_0000001262655738_li13319142118223"><strong id="dli_08_0406__en-us_topic_0000001262655738_b698753481414">skip-row</strong>: Skip the current data.</li><li id="dli_08_0406__en-us_topic_0000001262655738_li23961035182214"><strong id="dli_08_0406__en-us_topic_0000001262655738_b492118457146">null-field</strong>: Set the current data to null.</li></ul>
</td>
</tr>
<tr id="dli_08_0406__en-us_topic_0000001262655738_row142541352162412"><td class="cellrowborder" valign="top" width="20.23%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0406__en-us_topic_0000001262655738_p2254135215244">skip-null-values</p>
</td>
<td class="cellrowborder" valign="top" width="8.469999999999999%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0406__en-us_topic_0000001262655738_p225445262410">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.86%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0406__en-us_topic_0000001262655738_p425405214241">true</p>
</td>
<td class="cellrowborder" valign="top" width="16.91%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0406__en-us_topic_0000001262655738_p52541552162411">Boolean</p>
</td>
<td class="cellrowborder" valign="top" width="42.53%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0406__en-us_topic_0000001262655738_p02543525248">Whether null values will be skipped</p>
</td>
</tr>
<tr id="dli_08_0406__en-us_topic_0000001262655738_row599851484714"><td class="cellrowborder" valign="top" width="20.23%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0406__en-us_topic_0000001262655738_p13998101419474">lookup.async</p>
</td>
<td class="cellrowborder" valign="top" width="8.469999999999999%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0406__en-us_topic_0000001262655738_p799871434719">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.86%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0406__en-us_topic_0000001262655738_p1899871414710">false</p>
</td>
<td class="cellrowborder" valign="top" width="16.91%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0406__en-us_topic_0000001262655738_p0998151410475">Boolean</p>
</td>
<td class="cellrowborder" valign="top" width="42.53%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0406__en-us_topic_0000001262655738_p099851414477">Whether asynchronous I/O will be used when this table is used as a dimension table</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="dli_08_0406__en-us_topic_0000001262655738_section1078477489"><h4 class="sectiontitle">Example</h4><p id="dli_08_0406__en-us_topic_0000001262655738_dli_08_0261_p88543522162">Read data from a Kafka source table, use a Redis table as the dimension table. Write wide table information generated by the source and dimension tables to a Kafka result table. The procedure is as follows:</p>
<ol id="dli_08_0406__en-us_topic_0000001262655738_ol10282201165111"><li id="dli_08_0406__en-us_topic_0000001262655738_li6127741153217">Create an enhanced datasource connection in the VPC and subnet where Redis and Kafka locates, and bind the connection to the required Flink elastic resource pool.</li><li id="dli_08_0406__en-us_topic_0000001262655738_li1233002211514">Set Redis and Kafka security groups and add inbound rules to allow access from the Flink queue. Test the connectivity using the Redis address. If the connection passes the test, it is bound to the queue.</li><li id="dli_08_0406__en-us_topic_0000001262655738_li12282171110516">Run the following commands on the Redis client to send data to Redis:<pre class="screen" id="dli_08_0406__en-us_topic_0000001262655738_screen195757190557">HMSET 330102 area_province_name a1 area_province_name b1 area_county_name c1 area_street_name d1 region_name e1
HMSET 330106 area_province_name a1 area_province_name b1 area_county_name c2 area_street_name d2 region_name e1
HMSET 330108 area_province_name a1 area_province_name b1 area_county_name c3 area_street_name d3 region_name e1
HMSET 330110 area_province_name a1 area_province_name b1 area_county_name c4 area_street_name d4 region_name e1</pre>
</li><li id="dli_08_0406__en-us_topic_0000001262655738_li228251135112">Create a Flink OpenSource SQL job Enter the following job script and submit the job. The job script uses Kafka as the data source and a Redis table as the dimension table. Data is output to a Kafka result table.<div class="p" id="dli_08_0406__en-us_topic_0000001262655738_p265512280533"><a name="dli_08_0406__en-us_topic_0000001262655738_li228251135112"></a><a name="en-us_topic_0000001262655738_li228251135112"></a>When you create a job, set <strong id="dli_08_0406__en-us_topic_0000001262655738_b1869621016311">Flink Version</strong> to <strong id="dli_08_0406__en-us_topic_0000001262655738_b7697171013312">1.12</strong> in the <strong id="dli_08_0406__en-us_topic_0000001262655738_b1069721013119">Running Parameters</strong> tab. Select <strong id="dli_08_0406__en-us_topic_0000001262655738_b12697201063110">Save Job Log</strong>, and specify the OBS bucket for saving job logs. <strong id="dli_08_0406__en-us_topic_0000001262655738_b596213125315">Set the values of the parameters in bold in the following script as needed.</strong><pre class="screen" id="dli_08_0406__en-us_topic_0000001262655738_screen2470112715531">CREATE TABLE orders (
order_id string,
order_channel string,
order_time string,
pay_amount double,
real_pay double,
pay_time string,
user_id string,
user_name string,
area_id string,
proctime as Proctime()
) WITH (
'connector' = 'kafka',
'topic' = '<em id="dli_08_0406__en-us_topic_0000001262655738_i4179144718454"><strong id="dli_08_0406__en-us_topic_0000001262655738_b203861310114613">kafkaSourceTopic</strong></em>',
'properties.bootstrap.servers' = '<em id="dli_08_0406__en-us_topic_0000001262655738_i9319103215313"><strong id="dli_08_0406__en-us_topic_0000001262655738_b0322115133213">KafkaAddress1:KafkaPort,KafkaAddress2:KafkaPort</strong></em>',
'properties.group.id' = '<em id="dli_08_0406__en-us_topic_0000001262655738_i14535174011311"><strong id="dli_08_0406__en-us_topic_0000001262655738_b1466610178329">GroupId</strong></em>',
'scan.startup.mode' = 'latest-offset',
'format' = 'json'
);
-- Create an address dimension table
create table area_info (
area_id string,
area_province_name string,
area_city_name string,
area_county_name string,
area_street_name string,
region_name string,
primary key (area_id) not enforced -- Redis key
) WITH (
'connector' = 'redis',
'host' = '<em id="dli_08_0406__en-us_topic_0000001262655738_i846017301461"><strong id="dli_08_0406__en-us_topic_0000001262655738_b4453552144615">RedisIP</strong></em>',
'password' = '<em id="dli_08_0406__en-us_topic_0000001262655738_i20368164784613"><strong id="dli_08_0406__en-us_topic_0000001262655738_b4161454134619">Redis</strong></em><em id="dli_08_0406__en-us_topic_0000001262655738_i207251450154610"><strong id="dli_08_0406__en-us_topic_0000001262655738_b181642544464">Password</strong></em>',
'data-type' = 'hash',
'deploy-mode' = 'master-replica'
);
-- Generate a wide table based on the address dimension table containing detailed order information.
create table order_detail(
order_id string,
order_channel string,
order_time string,
pay_amount double,
real_pay double,
pay_time string,
user_id string,
user_name string,
area_id string,
area_province_name string,
area_city_name string,
area_county_name string,
area_street_name string,
region_name string
) with (
'connector' = 'kafka',
'topic' = '<em id="dli_08_0406__en-us_topic_0000001262655738_i1469591335514"><strong id="dli_08_0406__en-us_topic_0000001262655738_b76951132557">kafkaSinkTopic</strong></em>',
'properties.bootstrap.servers' = '<em id="dli_08_0406__en-us_topic_0000001262655738_i116116315553"><strong id="dli_08_0406__en-us_topic_0000001262655738_b196113312552">KafkaAddress1:KafkaPort,KafkaAddress2:KafkaPort</strong></em>',
'format' = 'json'
);
insert into order_detail
select orders.order_id, orders.order_channel, orders.order_time, orders.pay_amount, orders.real_pay, orders.pay_time, orders.user_id, orders.user_name,
area.area_id, area.area_province_name, area.area_city_name, area.area_county_name,
area.area_street_name, area.region_name from orders
left join area_info for system_time as of orders.proctime as area on orders.area_id = area.area_id;</pre>
</div>
</li><li id="dli_08_0406__en-us_topic_0000001262655738_li62821711185115">Connect to the Kafka cluster and insert the following test data into the source topic in Kafka:<pre class="screen" id="dli_08_0406__en-us_topic_0000001262655738_screen18607164314615">{"order_id":"202103241606060001", "order_channel":"appShop", "order_time":"2021-03-24 16:06:06", "pay_amount":"200.00", "real_pay":"180.00", "pay_time":"2021-03-24 16:10:06", "user_id":"0001", "user_name":"Alice", "area_id":"330106"}
{"order_id":"202103251202020001", "order_channel":"miniAppShop", "order_time":"2021-03-25 12:02:02", "pay_amount":"60.00", "real_pay":"60.00", "pay_time":"2021-03-25 12:03:00", "user_id":"0002", "user_name":"Bob", "area_id":"330110"}
{"order_id":"202103251505050001", "order_channel":"qqShop", "order_time":"2021-03-25 15:05:05", "pay_amount":"500.00", "real_pay":"400.00", "pay_time":"2021-03-25 15:10:00", "user_id":"0003", "user_name":"Cindy", "area_id":"330108"}</pre>
</li><li id="dli_08_0406__en-us_topic_0000001262655738_li1328210111516">Connect to the Kafka cluster and read data from the sink topic of Kafka. The result data is as follows:<pre class="screen" id="dli_08_0406__en-us_topic_0000001262655738_screen1853311201071">{"order_id":"202103241606060001","order_channel":"appShop","order_time":"2021-03-24 16:06:06","pay_amount":200.0,"real_pay":180.0,"pay_time":"2021-03-24 16:10:06","user_id":"0001","user_name":"Alice","area_id":"330106","area_province_name":"a1","area_city_name":"b1","area_county_name":"c2","area_street_name":"d2","region_name":"e1"}
{"order_id":"202103251202020001","order_channel":"miniAppShop","order_time":"2021-03-25 12:02:02","pay_amount":60.0,"real_pay":60.0,"pay_time":"2021-03-25 12:03:00","user_id":"0002","user_name":"Bob","area_id":"330110","area_province_name":"a1","area_city_name":"b1","area_county_name":"c4","area_street_name":"d4","region_name":"e1"}
{"order_id":"202103251505050001","order_channel":"qqShop","order_time":"2021-03-25 15:05:05","pay_amount":500.0,"real_pay":400.0,"pay_time":"2021-03-25 15:10:00","user_id":"0003","user_name":"Cindy","area_id":"330108","area_province_name":"a1","area_city_name":"b1","area_county_name":"c3","area_street_name":"d3","region_name":"e1"}
</pre>
</li></ol>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0402.html">Creating Dimension Tables</a></div>
</div>
</div>
View Git Blame Copy Permalink