forked from docs/doc-exports
Su, Xiaomeng
be9eabe464
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com> Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
33 lines
2.4 KiB
HTML
33 lines
2.4 KiB
HTML
<a name="dli_08_15080"></a><a name="dli_08_15080"></a>
|
|
|
|
<h1 class="topictitle1">Deduplication</h1>
|
|
<div id="body0000001262495802"><div class="section" id="dli_08_15080__section105585025918"><h4 class="sectiontitle">Function</h4><p id="dli_08_15080__p205386514595">Deduplication removes rows that duplicate over a set of columns, keeping only the first one or the last one.</p>
|
|
</div>
|
|
<div class="section" id="dli_08_15080__section155961853135910"><h4 class="sectiontitle">Syntax</h4><pre class="screen" id="dli_08_15080__screen152822010020">SELECT [column_list]
|
|
FROM (
|
|
SELECT [column_list],
|
|
ROW_NUMBER() OVER ([PARTITION BY col1[, col2...]]
|
|
ORDER BY time_attr [asc|desc]) AS rownum
|
|
FROM table_name)
|
|
WHERE rownum = 1</pre>
|
|
</div>
|
|
<div class="section" id="dli_08_15080__section1627916611011"><h4 class="sectiontitle">Description</h4><ul id="dli_08_15080__ul12220145108"><li id="dli_08_15080__li82201045703">ROW_NUMBER(): Assigns a unique, sequential number to each row, starting with one.</li><li id="dli_08_15080__li152201045504">PARTITION BY col1[, col2...]: Specifies the partition columns, i.e. the deduplicate key.</li><li id="dli_08_15080__li1422020451403">ORDER BY time_attr [asc|desc]: Specifies the ordering column, it must be a time attribute. Currently Flink supports proctime only. Ascending (ASC) sorting keeps only the first row, while descending (DESC) sorting keeps only the last row.</li><li id="dli_08_15080__li1922012451800">WHERE rownum = 1: The rownum = 1 is required for Flink to recognize this query is deduplication.</li></ul>
|
|
</div>
|
|
<div class="section" id="dli_08_15080__section17171113913120"><h4 class="sectiontitle">Precautions</h4><p id="dli_08_15080__p10805114812117">None</p>
|
|
</div>
|
|
<div class="section" id="dli_08_15080__section1085612512115"><h4 class="sectiontitle">Example</h4><p id="dli_08_15080__p10785617548">The following examples show how to remove duplicate rows on <strong id="dli_08_15080__b93306903145546">order_id</strong>. The proctime is an event time attribute.</p>
|
|
<pre class="screen" id="dli_08_15080__screen13863297215">SELECT order_id, user, product, number
|
|
FROM (
|
|
SELECT *,
|
|
ROW_NUMBER() OVER (PARTITION BY order_id ORDER BY proctime ASC) as row_num
|
|
FROM Orders)
|
|
WHERE row_num = 1;</pre>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_15066.html">DML Snytax</a></div>
|
|
</div>
|
|
</div>
|
|
|