forked from docs/doc-exports
Su, Xiaomeng
be9eabe464
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com> Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
185 lines
12 KiB
HTML
185 lines
12 KiB
HTML
<a name="dli_08_15084"></a><a name="dli_08_15084"></a>
|
|
|
|
<h1 class="topictitle1">Parameter Transfer</h1>
|
|
<div id="body0000001401516552"><div class="section" id="dli_08_15084__section13531550393"><h4 class="sectiontitle">Scenario</h4><p id="dli_08_15084__p61072059594">A UDF can be used in many jobs, and some parameter values vary with jobs. To easily modify the parameter values, you can set <strong id="dli_08_15084__b153977517119">pipeline.global-job-parameters</strong> in the <strong id="dli_08_15084__b836521814551">Runtime Configuration</strong> tab on the Flink OpenSource SQL editing page, and then get the parameter values in the UDF code and use the values as you need. You only need to change the parameter values in the runtime configuration tab to pass the new values to the UDF.</p>
|
|
</div>
|
|
<div class="section" id="dli_08_15084__section61961713131213"><h4 class="sectiontitle">Procedure</h4><p id="dli_08_15084__p658892081218">Use the open(FunctionContext context) method in your UDF to pass parameters through a FunctionContext object. To pass parameters to a job, perform the following steps:</p>
|
|
<ol id="dli_08_15084__ol81481830141719"><li id="dli_08_15084__li10148123018170">Add <strong id="dli_08_15084__b11172033153414">pipeline.global-job-parameters</strong> to <strong id="dli_08_15084__b198145442342">Runtime Configuration</strong> on the Flink OpenSource SQL editing page. The format is as follows:<pre class="screen" id="dli_08_15084__screen571181914134">pipeline.global-job-parameters=k1:v1,"k2:v1,v2",k3:"str:ing","k4:str""ing"</pre>
|
|
<p id="dli_08_15084__p3108181451311">This configuration defines a map as shown in <a href="#dli_08_15084__table1658123913138">Table 1</a></p>
|
|
|
|
<div class="tablenoborder"><a name="dli_08_15084__table1658123913138"></a><a name="table1658123913138"></a><table cellpadding="4" cellspacing="0" summary="" id="dli_08_15084__table1658123913138" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Examples for pipeline.global-job-parameters</caption><thead align="left"><tr id="dli_08_15084__row659143913139"><th align="left" class="cellrowborder" valign="top" width="39.68%" id="mcps1.3.2.3.1.5.2.3.1.1"><p id="dli_08_15084__p16598391136">Key</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="60.31999999999999%" id="mcps1.3.2.3.1.5.2.3.1.2"><p id="dli_08_15084__p165993901311">Value</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="dli_08_15084__row125953931313"><td class="cellrowborder" valign="top" width="39.68%" headers="mcps1.3.2.3.1.5.2.3.1.1 "><p id="dli_08_15084__p185983913136">k1</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.31999999999999%" headers="mcps1.3.2.3.1.5.2.3.1.2 "><p id="dli_08_15084__p95913911130">v1</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_15084__row185923941316"><td class="cellrowborder" valign="top" width="39.68%" headers="mcps1.3.2.3.1.5.2.3.1.1 "><p id="dli_08_15084__p205914395139">k2</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.31999999999999%" headers="mcps1.3.2.3.1.5.2.3.1.2 "><p id="dli_08_15084__p359173961319">v1,v2</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_15084__row527917471138"><td class="cellrowborder" valign="top" width="39.68%" headers="mcps1.3.2.3.1.5.2.3.1.1 "><p id="dli_08_15084__p9279144751318">k3</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.31999999999999%" headers="mcps1.3.2.3.1.5.2.3.1.2 "><p id="dli_08_15084__p5279347171311">str:ing</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_15084__row16240545101314"><td class="cellrowborder" valign="top" width="39.68%" headers="mcps1.3.2.3.1.5.2.3.1.1 "><p id="dli_08_15084__p1424020451132">k4</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.31999999999999%" headers="mcps1.3.2.3.1.5.2.3.1.2 "><p id="dli_08_15084__p15240845161316">str""ing</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
<div class="note" id="dli_08_15084__note16172709153"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="dli_08_15084__ul168496501811"><li id="dli_08_15084__li1081941620180"><strong id="dli_08_15084__b1483316149363">FunctionContext#getJobParameter</strong> obtains only the value of <strong id="dli_08_15084__b2886143593616">pipeline.global-job-parameters</strong>. You need to add all key-value pairs that will be used in the UDF to <strong id="dli_08_15084__b178004293370">pipeline.global-job-parameters</strong>.</li><li id="dli_08_15084__li1184915141820">Keys and values are separated by colons (:). All key-values are connected by commas (,).</li><li id="dli_08_15084__li78496551812">If the key or value contains commas (,), use double quotation marks (") to enclose key or value, for example, <strong id="dli_08_15084__b133191320143916">"v1,v2"</strong>.</li><li id="dli_08_15084__li118494510180">If the key or value contains colons (:), use double quotation marks (") to enclose the key or value, for example, <strong id="dli_08_15084__b10491445154015">"str:ing"</strong>.</li><li id="dli_08_15084__li08491571820">If the key or value contains a double quotation mark("), use another double quotation mark ("") to escape the first one, and use double quotation marks (") to enclose the key or value, for example, <strong id="dli_08_15084__b10811155811349">"str""ing"</strong>.</li></ul>
|
|
</div></div>
|
|
</li><li id="dli_08_15084__li954133871711">In your UDF code, use <strong id="dli_08_15084__b1562573494410">FunctionContext#getJobParameter</strong> to obtain the key-value pairs you set. The code example is as follows:<pre class="screen" id="dli_08_15084__screen12668149201811">context.getJobParameter("url","jdbc:mysql://xx.xx.xx.xx:3306/table");
|
|
context.getJobParameter("driver","com.mysql.jdbc.Driver");
|
|
context.getJobParameter("user","user");
|
|
context.getJobParameter("password","password");</pre>
|
|
</li></ol>
|
|
</div>
|
|
<div class="section" id="dli_08_15084__section10300195911185"><h4 class="sectiontitle">Code Samples</h4><p id="dli_08_15084__p8501181881910">The following sample UDF uses <strong id="dli_08_15084__b18308133234619">pipeline.global-job-parameters</strong> to pass parameters such as <strong id="dli_08_15084__b147710382462">url</strong>, <strong id="dli_08_15084__b10883039124615">user</strong>, and <strong id="dli_08_15084__b2243134274620">password</strong> required for connecting to the database, obtains the <strong id="dli_08_15084__b02695974613">udf_info</strong> table data, and combines this data with the stream data into JSON output.</p>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_15084__table85769310354" frame="border" border="1" rules="all"><caption><b>Table 2 </b>udf_info</caption><thead align="left"><tr id="dli_08_15084__row1257616312359"><th align="left" class="cellrowborder" valign="top" width="39.68%" id="mcps1.3.3.3.2.3.1.1"><p id="dli_08_15084__p65761039357">key</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="60.31999999999999%" id="mcps1.3.3.3.2.3.1.2"><p id="dli_08_15084__p1057612323514">value</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="dli_08_15084__row1957619317351"><td class="cellrowborder" valign="top" width="39.68%" headers="mcps1.3.3.3.2.3.1.1 "><p id="dli_08_15084__p55765314354">class</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.31999999999999%" headers="mcps1.3.3.3.2.3.1.2 "><p id="dli_08_15084__p155763319354">class-4</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
<p id="dli_08_15084__p1824410511913">SimpleJsonBuild.java</p>
|
|
<pre class="screen" id="dli_08_15084__screen24461234122312">package udf;
|
|
|
|
import com.fasterxml.jackson.databind.ObjectMapper;
|
|
|
|
import org.apache.flink.table.functions.FunctionContext;
|
|
import org.apache.flink.table.functions.ScalarFunction;
|
|
import org.slf4j.Logger;
|
|
import org.slf4j.LoggerFactory;
|
|
|
|
import java.io.IOException;
|
|
import java.sql.Connection;
|
|
import java.sql.DriverManager;
|
|
import java.sql.PreparedStatement;
|
|
import java.sql.ResultSet;
|
|
import java.util.HashMap;
|
|
import java.util.Map;
|
|
|
|
public class SimpleJsonBuild extends ScalarFunction {
|
|
private static final Logger LOG = LoggerFactory.getLogger(SimpleJsonBuild.class);
|
|
String remainedKey;
|
|
String remainedValue;
|
|
|
|
private Connection initConnection(Map<String, String> userParasMap) {
|
|
String url = userParasMap.get("url");
|
|
String driver = userParasMap.get("driver");
|
|
String user = userParasMap.get("user");
|
|
String password = userParasMap.get("password");
|
|
Connection conn = null;
|
|
try {
|
|
Class.forName(driver);
|
|
conn = DriverManager.getConnection(url, user, password);
|
|
LOG.info("connect successfully");
|
|
} catch (Exception e) {
|
|
LOG.error(String.valueOf(e));
|
|
}
|
|
return conn;
|
|
}
|
|
|
|
@Override
|
|
public void open(FunctionContext context) throws Exception {
|
|
Map<String, String> userParasMap = new HashMap<>();
|
|
Connection connection;
|
|
PreparedStatement pstmt;
|
|
ResultSet rs;
|
|
|
|
String url = context.getJobParameter("url","jdbc:mysql://xx.xx.xx.xx:3306/table");
|
|
String driver = context.getJobParameter("driver","com.mysql.jdbc.Driver");
|
|
String user = context.getJobParameter("user","user");
|
|
String password = context.getJobParameter("password","password");
|
|
|
|
userParasMap.put("url", url);
|
|
userParasMap.put("driver", driver);
|
|
userParasMap.put("user", user);
|
|
userParasMap.put("password", password);
|
|
|
|
connection = initConnection(userParasMap);
|
|
String sql = "select `key`, `value` from udf_info";
|
|
pstmt = connection.prepareStatement(sql);
|
|
rs = pstmt.executeQuery();
|
|
|
|
while (rs.next()) {
|
|
remainedKey = rs.getString(1);
|
|
remainedValue = rs.getString(2);
|
|
}
|
|
}
|
|
|
|
public String eval(String... params) throws IOException {
|
|
if (params != null && params.length != 0 && params.length % 2 <= 0) {
|
|
HashMap<String, String> hashMap = new HashMap();
|
|
for (int i = 0; i < params.length; i += 2) {
|
|
hashMap.put(params[i], params[i + 1]);
|
|
LOG.debug("now the key is " + params[i].toString() + "; now the value is " + params[i + 1].toString());
|
|
}
|
|
hashMap.put(remainedKey, remainedValue);
|
|
ObjectMapper mapper = new ObjectMapper();
|
|
String result = "{}";
|
|
try {
|
|
result = mapper.writeValueAsString(hashMap);
|
|
} catch (Exception ex) {
|
|
LOG.error("Get result failed." + ex.getMessage());
|
|
}
|
|
LOG.debug(result);
|
|
return result;
|
|
} else {
|
|
return "{}";
|
|
}
|
|
}
|
|
|
|
public static void main(String[] args) throws IOException {
|
|
SimpleJsonBuild sjb = new SimpleJsonBuild();
|
|
System.out.println(sjb.eval("json1", "json2", "json3", "json4"));
|
|
}
|
|
}</pre>
|
|
<p id="dli_08_15084__p66162092313">Add <strong id="dli_08_15084__b1289017456480">pipeline.global-job-parameters</strong> to <strong id="dli_08_15084__b17891174515488">Runtime Configuration</strong> on the Flink OpenSource SQL editing page. The format is as follows:</p>
|
|
<pre class="screen" id="dli_08_15084__screen911731152416">pipeline.global-job-parameters=url:'jdbc:mysql://x.x.x.x:xxxx/test',driver:com.mysql.jdbc.Driver,user:xxx,password:xxx</pre>
|
|
<p id="dli_08_15084__p1896232819242">Flink OpenSource SQL</p>
|
|
<pre class="screen" id="dli_08_15084__screen1188611527259">create function SimpleJsonBuild AS 'udf.SimpleJsonBuild';
|
|
create table dataGenSource(user_id string, amount int) with (
|
|
'connector' = 'datagen',
|
|
'rows-per-second' = '1', --Generate a data record per second.
|
|
'fields.user_id.kind' = 'random', --Specify a random generator for the user_id field.
|
|
'fields.user_id.length' = '3' --Limit the length of user_id to 3.
|
|
);
|
|
create table printSink(message STRING) with ('connector' = 'print');
|
|
insert into
|
|
printSink
|
|
SELECT
|
|
SimpleJsonBuild("name", user_id, "age", cast(amount as string))
|
|
from
|
|
dataGenSource;</pre>
|
|
</div>
|
|
<div class="section" id="dli_08_15084__section1247126192613"><h4 class="sectiontitle">Output</h4><p id="dli_08_15084__p2315155333014">On the Flin Jobs page, locate your job, and click <strong id="dli_08_15084__b158561459179">More</strong> > <strong id="dli_08_15084__b1863171685">FlinkUI</strong> in the <strong id="dli_08_15084__b1282416261810">Operation</strong> column. On the displayed page, click <strong id="dli_08_15084__b18944183819810">Task Managers</strong> > <strong id="dli_08_15084__b3633144018813">Stdout</strong> to view the job output.</p>
|
|
<p id="dli_08_15084__p2143151011263"><span><img id="dli_08_15084__image1350521020269" src="en-us_image_0000001742720709.gif"></span></p>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_15081.html">Functions</a></div>
|
|
</div>
|
|
</div>
|
|
|