vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/dws/umn/dws_03_0080.html
Lu, Huayi 7505fe85a9 DWS UMN 8.1.3.200 VERSION 
Reviewed-by: Kacur, Michal <michal.kacur@t-systems.com>
Co-authored-by: Lu, Huayi <luhuayi@huawei.com>
Co-committed-by: Lu, Huayi <luhuayi@huawei.com>
2023-08-31 14:40:33 +00:00

144 lines
24 KiB
HTML
Raw Blame History

<a name="EN-US_TOPIC_0000001587507168"></a><a name="EN-US_TOPIC_0000001587507168"></a>
<h1 class="topictitle1">How Do I Implement Fault Tolerance Import Between Different Encoding Libraries</h1>
<div id="body0000001587507168"><p id="EN-US_TOPIC_0000001587507168__p57961327143912">To import data from database A (UTF8) to database B (GBK), there may be a character set mismatch error which causes the data import to fail.</p>
<p id="EN-US_TOPIC_0000001587507168__p8060118">To import a small amount of data, run the <strong id="EN-US_TOPIC_0000001587507168__b12626323155620">\COPY</strong> command. The procedure is as follows:</p>
<ol id="EN-US_TOPIC_0000001587507168__ol154051915164213"><li id="EN-US_TOPIC_0000001587507168__li24051715124211"><span>Create databases A and B. The encoding format of database A is UTF8, and that of database B is GBK.</span><p><div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001587507168__screen1555159154219"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">postgres</span><span class="o">=&gt;</span><span class="w"> </span><span class="k">CREATE</span><span class="w"> </span><span class="k">DATABASE</span><span class="w"> </span><span class="n">A</span><span class="w"> </span><span class="k">ENCODING</span><span class="w"> </span><span class="s1">'UTF8'</span><span class="w"> </span><span class="k">template</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">template0</span><span class="p">;</span>
<span class="n">postgres</span><span class="o">=&gt;</span><span class="w"> </span><span class="k">CREATE</span><span class="w"> </span><span class="k">DATABASE</span><span class="w"> </span><span class="n">B</span><span class="w"> </span><span class="k">ENCODING</span><span class="w"> </span><span class="s1">'GBK'</span><span class="w"> </span><span class="k">template</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">template0</span><span class="p">;</span>
</pre></div></td></tr></table></div>
</div>
</p></li><li id="EN-US_TOPIC_0000001587507168__li1780012818458"><span>View the database list. You can view the created databases A and B.</span><p><div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001587507168__screen20136125420587"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">postgres</span><span class="o">=&gt;</span><span class="w"> </span><span class="err">\</span><span class="n">l</span>
<span class="w"> </span><span class="n">List</span><span class="w"> </span><span class="k">of</span><span class="w"> </span><span class="n">databases</span>
<span class="w"> </span><span class="n">Name</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="k">Owner</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="k">Encoding</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="k">Collate</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">Ctype</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="k">Access</span><span class="w"> </span><span class="k">privileges</span>
<span class="c1">-----------+---------+-----------+---------+-------+-------------------</span>
<span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">dbadmin</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">UTF8</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="k">C</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="k">C</span><span class="w"> </span><span class="o">|</span>
<span class="w"> </span><span class="n">b</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">dbadmin</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">GBK</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="k">C</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="k">C</span><span class="w"> </span><span class="o">|</span>
<span class="w"> </span><span class="n">gaussdb</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">Ruby</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">SQL_ASCII</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="k">C</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="k">C</span><span class="w"> </span><span class="o">|</span>
<span class="w"> </span><span class="n">postgres</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">Ruby</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">SQL_ASCII</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="k">C</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="k">C</span><span class="w"> </span><span class="o">|</span>
<span class="w"> </span><span class="n">template0</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">Ruby</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">SQL_ASCII</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="k">C</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="k">C</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="o">=</span><span class="k">c</span><span class="o">/</span><span class="n">Ruby</span><span class="w"> </span><span class="o">+</span>
<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">Ruby</span><span class="o">=</span><span class="n">CTc</span><span class="o">/</span><span class="n">Ruby</span>
<span class="w"> </span><span class="n">template1</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">Ruby</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">SQL_ASCII</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="k">C</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="k">C</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="o">=</span><span class="k">c</span><span class="o">/</span><span class="n">Ruby</span><span class="w"> </span><span class="o">+</span>
<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">Ruby</span><span class="o">=</span><span class="n">CTc</span><span class="o">/</span><span class="n">Ruby</span>
<span class="w"> </span><span class="n">xiaodi</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">dbadmin</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">UTF8</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="k">C</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="k">C</span><span class="w"> </span><span class="o">|</span>
<span class="p">(</span><span class="mi">7</span><span class="w"> </span><span class="k">rows</span><span class="p">)</span>
</pre></div></td></tr></table></div>
</div>
</p></li><li id="EN-US_TOPIC_0000001587507168__li144051715134219"><span>Switch to database A and enter the user password. Create a table named <strong id="EN-US_TOPIC_0000001587507168__b156951417121516">test01</strong> and insert data into the table.</span><p><div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001587507168__screen153566567118"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span>
<span class="normal">16</span>
<span class="normal">17</span>
<span class="normal">18</span>
<span class="normal">19</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">postgres</span><span class="o">=&gt;</span><span class="w"> </span><span class="err">\</span><span class="k">c</span><span class="w"> </span><span class="n">a</span>
<span class="n">Password</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="k">user</span><span class="w"> </span><span class="n">dbadmin</span><span class="p">:</span>
<span class="n">SSL</span><span class="w"> </span><span class="k">connection</span><span class="w"> </span><span class="p">(</span><span class="n">protocol</span><span class="p">:</span><span class="w"> </span><span class="n">TLSv1</span><span class="p">.</span><span class="mi">3</span><span class="p">,</span><span class="w"> </span><span class="n">cipher</span><span class="p">:</span><span class="w"> </span><span class="n">TLS_AES_128_GCM_SHA256</span><span class="p">,</span><span class="w"> </span><span class="n">bits</span><span class="p">:</span><span class="w"> </span><span class="mi">128</span><span class="p">)</span>
<span class="n">You</span><span class="w"> </span><span class="k">are</span><span class="w"> </span><span class="n">now</span><span class="w"> </span><span class="n">connected</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="k">database</span><span class="w"> </span><span class="ss">&quot;a&quot;</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="k">user</span><span class="w"> </span><span class="ss">&quot;dbadmin&quot;</span><span class="p">.</span>
<span class="n">a</span><span class="o">=&gt;</span><span class="w"> </span><span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">test01</span>
<span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="n">c_customer_sk</span><span class="w"> </span><span class="nb">integer</span><span class="p">,</span>
<span class="w"> </span><span class="n">c_customer_id</span><span class="w"> </span><span class="nb">char</span><span class="p">(</span><span class="mi">5</span><span class="p">),</span>
<span class="w"> </span><span class="n">c_first_name</span><span class="w"> </span><span class="nb">char</span><span class="p">(</span><span class="mi">6</span><span class="p">),</span>
<span class="w"> </span><span class="n">c_last_name</span><span class="w"> </span><span class="nb">char</span><span class="p">(</span><span class="mi">8</span><span class="p">)</span>
<span class="w"> </span><span class="p">)</span>
<span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="p">(</span><span class="n">orientation</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">column</span><span class="p">,</span><span class="n">compression</span><span class="o">=</span><span class="n">middle</span><span class="p">)</span>
<span class="w"> </span><span class="n">distribute</span><span class="w"> </span><span class="k">by</span><span class="w"> </span><span class="n">hash</span><span class="w"> </span><span class="p">(</span><span class="n">c_last_name</span><span class="p">);</span>
<span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span>
<span class="n">a</span><span class="o">=&gt;</span><span class="w"> </span><span class="k">INSERT</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">test01</span><span class="p">(</span><span class="n">c_customer_sk</span><span class="p">,</span><span class="w"> </span><span class="n">c_customer_id</span><span class="p">,</span><span class="w"> </span><span class="n">c_first_name</span><span class="p">)</span><span class="w"> </span><span class="k">VALUES</span><span class="w"> </span><span class="p">(</span><span class="mi">3769</span><span class="p">,</span><span class="w"> </span><span class="s1">'hello'</span><span class="p">,</span><span class="w"> </span><span class="s1">'Grace'</span><span class="p">);</span>
<span class="k">INSERT</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">1</span>
<span class="n">a</span><span class="o">=&gt;</span><span class="w"> </span><span class="k">INSERT</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">test01</span><span class="w"> </span><span class="k">VALUES</span><span class="w"> </span><span class="p">(</span><span class="mi">456</span><span class="p">,</span><span class="w"> </span><span class="s1">'good'</span><span class="p">);</span>
<span class="k">INSERT</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">1</span>
</pre></div></td></tr></table></div>
</div>
</p></li><li id="EN-US_TOPIC_0000001587507168__li156031819487"><span>Run the <strong id="EN-US_TOPIC_0000001587507168__b137012418155">\COPY</strong> command to export data from the UTF8 library in Unicode format to the <strong id="EN-US_TOPIC_0000001587507168__b1481710171613">test01.dat</strong> file.</span><p><div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001587507168__screen14919185561019"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="err">\</span><span class="k">copy</span><span class="w"> </span><span class="n">test01</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="s1">'/opt/test01.dat'</span><span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="p">(</span><span class="k">ENCODING</span><span class="w"> </span><span class="s1">'Unicode'</span><span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</p></li><li id="EN-US_TOPIC_0000001587507168__li189224428473"><span>Switch to database B and create a table with the same name <strong id="EN-US_TOPIC_0000001587507168__b761113941610">test01</strong>.</span><p><div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001587507168__screen243222819576"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">a</span><span class="o">=&gt;</span><span class="w"> </span><span class="err">\</span><span class="k">c</span><span class="w"> </span><span class="n">b</span>
<span class="n">Password</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="k">user</span><span class="w"> </span><span class="n">dbadmin</span><span class="p">:</span>
<span class="n">SSL</span><span class="w"> </span><span class="k">connection</span><span class="w"> </span><span class="p">(</span><span class="n">protocol</span><span class="p">:</span><span class="w"> </span><span class="n">TLSv1</span><span class="p">.</span><span class="mi">3</span><span class="p">,</span><span class="w"> </span><span class="n">cipher</span><span class="p">:</span><span class="w"> </span><span class="n">TLS_AES_128_GCM_SHA256</span><span class="p">,</span><span class="w"> </span><span class="n">bits</span><span class="p">:</span><span class="w"> </span><span class="mi">128</span><span class="p">)</span>
<span class="n">You</span><span class="w"> </span><span class="k">are</span><span class="w"> </span><span class="n">now</span><span class="w"> </span><span class="n">connected</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="k">database</span><span class="w"> </span><span class="ss">&quot;b&quot;</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="k">user</span><span class="w"> </span><span class="ss">&quot;dbadmin&quot;</span><span class="p">.</span>
<span class="n">b</span><span class="o">=&gt;</span><span class="w"> </span><span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">test01</span>
<span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="n">c_customer_sk</span><span class="w"> </span><span class="nb">integer</span><span class="p">,</span>
<span class="w"> </span><span class="n">c_customer_id</span><span class="w"> </span><span class="nb">char</span><span class="p">(</span><span class="mi">5</span><span class="p">),</span>
<span class="w"> </span><span class="n">c_first_name</span><span class="w"> </span><span class="nb">char</span><span class="p">(</span><span class="mi">6</span><span class="p">),</span>
<span class="w"> </span><span class="n">c_last_name</span><span class="w"> </span><span class="nb">char</span><span class="p">(</span><span class="mi">8</span><span class="p">)</span>
<span class="w"> </span><span class="p">)</span>
<span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="p">(</span><span class="n">orientation</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">column</span><span class="p">,</span><span class="n">compression</span><span class="o">=</span><span class="n">middle</span><span class="p">)</span>
<span class="w"> </span><span class="n">distribute</span><span class="w"> </span><span class="k">by</span><span class="w"> </span><span class="n">hash</span><span class="w"> </span><span class="p">(</span><span class="n">c_last_name</span><span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</p></li><li id="EN-US_TOPIC_0000001587507168__li1462365991"><span>Run the <strong id="EN-US_TOPIC_0000001587507168__b8698719141610">\COPY</strong> command to import the <strong id="EN-US_TOPIC_0000001587507168__b1312342310165">test01.dat</strong> file to database B.</span><p><div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001587507168__screen1059253316165"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="err">\</span><span class="k">copy</span><span class="w"> </span><span class="n">test01</span><span class="w"> </span><span class="k">from</span><span class="w"> </span><span class="s1">'/opt/test01.dat'</span><span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="p">(</span><span class="k">ENCODING</span><span class="w"> </span><span class="s1">'Unicode'</span><span class="w"> </span><span class="p">,</span><span class="n">COMPATIBLE_ILLEGAL_CHARS</span><span class="w"> </span><span class="s1">'true'</span><span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
<div class="note" id="EN-US_TOPIC_0000001587507168__note1978762612178"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="EN-US_TOPIC_0000001587507168__ul315283421920"><li id="EN-US_TOPIC_0000001587507168__li14153334171911">The error tolerance parameter <strong id="EN-US_TOPIC_0000001587507168__b19208103912164">COMPATIBLE_ILLEGAL_CHARS</strong> specifies that invalid characters are tolerated during data import. Invalid characters are converted and then imported to the database. No error message is displayed. The import is not interrupted.</li><li id="EN-US_TOPIC_0000001587507168__li15153143451916">The BINARY format is not supported. When data of such format is imported, error "cannot specify bulkload compatibility options in BINARY mode" will occur.</li><li id="EN-US_TOPIC_0000001587507168__li415353471918">The parameter is valid only for data importing using the <strong id="EN-US_TOPIC_0000001587507168__b175737073433332">COPY FROM</strong> option.</li></ul>
</div></div>
</p></li><li id="EN-US_TOPIC_0000001587507168__li1850128194916"><span>View data in the <strong id="EN-US_TOPIC_0000001587507168__b380110451820">test01</strong> table in database B.</span><p><div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001587507168__screen954813169553"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span>
<span class="normal">6</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">b</span><span class="o">=&gt;</span><span class="w"> </span><span class="k">select</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">from</span><span class="w"> </span><span class="n">test01</span><span class="p">;</span>
<span class="w"> </span><span class="n">c_customer_sk</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">c_customer_id</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">c_first_name</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">c_last_name</span>
<span class="c1">---------------+---------------+--------------+-------------</span>
<span class="w"> </span><span class="mi">3769</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">hello</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">Grace</span><span class="w"> </span><span class="o">|</span>
<span class="w"> </span><span class="mi">456</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">good</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="o">|</span>
<span class="p">(</span><span class="mi">2</span><span class="w"> </span><span class="k">rows</span><span class="p">)</span>
</pre></div></td></tr></table></div>
</div>
</p></li><li id="EN-US_TOPIC_0000001587507168__li444681515147"><span>After the preceding operations are performed, data is imported from database A (UTF8) to database B (GBK).</span></li></ol>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dws_03_0026.html">Data Import and Export</a></div>
</div>
</div>
View Git Blame Copy Permalink