forked from docs/doc-exports
chenxiaoxiong
f9e2808b7c
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: chenxiaoxiong <chenxiaoxiong@huawei.com> Co-committed-by: chenxiaoxiong <chenxiaoxiong@huawei.com>
19 lines
7.2 KiB
HTML
19 lines
7.2 KiB
HTML
<a name="dataartsstudio_01_0591"></a><a name="dataartsstudio_01_0591"></a>
|
|
|
|
<h1 class="topictitle1">MD5 Verification</h1>
|
|
<div id="body32001227"><p id="dataartsstudio_01_0591__en-us_topic_0000001197579313_en-us_topic_0000001151779052_en-us_topic_0108275377_p144251203018">CDM extracts data from the migration source and writes the data to the migration destination. <a href="#dataartsstudio_01_0591__en-us_topic_0000001197579313_en-us_topic_0000001151779052_en-us_topic_0108275377_fig4667102414522">Figure 1</a> shows the migration mode when files are migrated to OBS.</p>
|
|
<div class="fignone" id="dataartsstudio_01_0591__en-us_topic_0000001197579313_en-us_topic_0000001151779052_en-us_topic_0108275377_fig4667102414522"><a name="dataartsstudio_01_0591__en-us_topic_0000001197579313_en-us_topic_0000001151779052_en-us_topic_0108275377_fig4667102414522"></a><a name="en-us_topic_0000001197579313_en-us_topic_0000001151779052_en-us_topic_0108275377_fig4667102414522"></a><span class="figcap"><b>Figure 1 </b>Migrating files to OBS</span><br><span><img class="vsd" id="dataartsstudio_01_0591__en-us_topic_0000001197579313_en-us_topic_0000001151779052_en-us_topic_0108275377_image4668132445220" src="en-us_image_0000002269198113.png"></span></div>
|
|
<p id="dataartsstudio_01_0591__en-us_topic_0000001197579313_en-us_topic_0000001151779052_en-us_topic_0108275377_p2732132114215">During the process, CDM uses MD5 to verify file consistency.</p>
|
|
<ul id="dataartsstudio_01_0591__en-us_topic_0000001197579313_en-us_topic_0000001151779052_en-us_topic_0108275377_ul122505541835"><li id="dataartsstudio_01_0591__en-us_topic_0000001197579313_en-us_topic_0000001151779052_en-us_topic_0108275377_li792640434"><strong id="dataartsstudio_01_0591__en-us_topic_0000001197579313_en-us_topic_0000001151779052_en-us_topic_0108275377_b0871171112713">Extract</strong><ul id="dataartsstudio_01_0591__en-us_topic_0000001197579313_en-us_topic_0000001151779052_en-us_topic_0108275377_ul84291369441"><li id="dataartsstudio_01_0591__en-us_topic_0000001197579313_en-us_topic_0000001151779052_en-us_topic_0108275377_li493461316442">The migration source can be OBS, HDFS, FTP, SFTP, or HTTP. It can check whether the files extracted by CDM are consistent with source files.</li><li id="dataartsstudio_01_0591__en-us_topic_0000001197579313_en-us_topic_0000001151779052_en-us_topic_0108275377_li429412114413">This function is controlled by the <strong id="dataartsstudio_01_0591__en-us_topic_0000001197579313_en-us_topic_0000001151779052_en-us_topic_0108275377_b10245193514120">MD5 File Extension</strong> parameter (available when <strong id="dataartsstudio_01_0591__en-us_topic_0000001197579313_en-us_topic_0000001151779052_en-us_topic_0108275377_b179261118425">File Format</strong> is set to <strong id="dataartsstudio_01_0591__en-us_topic_0000001197579313_en-us_topic_0000001151779052_en-us_topic_0108275377_b1488715181425">Binary</strong>) in <strong id="dataartsstudio_01_0591__en-us_topic_0000001197579313_en-us_topic_0000001151779052_en-us_topic_0108275377_b1236144115416">Source Job Configuration</strong>. Set this parameter to the file name extension of the MD5 file in the source file system.</li><li id="dataartsstudio_01_0591__en-us_topic_0000001197579313_en-us_topic_0000001151779052_en-us_topic_0108275377_li797482173310">If a source file <strong id="dataartsstudio_01_0591__en-us_topic_0000001197579313_en-us_topic_0000001151779052_en-us_topic_0108275377_b7111348132212">build.sh</strong> and a file for saving MD5 value <strong id="dataartsstudio_01_0591__en-us_topic_0000001197579313_en-us_topic_0000001151779052_en-us_topic_0108275377_b7878105413227">build.sh.md5</strong> are located in the same directory, and <strong id="dataartsstudio_01_0591__en-us_topic_0000001197579313_en-us_topic_0000001151779052_en-us_topic_0108275377_b19562133612263">MD5 File Extension</strong> is configured, only the file <strong id="dataartsstudio_01_0591__en-us_topic_0000001197579313_en-us_topic_0000001151779052_en-us_topic_0108275377_b985815712288">build.sh.md5</strong> is migrated to the destination. Files without the MD5 value or whose MD5 values do not match fail to be migrated, and the MD5 file is not migrated.</li><li id="dataartsstudio_01_0591__en-us_topic_0000001197579313_en-us_topic_0000001151779052_en-us_topic_0108275377_li1663153616407">If <strong id="dataartsstudio_01_0591__en-us_topic_0000001197579313_en-us_topic_0000001151779052_en-us_topic_0108275377_b1878593417226">MD5 File Extension</strong> is not configured, all files are migrated.</li></ul>
|
|
</li><li id="dataartsstudio_01_0591__en-us_topic_0000001197579313_en-us_topic_0000001151779052_en-us_topic_0108275377_li6467734184312"><strong id="dataartsstudio_01_0591__en-us_topic_0000001197579313_en-us_topic_0000001151779052_en-us_topic_0108275377_b15609616192910">Write</strong><ul id="dataartsstudio_01_0591__en-us_topic_0000001197579313_en-us_topic_0000001151779052_en-us_topic_0108275377_ul2056152613454"><li id="dataartsstudio_01_0591__en-us_topic_0000001197579313_en-us_topic_0000001151779052_en-us_topic_0108275377_li339831516454">Currently, this function can be used only when OBS serves as the migration destination. It can check whether the files written to OBS are consistent with those extracted from CDM.</li><li id="dataartsstudio_01_0591__en-us_topic_0000001197579313_en-us_topic_0000001151779052_en-us_topic_0108275377_li9177142084510">This function is controlled by the <span class="parmname" id="dataartsstudio_01_0591__en-us_topic_0000001197579313_en-us_topic_0000001151779052_en-us_topic_0108275377_parmname4143253173015"><b>Validate MD5 Value</b></span> parameter in <strong id="dataartsstudio_01_0591__en-us_topic_0000001197579313_en-us_topic_0000001151779052_en-us_topic_0108275377_b2716155943013">Destination Job Configuration</strong>. After the files are read and written to OBS, the MD5 value in the HTTP header is used to verify the files on OBS and the verification result is written to an OBS bucket (the bucket can be the one that does not store migration files). If the migration source does not have the MD5 file, the verification will not be performed.</li></ul>
|
|
</li></ul>
|
|
<div class="note" id="dataartsstudio_01_0591__en-us_topic_0000001197579313_en-us_topic_0000001151779052_en-us_topic_0108275377_note135931921105610"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="dataartsstudio_01_0591__en-us_topic_0000001197579313_en-us_topic_0000001151779052_en-us_topic_0108275377_ul19444123112549"><li id="dataartsstudio_01_0591__en-us_topic_0000001197579313_en-us_topic_0000001151779052_en-us_topic_0108275377_li559173215549">When files are migrated to a file system, only the extracted files are verified.</li><li id="dataartsstudio_01_0591__en-us_topic_0000001197579313_en-us_topic_0000001151779052_en-us_topic_0108275377_li6227124915589">When files are migrated to OBS, both the extracted files and files written to OBS are verified.</li><li id="dataartsstudio_01_0591__en-us_topic_0000001197579313_en-us_topic_0000001151779052_en-us_topic_0108275377_li124443316546">If MD5 verification is used, <a href="dataartsstudio_01_0590.html">KMS encryption</a> cannot be used.</li></ul>
|
|
</div></div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="dataartsstudio_01_0110.html">Key Operation Guide</a></div>
|
|
</div>
|
|
</div>
|
|
|