forked from docs/doc-exports
weihongmin1
a41a4e0331
Reviewed-by: Sabelnikov, Dmitriy <dmitriy.sabelnikov@t-systems.com> Co-authored-by: weihongmin1 <weihongmin1@huawei.com> Co-committed-by: weihongmin1 <weihongmin1@huawei.com>
52 lines
8.3 KiB
HTML
52 lines
8.3 KiB
HTML
<a name="obs_11_0052"></a><a name="obs_11_0052"></a>
|
|
|
|
<h1 class="topictitle1">Fine-Tuning obsutil Performance</h1>
|
|
<div id="body1553669911403"><p id="obs_11_0052__p085575619459">By default, obsutil uploads, downloads, and copies files or objects whose size is greater than 50 MB in multiple parts. <a href="#obs_11_0052__table8301721164611">Table 1</a> details related parameters in the <strong id="obs_11_0052__b2093922612114">.obsutilconfig</strong> file.</p>
|
|
|
|
<div class="tablenoborder"><a name="obs_11_0052__table8301721164611"></a><a name="table8301721164611"></a><table cellpadding="4" cellspacing="0" summary="" id="obs_11_0052__table8301721164611" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Multipart-related parameters</caption><thead align="left"><tr id="obs_11_0052__row1930332134612"><th align="left" class="cellrowborder" valign="top" width="26.51%" id="mcps1.3.2.2.3.1.1"><p id="obs_11_0052__p203035211465">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="73.49%" id="mcps1.3.2.2.3.1.2"><p id="obs_11_0052__p3303721134611">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="obs_11_0052__row16303112119462"><td class="cellrowborder" valign="top" width="26.51%" headers="mcps1.3.2.2.3.1.1 "><p id="obs_11_0052__p1050292773816">defaultBigfileThreshold</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="73.49%" headers="mcps1.3.2.2.3.1.2 "><p id="obs_11_0052__p14525135010317">Indicates the threshold for triggering multipart tasks, in bytes. If the size of a file to be uploaded, downloaded, or copied is greater than the threshold, the file is uploaded, downloaded, or copied in multiple parts. The default value is <strong id="obs_11_0052__b3819144474816">50MB</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="obs_11_0052__row19303122184613"><td class="cellrowborder" valign="top" width="26.51%" headers="mcps1.3.2.2.3.1.1 "><p id="obs_11_0052__p077914845118">defaultPartSize</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="73.49%" headers="mcps1.3.2.2.3.1.2 "><p id="obs_11_0052__p45251650113113">Size of each part, in bytes. The default value is <strong id="obs_11_0052__b196723543534">auto</strong>.</p>
|
|
<div class="note" id="obs_11_0052__note12525105093118"><span class="notetitle"> NOTE: </span><div class="notebody"><ul id="obs_11_0052__ul13525175063117"><li id="obs_11_0052__li5526175017316">For a multipart upload and copy, the value ranges from <strong id="obs_11_0052__b754712375164">100KB</strong> to <strong id="obs_11_0052__b6548113721612">5GB</strong>.</li><li id="obs_11_0052__li5526950193114">For multipart download, the value is unrestricted.</li></ul>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
<tr id="obs_11_0052__row630320214468"><td class="cellrowborder" valign="top" width="26.51%" headers="mcps1.3.2.2.3.1.1 "><p id="obs_11_0052__p4866163316380">defaultParallels</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="73.49%" headers="mcps1.3.2.2.3.1.2 "><p id="obs_11_0052__p35261450123120">Maximum number of concurrent tasks in the multipart mode. The default value is 5.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
<p id="obs_11_0052__p29025817521">Generally, multipart tasks not only speed up transmission but also allow you to resume failed tasks. By default, the part size of a multipart task can be automatically adjusted by the obsutil in the <strong id="obs_11_0052__b122571058105414">auto</strong> mode. In practice, however, to further improve the upload and download performance, you can adjust the part size according to the file size and the network conditions, to obtain the maximum transmission efficiency and ensure the successful completion of a transmission task.</p>
|
|
<p id="obs_11_0052__p151523105511">Adjust the number of concurrent tasks in the multipart mode according to the following formula:</p>
|
|
<p id="obs_11_0052__p1476315299439"><strong id="obs_11_0052__b202655312186">defaultParallels = Min(Number of CPUs x 2, Object size/defaultPartSize x 1.5)</strong></p>
|
|
<p id="obs_11_0052__p132201415513"></p>
|
|
<p id="obs_11_0052__p5450192314325">In the upload, download, and copy commands, parameters <strong id="obs_11_0052__b735714121314">-p</strong> and <strong id="obs_11_0052__b1657311573114">-ps</strong> are used to modify the number of concurrent tasks in the multipart mode and part size respectively, and then deliver the multipart task based on the parameter values configured in the command. The default values in the configuration file are used if you do not set them in a command.</p>
|
|
<p id="obs_11_0052__p72841972565">Adjust the number of concurrent tasks in the multipart mode according to the following formula:</p>
|
|
<p id="obs_11_0052__p1010243633111"><strong id="obs_11_0052__b1647413491491">p = Min(Number of CPUs x 2, Object size/ps x 1.5)</strong></p>
|
|
<p id="obs_11_0052__p15924110112019"></p>
|
|
<p id="obs_11_0052__p359042816235">For batch upload and download tasks, adjust the maximum number of concurrent tasks in a multipart upload, indicated by the parameter <strong id="obs_11_0052__b84675518349">defaultJobs (-j)</strong>, for better performance.</p>
|
|
<p id="obs_11_0052__p1915115712619">If you have a large number of <strong id="obs_11_0052__b1151316176427">small files </strong>(each is usually several MB) to be uploaded or downloaded, set <strong id="obs_11_0052__b9513171704217">defaultJobs (-j)</strong> to a larger value for better performance. In this case, adjusting <strong id="obs_11_0052__b11104151114411">defaultParallels (-p)</strong> and <strong id="obs_11_0052__b126371224414">defaultPartSize (-ps)</strong> may be ineffective.</p>
|
|
<p id="obs_11_0052__p13225161645315">If you want to upload or download <strong id="obs_11_0052__b2811321114511">large files</strong>, set <strong id="obs_11_0052__b1581162112459">defaultParallels (-p)</strong> and <strong id="obs_11_0052__b13669181164618">defaultPartSize (-ps)</strong> to a larger value for better performance. However, if there are too many concurrent tasks (calculated from <strong id="obs_11_0052__b15949642184613">defaultJobs</strong> x <strong id="obs_11_0052__b1294944294613">defaultParallels</strong>), the upload and download performance may deteriorate because of resource switchover and preemption between threads, and some tasks may fail due to network fluctuations.</p>
|
|
<div class="note" id="obs_11_0052__note1342773675613"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="obs_11_0052__ul1792843115614"><li id="obs_11_0052__li992184316563">Resources of a running host are limited. Therefore, if the number of concurrent tasks in the multipart mode is set too large, the performance of obsutil upload, download, or copy may deteriorate due to resource switchover and preemption between threads. In this case, you need to adjust the values of <strong id="obs_11_0052__b94801169320">defaultParallels</strong> (<strong id="obs_11_0052__b18452815232">-p</strong>) and <strong id="obs_11_0052__b74014198310">defaultPartSize</strong> (<strong id="obs_11_0052__b1532362212312">-ps</strong>) based on the actual file size and network status. To perform a pressure test, lower the two values at first, and then gradually increase them to determine the optimal values.</li><li id="obs_11_0052__li670114156574">If the values of <strong id="obs_11_0052__b4218285211">defaultParallels</strong> (<strong id="obs_11_0052__b10911949523">-p</strong>) and <strong id="obs_11_0052__b1546108125214">defaultPartSize</strong> (<strong id="obs_11_0052__b653219110523">-ps</strong>) are too large, an EOF error may occur due to network instability. In this case, set the two parameters to smaller values.</li><li id="obs_11_0052__li919991191918">If a batch operation is performed, the destination object size can be set to the average size of the objects to be operated.</li><li id="obs_11_0052__li62331513125417">If a batch task fails due to timeout, EOF, and other common network issues, retry the task through an incremental operation (specifically, configuring parameter <strong id="obs_11_0052__b126331348184120">-u</strong> in the <strong id="obs_11_0052__b54411561419">cp</strong> command). You can also restore the failed task (by configuring parameter <strong id="obs_11_0052__b1835714194314">-recover</strong> in the <strong id="obs_11_0052__b18153850114315">cp</strong> command) based on the generated task ID.</li></ul>
|
|
</div></div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="obs_11_0033.html">Best Practices</a></div>
|
|
</div>
|
|
</div>
|
|
|