weihongmin1
c4291b1dd5
Reviewed-by: Sabelnikov, Dmitriy <dmitriy.sabelnikov@t-systems.com> Co-authored-by: weihongmin1 <weihongmin1@huawei.com> Co-committed-by: weihongmin1 <weihongmin1@huawei.com>
4.2 KiB
Using obsutil for Resumable Data Transfer
obsutil supports resumable data transfer (upload, download, and copy) for large files by using the multipart algorithms for upload, download, and copy. You can set the threshold size for starting a multipart upload, download, or copy task based on your actual requirements to resume the upload, download, or copy task if the task fails or is interrupted. You can specify the threshold size for starting a multipart task in either of the following ways:
Method 1
Set defaultBigfileThreshold in the configuration file. For details, see Configuration Parameters.
Method 2
Set threshold, a command-level parameter, when you run commands for object uploads, object downloads, object copy, synchronous uploads of incremental objects, synchronous downloads of incremental objects, and synchronous copy of incremental objects.
Example: obsutil cp d:\temp\test.txt obs://bucket-test/key -threshold=52428800
In this command:
- obsutil cp d:\temp\test.txt obs://bucket-test/key uploads file test.txt in the temp directory under D drive to bucket bucket-test and renames the file key.
- -threshold=52428800 starts a multipart upload when the threshold 50 MB is reached.
obsutil cp d:\temp\test.txt obs://bucket-test/key -threshold=52428800 Parallel: 3 Jobs: 3 Threshold: 50.00MB PartSize: auto VerifyLength: false VerifyMd5: false CheckpointDir: xxxx [====================================================] 100.00% 1.68 MB/s 5s Upload successfully, 8.46MB, d:\temp\test.txt --> obs://bucket-test/key
- Priority: Command level parameter threshold has higher priority than the defaultBigfileThreshold in the configuration file.
- The threshold size of a multipart task applies to single files or objects. When the size of a file or object is greater than the threshold value, the multipart algorithm is applied to the file or object.
- The multipart algorithm and resumable data transfer are tightly coupled. Using the multipart algorithm automatically enables resumable data transfer for the task.