yexinru/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
Files
c4291b1dd5fc1097b89f17c7eca946120cea0b3c
doc-exports/docs/obs/tool-obsutil/obs_11_0044.html
weihongmin1 c4291b1dd5 OBS Util 0309 Verion 
Reviewed-by: Sabelnikov, Dmitriy <dmitriy.sabelnikov@t-systems.com>
Co-authored-by: weihongmin1 <weihongmin1@huawei.com>
Co-committed-by: weihongmin1 <weihongmin1@huawei.com>
2026-03-27 12:15:19 +00:00

36 KiB
Raw Blame History

Synchronously Copying Incremental Objects

Function

You can use this command to synchronize all objects in a specified path of the source bucket to a specified path in the target bucket to ensure data consistency. Incremental synchronization has the following meanings:

  • "Incremental" means that the source objects are compared with their counterparts in the target bucket and only the source objects with content changes are copied.
  • "Synchronization" means that after the command is executed, all source objects in the specified path of the source bucket have their counterparts in the target bucket.
  • Do not change the source objects in the OBS bucket when synchronously copying objects. Otherwise, the synchronization may fail or data may be inconsistent.
  • To copy objects, you must have the read permission on the objects to be copied and the write permission on the destination bucket.
  • If the client-side cross-region replication function is not enabled, ensure that the source bucket and destination bucket are in the same region.
  • Each object can be synchronously copied only when it does not exist in the destination bucket, its size is different from the namesake one in the destination bucket, or it has the latest modification time.
  • If the source bucket is a parallel file system (supporting POSIX), the destination bucket cannot be an object storage bucket.

Command Line Structure

  • Windows
    obsutil sync obs://srcbucket[/key] obs://dstbucket[/dest] [-dryRun] [-crr] [-vlength] [-vmd5] [-j=1] [-p=1] [-threshold=52428800] [-acl=xxx] [-sc=xxx] [-meta=aaa:bbb#ccc:ddd] [-ps=auto] [-include=*.xxx] [-exclude=*.xxx] [-timeRange=time1-time2] [-mf] [-o=xxx] [-cpd=xxx] [-config=xxx]
  • macOS or Linux
    ./obsutil sync obs://srcbucket[/key] obs://dstbucket[/dest] [-dryRun] [-crr] [-vlength] [-vmd5] [-j=1] [-p=1] [-threshold=52428800] [-acl=xxx] [-sc=xxx] [-meta=aaa:bbb#ccc:ddd] [-ps=auto] [-include=*.xxx] [-exclude=*.xxx] [-timeRange=time1-time2] [-mf] [-o=xxx] [-cpd=xxx] [-config=xxx]

The source path and destination path cannot be the same or nested when synchronously copying objects.

Examples

  • In Windows, run obsutil sync obs://bucket-test/temp/ obs://bucket-test2/temp/ to synchronously copy objects.
    obsutil sync obs://bucket-test/temp/ obs://bucket-test2/temp 

Start at 2024-09-25 04:48:10.1147483 +0000 UTC

Parallel:      5                   Jobs:          5
Threshold:     50.00MB             PartSize:      auto
CheckpointDir: C:\Users\Administrator\.obsutil_checkpoint

Task id: 104786c8-27c2-48fc-bc6a-5886596fb0e
OutputDir: C:\Users\Administrator\.obsutil_output

[=============================================================] 100.00% 10/s 0s
Succeed count:   5         Failed count:    0
Metrics [max cost:298 ms, min cost:192 ms, average cost:238.00 ms, average tps:9.71, transferred size: 7.20MB]
Task id: 0476929d-9d23-4dc5-b2f8-0a0493f027c5

Parameter Description

Parameter

Optional or Mandatory

Description

srcbucket

Mandatory

The source bucket name

dstbucket

Mandatory

The destination bucket name

dest

Optional

The name prefix of destination objects

NOTE:

If the value of this parameter does not end with a slash (/), the obsutil tool automatically adds a slash (/) at the end of the configured value as the name prefix of destination objects.

key

Optional

The name prefix of source objects

The rules are as follows:

  • If this parameter is left blank, objects whose name prefix is the value of dest in the destination bucket are the same as all objects in the source bucket.
  • If this parameter is configured, objects whose name prefix is the value of dest in the destination bucket are the same as objects whose name prefix is this configured value in the source bucket.
NOTE:
  • If the value of this parameter does not end with a slash (/), the obsutil tool automatically adds a slash (/) at the end of the configured value as the name prefix of source objects.
  • For details about how to use this parameter, see Synchronous Copy.

dryRun

Optional (additional parameter)

Conducts a dry run.

crr

Optional (additional parameter)

Enables the client-side cross-region replication function. In this mode, data is directly copied to the destination bucket from the source bucket through data stream. The buckets can be any two OBS buckets.

NOTE:
  • If this parameter is configured, ensure that the configuration of client-side cross-region replication is updated in the configuration file. For details, see Updating a Configuration File.
  • The configurations of the source bucket and destination bucket are respectively akCrr/skCrr/tokenCrr/endpointCrr and ak/sk/token/endpoint in the configuration file.
NOTICE:

When cross-region replication is enabled, the upload/download bandwidth, CPU, and memory resources of the host where commands are executed will be occupied, which may deteriorate the host performance.

vlength

Optional (additional parameter)

Verifies whether the object size in the destination bucket is the same as that in the source bucket after the copy task completes.

NOTE:

This parameter must be used together with crr.

vmd5

Optional (additional parameter)

Verifies whether the MD5 value of the destination bucket is the same as that of the source bucket after the copy task completes.

NOTE:
  • This parameter must be used together with crr.
  • Objects in the source bucket must contain metadata x-obs-meta-md5chksum, or MD5 verification will be skipped.

    After the MD5 verification is successful, this parameter value is used for metadata x-obs-meta-md5chksum of the destination object, for later MD5 verification during download or copy.

CAUTION:

If your object needs encryption, do not use this parameter.

p

Optional (additional parameter)

The maximum number of concurrent multipart copy tasks when copying an object. The default value is the value of defaultParallels in the configuration file.

CAUTION:

For an inter-bucket replication task that does not include the crr parameter, the maximum allowable value for this parameter is 10,000.

threshold

Optional (additional parameter)

The threshold for enabling multipart copy, in bytes. The default value is the value of defaultBigfileThreshold in the configuration file.

NOTE:
  • If the size of the object to be copied is smaller than the threshold, copy the object directly. If not, a multipart copy is required.
  • If you copy an object directly, no part record is generated, and resumable transmission is not supported.
  • This parameter value can contain a unit, for example, 1MB (indicating 1,048,576 bytes).

acl

Optional (additional parameter)

The access control policies for destination objects that can be specified when copying objects. The value can be:

  • private
  • public-read
  • public-read-write
  • bucket-owner-full-control
NOTE:

The preceding four values indicate private read and write, public read, public read and write, and bucket owner full control.

sc

Optional (additional parameter)

The storage classes of the destination objects that can be specified when copying objects. The value can be:

  • standard: Standard storage class. It features low access latency and high throughput, and is applicable to storing frequently accessed data (multiple accesses per month) or data that is smaller than 1 MB.
  • warm: Warm storage class. It is ideal for storing infrequently accessed (less than 12 times a year) data, but when needed, the access has to be fast.
  • cold: Cold storage class. It provides secure, durable, and inexpensive storage for rarely-accessed (once a year) data.

meta

Optional (additional parameter)

The metadata of destination objects that can be specified when copying objects. This parameter should be configured in the following format: key1:value1#key2:value2#key3:value3.

NOTE:

The format example above indicates that the destination objects contain three groups of custom metadata: key1:value1, key2:value2, and key3:value3.

ps

Optional (additional parameter)

The size of each part in a multipart copy task, in bytes. The value ranges from 100KB to 5GB. The default value is the value of defaultPartSize in the configuration file.

NOTE:
  • This parameter value can contain a unit, for example, 1MB (indicating 1,048,576 bytes).
  • The parameter can be set to auto. In this case, obsutil automatically sets the part size for each multipart task based on the source object size.

cpd

Optional (additional parameter)

The folder where the part records reside. The default value is .obsutil_checkpoint, the subfolder in the home directory of the user who executes obsutil commands.

NOTE:

A part record is generated during a multipart copy and saved to the copy subfolder. After the copy succeeds, its part record is deleted automatically. If the copy fails or is suspended, the system attempts to resume the task according to its part record when you perform the copy the next time.

j

Optional for copying objects in batches (additional parameter)

The maximum number of concurrent tasks for copying objects synchronously. The default value is the value of defaultJobs in the configuration file.

CAUTION:

For an inter-bucket replication task that does not include the crr parameter, the maximum allowable value for this parameter is 10,000.

NOTE:

The tool ensures that this value is at least 1.

exclude

Optional for copying objects in batches (additional parameter)

The matching patterns of source objects that are excluded, for example, *.txt

NOTE:
  • The asterisk (*) represents any group of characters, and the question mark (?) represents any single character. For instance, abc*.txt indicates any file whose name starts with abc and ends with .txt.
  • You can use \* to represent * and \? to represent ?.
  • If the name of the object to be copied matches the value of this parameter, the object is skipped.
NOTICE:
  • You are advised to use quotation marks for the matching pattern to prevent special characters from being escaped by the OS and leading to unexpected results. Use single quotation marks for Linux or macOS and double quotation marks for Windows.
  • The matching pattern applies to the object's absolute path, which includes the object name prefix and the object name starting from the root directory. For example, if an object's path in the bucket is obs://bucket/src1/src2/test.txt, its absolute path is src1/src2/test.txt.
  • This matching pattern applies only to objects whose names do not end with a slash (/).
  • Multiple exclude parameters can be specified, for example, -exclude=*.xxx -exclude=*.xxx.

include

Optional for copying objects in batches (additional parameter)

The matching patterns of source objects that are included, for example: *.jpg

NOTE:
  • The asterisk (*) represents any group of characters, and the question mark (?) represents any single character.
  • You can use \* to represent * and \? to represent ?.
  • The exclude rule is applied first. If the name of the object to be copied does not match the exclude rule, the system then checks whether the object name matches this parameter. If it does, the object is copied. If it does not, the object is skipped.
NOTICE:
  • You are advised to use quotation marks for the matching pattern to prevent special characters from being escaped by the OS and leading to unexpected results. Use single quotation marks for Linux or macOS and double quotation marks for Windows.
  • The matching pattern applies to the object's absolute path, which includes the object name prefix and the object name starting from the root directory. For example, if an object's path in the bucket is obs://bucket/src1/src2/test.txt, its absolute path is src1/src2/test.txt.
  • This matching pattern applies only to objects whose names do not end with a slash (/).
  • Multiple include parameters can be specified, for example, -include=*.xxx -include=*.xxx.

timeRange

Optional (additional parameter)

The time range matching pattern when synchronously copying objects. Only objects whose latest modification time is within the configured time range are copied.

This pattern has a lower priority than the object matching patterns (exclude/include). That is, the time range matching pattern is executed after the configured object matching patterns.

NOTE:
  • The matching time range is represented in time1-time2, where time1 must be earlier than or the same as time2. The time format is yyyyMMddHHmmss.
  • Automatic formatting is supported. For example, yyyyMMdd is equivalent to yyyyMMdd000000, and yyyyMM is equivalent to yyyyMM01000000.
  • If this parameter is set to *-time2, all files whose last modification time is earlier than time2 are matched. If it is set to time1-*, all files whose last modification time is later than time1 are matched.
NOTICE:
  • Time in the matching pattern is the UTC time.
  • This matching pattern applies only to objects whose names do not end with a slash (/).

mf

Optional (additional parameter)

Indicates that the name matching pattern (include or exclude) and the time matching pattern (timeRange) also take effect on objects whose names end with a slash (/).

o

Optional (additional parameter)

The folder that stores the result files. After the command is executed, result files (possibly success, failure, and warning files) will be created in the specified folder. The default value is .obsutil_output, a subfolder in the user's home directory where obsutil commands are executed.

NOTE:
  • A result file should be named as follows: sync_{succeed | failed | warning}_report_time_TaskId.txt.

    By default, the maximum size of a single result file is 30 MB and the maximum number of result files that can be retained is 1,024. You can set the maximum size and number by configuring recordMaxLogSize and recordBackups in the configuration file.

  • If there are multiple folders and files and you need to confirm the details of a failed task, refer to the failure result file cp_failed_report_time_TaskId.txt in the result folder and the log files in the log path.

config

Optional (additional parameter)

The user-defined configuration file for executing the current command. For details about parameters that can be configured, see Configuration Parameters.

Response

Refer to Response for uploading an object.

Parent topic: Object Commands