Version 2
Detail of the version
1. Destination folder path construction — The original code used += instead of = when prepending /, causing the folder name to double (e.g. "myfolder" became "myfolder/myfolder"). Fixed to destination_folder = "/" + destination_folder.
2. Trailing slash on S3 destination — Without a trailing slash, rclone interprets :s3:bucket/path as a file, not a directory prefix. This caused the "is a file not a directory" critical error. Added automatic trailing slash to the destination folder.
3. Critical log level not captured — The JSON log parser only matched "error" level messages, but rclone sends fatal failures as "critical". This meant the real error was silently ignored (logged as "JSON log ignored by parser"). Fixed to match both "error" and "critical" levels.
4. Separated stdout and stderr streams — The original code merged stderr into stdout with stderr=subprocess.STDOUT. Since rclone sends JSON logs to stderr and progress to stdout, this could cause parsing issues. Each stream now has its own reader thread.
Parameters
s3_endpoint
Url of the lab S3 server (should end with 's3-server/v1'
stringbucket
Name of the destination bucket
stringaccess_key_id
Access key id for the S3 bucket
stringsecret_access_key
Secret access key for the S3 bucket
stringsource_folder
The source folder to sync with the destination lab. Must be absolute.
stringdestination_folder
Destination folder in the bucket. If empty, write in bucket root folder.
stringtransfers
Number of parallel file to transfer
int16chunk_size
Size of the chunk to send
string128M
Comments (0)
Write a comment