Sync folder with another lab

Necessary bricks
Contributor(s)
Publication date

Aug 4, 2025

Confidentiality
Public
Reactions
0
0
Share

Version 2

Detail of the version

1. Destination folder path construction — The original code used += instead of = when prepending /, causing the folder name to double (e.g. "myfolder" became "myfolder/myfolder"). Fixed to destination_folder = "/" + destination_folder.


2. Trailing slash on S3 destination — Without a trailing slash, rclone interprets :s3:bucket/path as a file, not a directory prefix. This caused the "is a file not a directory" critical error. Added automatic trailing slash to the destination folder.


3. Critical log level not captured — The JSON log parser only matched "error" level messages, but rclone sends fatal failures as "critical". This meant the real error was silently ignored (logged as "JSON log ignored by parser"). Fixed to match both "error" and "critical" levels.


4. Separated stdout and stderr streams — The original code merged stderr into stdout with stderr=subprocess.STDOUT. Since rclone sends JSON logs to stderr and progress to stdout, this could cause parsing issues. Each stream now has its own reader thread.

Parameters

s3_endpoint

Url of the lab S3 server (should end with 's3-server/v1'

Type : string

bucket

Name of the destination bucket

Type : string

access_key_id

Access key id for the S3 bucket

Type : string

secret_access_key

Secret access key for the S3 bucket

Type : string

source_folder

The source folder to sync with the destination lab. Must be absolute.

Type : string

destination_folder

Optional

Destination folder in the bucket. If empty, write in bucket root folder.

Type : string

transfers

Optional

Number of parallel file to transfer

Type : intDefault value : 16

chunk_size

Optional

Size of the chunk to send

Type : stringDefault value : 128M

Environment file

Code

Comments (0)

Write a comment
Shine Logo
Ready-to-use smart agents

Have you developed an agent?

Share it to accelerate projects for the entire community.