2023-01-07 s3cmd SIGPIPE accessing non-AWS s3 storage buckets
Scouring the internet for good unix command line tools to access s3-style, but non-AWS storage buckets landed me on s3cmd, a Python tool. Apart from being very slow, I have found it generally very unreliable when accessing non-AWS providers - for one reason or another, I had settled on Wasabi for my cloud storage.
I found that s3cmd often restarted transfers with SIGPIPE:
ksh$ s3cmd put bigfile.xz s3://mybucket-ap-southeast-2/misc/
bigfile.xz -> s3://mybucket-ap-southeast-2/misc/bigfile.xz [1 of 1]
2351104 of 7064777700 0% in 0s 8.33 MB/s failed
WARNING: Upload failed: /misc/bigfile.xz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.00)
WARNING: Waiting 3 sec...
bigfile.xz -> s3://mybucket-ap-southeast-2/misc/bigfile.xz [1 of 1]
2351104 of 7064777700 0% in 0s 10.34 MB/s failed
WARNING: Upload failed: /misc/bigfile.xz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.01)
WARNING: Waiting 6 sec...
bigfile.xz -> s3://mybucket-ap-southeast-2/misc/bigfile.xz [1 of 1]
192512 of 7064777700 0% in 0s 349.33 kB/s failed
WARNING: Upload failed: /misc/bigfile.xz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.05)
WARNING: Waiting 9 sec...Looking at packet traces, nothing was obviously amiss on my side. I know that providers often return redirects for the correct regional endpoint - but looking at the output with -d debugging turned on, the correct endpoint was being selected.
So, after looking at the code for a while, I went looking for alternatives, and stumbled upon s5cmd. This tool is written in go - while not as portable, this is far more efficient, explicitly performs parallel copies, which allows it to easily saturate whatever network link you happen to possess.
ksh$ s5cmd --stat --endpoint-url s3.ap-southeast-2.wasabisys.com cp bigfile.xz s3://mybucket-ap-southeast-2/misc/
cp bigfile.xz s3://mybucket-ap-southeast-2/misc/bigfile.xz
Operation Total Error Success
cp 1 0 1
