coreweave/s5cmd

coreweave/s5cmd

Description: Parallel S3 and local filesystem execution tool.

License: MIT

Stars: 5

Forks: 0

Open issues: 0

Created: 2025-11-04T17:18:09Z

Pushed: 2025-11-06T22:11:53Z

Default branch: master

Fork: yes

Parent repository: peak/s5cmd

Archived: no

README:  !Github Actions Status


Overview

s5cmd is a very fast S3 and local filesystem execution tool. It comes with support for a multitude of operations including tab completion and wildcard support for files, which can be very handy for your object storage workflow while working with large number of files.

There are already other utilities to work with S3 and similar object storage services, thus it is natural to wonder what s5cmd has to offer that others don't.

In short, *s5cmd offers a very fast speed.* Thanks to Joshua Robinson for his study and experimentation on s5cmd; to quote his medium post: > For uploads, s5cmd is 32x faster than s3cmd and 12x faster than aws-cli. >For downloads, s5cmd can saturate a 40Gbps link (~4.3 GB/s), whereas s3cmd >and aws-cli can only reach 85 MB/s and 375 MB/s respectively.

If you would like to know more about performance of s5cmd and the reasons for its fast speed, refer to [benchmarks](./README.md#Benchmarks) section

Features


s5cmd supports wide range of object management tasks both for cloud storage services and local filesystems.

• List buckets and objects
• Upload, download or delete objects
• Move, copy or rename objects
• Set Server Side Encryption using AWS Key Management Service (KMS)
• Set Access Control List (ACL) for objects/files on the upload, copy, move.
• Print object contents to stdout
• Select JSON records from objects using SQL expressions
• Create or remove buckets
• Summarize objects sizes, grouping by storage class
• Wildcard support for all operations
• Multiple arguments support for delete operation
• Command file support to run commands in batches at very high execution speeds
• Dry run support
• S3 Transfer Acceleration support
• Google Cloud Storage (and any other S3 API compatible service) support
• Structured logging for querying command outputs
• Shell auto-completion
• S3 ListObjects API backward compatibility

Installation

Official Releases

Binaries

The Releases page provides pre-built binaries for Linux, macOS and Windows.

Homebrew

For macOS, a homebrew tap is provided:

brew install peak/tap/s5cmd

Unofficial Releases (by Community)

 > Warning > These releases are maintained by the community. They might be out of date compared to the official releases.

MacPorts

You can also install s5cmd from MacPorts on macOS:

sudo port selfupdate sudo port install s5cmd

Conda

s5cmd is included in the [conda-forge]( https://conda-forge.org ) channel, and it can be downloaded through the Conda.

> Installing s5cmd from the conda-forge channel can be achieved by adding conda-forge to your channels with: > `` > conda config --add channels conda-forge > conda config --set channel_priority strict > > > Once the conda-forge channel has been enabled, s5cmd can be installed with conda: > > > conda install s5cmd > `` ps. Quoted from s5cmd feedstock. You can also find further instructions on its README.

FreeBSD

On FreeBSD you can install s5cmd as a package:

pkg install s5cmd

or via ports:

cd /usr/ports/net/s5cmd
make install clean

Build from source

You can build s5cmd from source if you have Go 1.19+ installed.

go install github.com/peak/s5cmd/v2@master

⚠️ Please note that building from master is not guaranteed to be stable since development happens on master branch.

Docker

Hub

$ docker pull peakcom/s5cmd $ docker run --rm -v ~/.aws:/root/.aws peakcom/s5cmd

ℹ️ /aws directory is the working directory of the image. Mounting your current working directory to it allows you to run s5cmd as if it was installed in your system;

docker run --rm -v $(pwd):/aws -v ~/.aws:/root/.aws peakcom/s5cmd

Build

$ git clone https://github.com/peak/s5cmd && cd s5cmd $ docker build -t s5cmd . $ docker run --rm -v ~/.aws:/root/.aws s5cmd

Usage

s5cmd supports multiple-level wildcards for all S3 operations. This is achieved by listing all S3 objects with the prefix up to the first wildcard, then filtering the results in-memory. For example, for the following command;

s5cmd cp 's3://bucket/logs/2020/03/*' .

first a ListObjects request is send, then the copy operation will be executed against each matching object, in parallel.

Specifying credentials

s5cmd uses official AWS SDK to access S3. SDK requires credentials to sign requests to AWS. Credentials can be provided in a variety of ways:

• Command line options --profile to use a named profile, --credentials-file flag to use the specified credentials file

# Use your company profile in AWS default credential file
s5cmd --profile my-work-profile ls s3://my-company-bucket/

# Use your company profile in your own credential file
s5cmd --credentials-file ~/.your-credentials-file --profile my-work-profile ls s3://my-company-bucket/

• Environment variables

# Export your AWS access key and secret pair
export AWS_ACCESS_KEY_ID=''
export AWS_SECRET_ACCESS_KEY=''
export AWS_PROFILE=''
export AWS_REGION=''

s5cmd ls s3://your-bucket/

• If s5cmd runs on an Amazon EC2 instance, EC2 IAM role
• If s5cmd runs on EKS, Kube IAM role
• Or, you can send requests anonymously with --no-sign-request option…