NVIDIA/storage-scale-test

NVIDIA/storage-scale-test

Description: Test high performance storage solutions at large scale with minimal dependencies

Language: Python

License: Apache-2.0

Stars: 0

Forks: 1

Open issues: 1

Created: 2026-08-07T20:31:35Z

Pushed: 2026-09-19T01:16:48Z

Default branch: main

Fork: no

Archived: no

README:

NVIDIA Storage Scale Test

Shell and Python tooling for running distributed storage benchmarks and turning their output into tables and plots.

Tests are supported for:

• Filesystems: data and metadata operations with elbencho
• S3-compatible object storage with Warp
• Storage networks with elbencho netbench (beta)

Warning: These Tests Destroy Data

These tests create, overwrite, and delete files and objects. Use only dedicated test paths and empty buckets. Root access is neither required nor recommended.

Resources

• [Getting Started](#getting-started) — quickstart
• [Benchmark recipes](BENCHMARK_RECIPES_FILESYSTEM.md) — filesystem env.sh

settings by use case

• [Roadmap](ROADMAP.md) — planned work
• Design & requirements: [docs/DESIGN.md](docs/DESIGN.md),

[docs/REQUIREMENTS.md](docs/REQUIREMENTS.md), [docs/ARCHITECTURE_DIAGRAMS.md](docs/ARCHITECTURE_DIAGRAMS.md)

• [Contributing](CONTRIBUTING.md) and [Security policy](SECURITY.md)
• Licensing: [LICENSE](LICENSE) (Apache-2.0) and [NOTICE](NOTICE) — third-party

benchmark tools carry their own licenses, including GPL-3.0 and AGPL-3.0

Global Prerequisites

• Linux on the orchestration host and on every client. Benchmark execution

relies on bash 4.3 or newer and a Linux userland. macOS can run static checks and unit tests that do not require benchmark binaries, but it cannot run the storage benchmarks. In particular, elbencho is not available for Apple silicon; binary-dependent tests are skipped when it is unavailable. Running the developer unit tests on macOS requires a newer Bash and GNU coreutils, which can be installed with brew install bash coreutils.

• Python 3.12 or newer is required on the host that runs the analysis and

reporting wrappers. The pinned current NumPy release establishes this minimum.

• Clients must be available through Slurm or passwordless SSH. For Slurm,

gather the account, reservation, and partition required to submit jobs.

• Use tuned clients with enough aggregate network bandwidth to saturate the

target. A starting count is 1.1 * target_Gbps / measured_per_client_Gbps.

Getting Started

The fastest path from a fresh checkout to a first result is a single-node filesystem sweep. The same pattern (configure env.sh → validate → sweep → report) applies to every test type.

1. Build a tarball and copy it to a client host. From your workstation:

./utils/build_tarball.sh

This produces storage-scale-test.tar.gz. See [Deployment Tarballs and Benchmark Binaries](#deployment-tarballs-and-benchmark-binaries) for what it includes and the binaries you must supply. Copy and unpack it on a host that can reach your client nodes (a Slurm login node, or any host with passwordless SSH to the clients).

2. Configure `env.sh`. Copy the template and edit it for your environment:

cp env.sh.template env.sh

At minimum, set how clients are reached and what to test:

# --- How to reach the client nodes (pick ONE) ---
# Slurm mode: set your account / reservation / partition (see env.sh.template).
# SSH mode: point at a host list file (SLURM_* vars are then ignored).
# export SSH_HOST_LIST=/absolute/path/to/host_list

# --- Filesystem test target (mounted on every client) ---
declare -A TEST_DIRS=(["/mnt/fs1/scaletest"]=1)

# --- A small, fast sweep to start with ---
export ELBENCHO_SCALE_IO_SIZES=("r4K" "1M")
export ELBENCHO_SCALE_THREAD_LIST=("1" "32" "64" "128")
export ELBENCHO_SCALE_READ_WRITE_DURATION=60

3. Validate the environment. Iterate until this runs cleanly:

./validate_env.sh

4. Run a single-node sweep.

./storage-tests/fs/nv-elbencho-sweep.sh --nodes 1

5. Generate a report from the results directory it prints:

./utils/extract-elbencho.sh "$RESULTS_DIR"/elbencho-/

From here, pick the thread counts that saturated a single node and move on to a multi-node sweep. For tuned, use-case-specific parameter sets (peak-finding, requirements confirmation, post-maintenance validation, single-large-file testing), see [BENCHMARK_RECIPES_FILESYSTEM.md](BENCHMARK_RECIPES_FILESYSTEM.md).

Deployment Tarballs and Benchmark Binaries

This repository does not ship benchmark binaries. utils/build_tarball.sh creates a user-local storage-scale-test.tar.gz with the available tools:

• elbencho: reuses utils/elbencho{,.aarch64} or downloads pinned upstream

static releases when missing or when --force-download is set. Download failure does not prevent tarball creation.

• Warp: includes existing utils/warp{,.aarch64} only. The suggested

utils/build/build_warp_from_source.sh defaults to NVIDIA/warp-minio, a fork of MinIO Warp, and accepts another GitHub/GitLab URL plus a tag, branch, or commit.

• s3test: builds utils/s3test{,.aarch64} from the in-tree C source with

local compilers or Docker/Buildx when missing or stale.

Review third-party licenses and provenance — see [NOTICE](NOTICE), which records that elbencho is GPL-3.0 and Warp is AGPL-3.0. Verify that the tarball contains working binaries for every target architecture; NVIDIA and this project do not publish or distribute the resulting tarball or its binaries.

Execution Modes

Slurm is the default execution substrate. Setting SSH_HOST_LIST switches the tool to passwordless SSH mode; the two modes are mutually exclusive.

Passwordless SSH

The host-list file accepts comma- or whitespace-separated hosts on multiple lines and ignores comment lines:

192.0.2.10,192.0.2.11
# comment
node1 node2 node3

Configure SSH mode in env.sh:

export SSH_HOST_LIST=/absolute/path/to/host_list
# export SSH_USER=ubuntu # otherwise the user or SSH config wins
# export SSH_HOMEDIR_SHARED=1 # any non-empty value

Setting SSH_HOST_LIST disables Slurm settings. Authentication must be non-interactive. Run validate_env.sh to verify connectivity.

Slurm options

SLURM_JOB_NAME_PREFIX prefixes job names. SLURM_EXTRA_ARGS is a Bash array appended after generated sbatch/srun options, so later duplicate options can override earlier ones...