NVIDIA/storage-lender

NVIDIA/storage-lender

Description: A service that lends NVMe I/O queues to unprivileged clients over a Unix domain socket.

Language: C++

License: Apache-2.0

Stars: 6

Forks: 1

Open issues: 0

Created: 2026-07-20T18:06:33Z

Pushed: 2026-08-17T22:33:40Z

Default branch: main

Fork: no

Archived: no

README:

storage-lender

A service that lends NVMe I/O queues to unprivileged clients over a Unix domain socket. Clients obtain DMA-mapped memory, open NVMe controllers, and create their own I/O queues — all without needing direct VFIO/IOMMU access themselves.

How it works

Storage Lender separates NVMe provisioning from I/O. The daemon admits clients, imports DMA memory, opens controllers, and creates queues through SPDK. It then returns queue IDs, a BAR0 resource path, and doorbell offsets. Clients submit commands, ring mapped doorbells, and poll completions directly; the daemon is not on that hot path.

Socket access therefore admits a client to a broad trusted boundary rather than an isolated per-device service. See [Storage Lender Architecture](docs/Architecture.md) for component ownership, resource lifecycles, failure semantics, and the control/data-path split.

Dependencies

| Dependency | Version | How located | | -------------------------------------------------- | ------------------------------------------- | ------------------------------------------- | | SPDK | ≥ 23.x | cmake/FindSPDK.cmake (pkg-config, static) | | Boost | ≥ 1.83 | find_package(Boost REQUIRED ...) | | Protobuf | ≥ 3.21 | find_package(Protobuf REQUIRED) | | spdlog | 1.17.0 | CPM (fetched at configure time) | | toml++ | 3.4.0 | CPM (fetched at configure time) | | GoogleTest | 1.17.0 | CPM (fetched at configure time) | | C++ compiler | Server: C++23; client API: C++17-compatible | GCC ≥ 13 or Clang ≥ 17 | | CMake | ≥ 3.26 | — |

SPDK must be built with DPDK (--with-dpdk) and installed so that pkg-config can find spdk_nvme and spdk_env_dpdk. The build links SPDK and DPDK statically.

Building

cmake -B build -DCMAKE_BUILD_TYPE=RelWithDebInfo
cmake --build build -j$(nproc)

The server binary is build/server/storage-lender-server.

Development checks

Formatting and shell linting use version-pinned pre-commit hooks. Install the orchestrator and repository hook once:

pipx install pre-commit==4.6.1
pre-commit install --install-hooks

The hook formats staged C++, CMake, shell, and Markdown files and runs ShellCheck on staged shell files. If a formatter changes a file, the commit stops so the changes can be reviewed and staged. Run every check across the repository with:

pre-commit run --all-files

Standalone formatter installations, such as Homebrew mdformat, can support editor integration, but the hook environments remain authoritative and include the pinned GFM plugin.

Packaging

Debian packages are built with debhelper 13:

debian/rules debian/control
dpkg-buildpackage -us -uc -b -d

Four binary packages are produced:

| Package | Contents | | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------ | | storage-lender-server | Server binary, /usr/bin/storage-lender-ctl, systemd unit, system group, and /etc/storage-lender/config.toml conffile | | libstorage-lender-client0 | Shared library (libstorage_lender_client.so.0) | | libstorage-lender-client-dev | Headers (/usr/include/storage_lender/), static archive, .so symlink, and CMake config | | storage-lender-examples | Runnable queue-provisioning example and standalone C++ source |

libstorage-lender-client-dev depends on libstorage-lender-client0 and libprotobuf-dev. storage-lender-examples depends on the matching server and client runtime.

Every binary package installs the Storage Lender license, the third-party component manifest, the open-source notices, and their license texts under /usr/share/doc//.

The systemd service (storage-lender-server.service) targets multi-user.target and restarts on failure with a 5-second delay, but the Debian package does not enable or start it automatically. Configure VFIO access and the hugepages required by your SPDK deployment, then start or enable the service explicitly with systemctl.

> Note: The build fetches spdlog and toml++ via CPM at configure time. For an air-gapped > package build, pre-populate build/_deps/spdlog-src and build/_deps/tomlplusplus-src before > invoking dpkg-buildpackage. Debian and release builds pass both cached roots to CMake > independently.

Open-source notice bundle

Create the distributable compliance notice archive from the same configured build used for release binaries:

cmake --build build --target NoticeBundle

The target needs the source tree corresponding to the linked SPDK build. If CMake cannot infer it from SPDK's pkg-config paths, configure with -DSPDK_NOTICE_SOURCE_DIR=/path/to/spdk.

The archive is written to build/compliance/storage-lender--notice-bundle.tar.gz. It contains the third-party component manifest, attribution notice, complete referenced license texts, upstream license files for code incorporated into the server, the Storage Lender license, build-specific SPDK/DPDK/source metadata, direct shared-library requirements, and SHA-256 checksums. See [compliance/README.md](compliance/README.md) for the release gate and scope.

Running

The server requires hugepages and VFIO access (typically run as root or with appropriate capabilities):

# Bind the NVMe device to vfio-pci first, e.g.:
echo "0000:03:00.0" > /sys/bus/pci/devices/0000:03:00.0/driver/unbind
echo "vfio-pci" > /sys/bus/pci/devices/0000:03:00.0/driver_override
echo "0000:03:00.0" > /sys/bus/pci/drivers/vfio-pci/bind

# When systemd is not starting the daemon, create the configured parent first:
sudo install -d -o root -g storage-lender -m 0750 /run/storage-lender

# Start the server with the repository configuration:
sudo ./build/server/storage-lender-server --config config/storage-lender.toml

The packaged server listens on /run/storage-lender/api.sock as root:storage-lender with mode 0660, and on /run/storage-lender/ctl.sock as...