NVIDIA/nvidia-mnccd

NVIDIA/nvidia-mnccd

Description: A usermode daemon responsible for key/iv management across nodes and multi-node discovery.

Language: Rust

License: Apache-2.0

Stars: 5

Forks: 1

Open issues: 1

Created: 2026-08-20T13:28:04Z

Pushed: 2026-10-13T04:03:18Z

Default branch: main

Fork: no

Archived: no

README:

nvidia-mnccd

Nvidia Multi-Node Confidential Compute Daemon (nvidia-mnccd) is an open source privileged usermode daemon that is supposed to run either on baremental or inside a confidential VM (CVM). It is responsible for key/IV management across nodes, multi-node discovery, multi-node attestation and ensuring that all the nodes are assigned to the same tenant.

Building nvidia-mnccd

The daemon is a Rust crate built with Cargo. There is a single build that always includes gRPC TLS support: TLS-PSK (default at runtime, via OpenSSL and tonic-tls), mutual TLS (rustls with operator-supplied PEM certificates), and plain HTTP/2 when disabled. HKDF key derivation for NVLE flows uses the hkdf crate. Whether gRPC connections use TLS, and which mode, is chosen at runtime via [tls].mode in config and the --no-tls flag (see [Running nvidia-mnccd](#running-nvidia-mnccd)).

Prerequisites

• A recent stable Rust toolchain (rustup recommended).
• OpenSSL development libraries on the build host (for example libssl-dev on Debian/Ubuntu, openssl-devel on RHEL). Required for the default PSK transport (openssl-sys / tonic-tls).
• `nvml.h` from CUDA 13.5 or newer. build.rs generates NVML bindings from this header (only the header is needed at compile time; libnvidia-ml.so is loaded at runtime). Older toolkits lack the NVLE/remap-table APIs and the build fails.

`nvml.h` location

The header is resolved in this order:

1. `NVML_HEADER_DIR` — directory that contains nvml.h (not the file itself). Use this when the CUDA toolkit is not in a default path, or when /usr/local/cuda does not point at 13.5+. 2. Otherwise the first readable file among:

• /usr/include/nvml.h
• /usr/local/include/nvml.h
• /usr/local/cuda/include/nvml.h (typical CUDA SDK install; /usr/local/cuda is usually a symlink to the active toolkit, e.g. /usr/local/cuda-13.5/include/nvml.h)

A versioned toolkit that is not linked as /usr/local/cuda is not searched automatically:

# Example: CUDA 13.5 installed at /usr/local/cuda-13.5
NVML_HEADER_DIR=/usr/local/cuda-13.5/include cargo build

Debug builds

cargo build

Release builds

cargo build --release

The compiled binary is target/debug/nvidia-mnccd or target/release/nvidia-mnccd, depending on the profile.

Running nvidia-mnccd

Configuration directory

At startup the process loads mnccd_config.toml from a directory chosen in this order:

1. `MNCCD_DATA_DIR` (runtime override). 2. `CARGO_MANIFEST_DIR` (set automatically when you use cargo run). 3. Compile-time default /etc/nvidia-mnccd (packagers can override at build time with MNCCD_DEFAULT_DATA_DIR= cargo build).

Example layout:

$MNCCD_DATA_DIR/
mnccd_config.toml
tls/ # only required when [tls].mode = "mtls"
ca-root.crt # cluster CA certificate (trust anchor)
server.key / server.crt
client.key / client.crt

TLS modes

Set [tls].mode in mnccd_config.toml (all nodes must use the same mode):

| Mode | Config value | Description | | --- | --- | --- | | TLS-PSK (default) | psk | OpenSSL TLS 1.2 DHE-PSK (forward secrecy). No certificate files under tls/. Same psk_identity and psk_key on every node. | | Mutual TLS | mtls | rustls with PEM certificates. Requires operator-supplied files under tls/ on every node (see below). | | Plaintext | none | Unencrypted gRPC. Usually selected with --no-tls on the CLI (overrides config). |

If the [tls] section is omitted, MNCCD defaults to `psk`, but `psk_identity` and `psk_key` are required and must be set explicitly (startup fails if either is missing or empty, unless --no-tls is set).

TLS-PSK (`psk`, default)

PSK encrypts the gRPC mesh using a shared cluster secret. Each node is both TLS client and server; peers authenticate with the same pre-shared identity and key. Ciphersuites are DHE-PSK-AES256-GCM-SHA384 and DHE-PSK-AES128-GCM-SHA256: ephemeral Diffie-Hellman is mixed with the PSK so session keys have forward secrecy—captured traffic cannot be decrypted later from the cluster PSK alone.

| Field | Meaning | | --- | --- | | [tls].mode | "psk" (default if [tls] is omitted). | | [tls].psk_identity | PSK identity sent in the TLS handshake (public label, like a username). Required when mode is psk. Must match on every node. | | [tls].psk_key | Pre-shared key bytes (the secret). Required when mode is psk. Must match on every node. |

Limits (OpenSSL): identity ≤ ~127 bytes; key ≤ 256 bytes.

Example:

[tls]
mode = "psk"
psk_identity = "mnccd-cluster"
psk_key = "replace-with-a-long-random-secret"

Provision the same psk_identity and psk_key on every node (ConfigMap, Secret, or baked into mnccd_config.toml). The sample placeholder key in the shipped mnccd_config.toml (replace-with-a-long-random-secret) is rejected at startup unless --no-tls is set—replace it before running with TLS-PSK.

Mutual TLS (`mtls`)

TLS material uses the same directory as mnccd_config.toml (see resolution order above). With [tls].mode = "mtls", install the following PEM files under tls/ on every node before starting the daemon. MNCCD does not generate certificates or keys.

| File | Role | | --- | --- | | tls/ca-root.crt | Cluster CA certificate (trust anchor). Must be the same on every node. | | tls/server.crt / tls/server.key | This node's gRPC server identity. | | tls/client.crt / tls/client.key | This node's gRPC client identity when dialing peers. |

Server certificates must present a DNS SAN (or name) of mnccd — peer clients verify against that domain. Private keys should be mode 0600.

tls/ca-root.key is not read by MNCCD at runtime; keep the CA private key offline and use it only when issuing node certificates.

The TOML file defines the gRPC listen port and the cluster membership:

| Field | Meaning | | --- | --- | | [server].server_port | TCP port the gRPC server binds on (each node uses the same port number). | | [cluster].node_ips | List of all node IP addresses in the...