siliconflow/rdma_exporter

siliconflow/rdma_exporter

Description: Prometheus exporter for collection of RDMA (RoCE) NIC statistics from Linux hosts

Language: Go

License: MIT

Stars: 0

Forks: 0

Open issues: 0

Created: 2026-10-28T08:08:25Z

Pushed: 2026-10-29T07:00:00Z

Default branch: main

Fork: no

Archived: no

README:

Prometheus RDMA Exporter

 

For details on GitHub Actions status badges, see the official documentation.

rdma_exporter collects RDMA (InfiniBand/RoCE) NIC statistics from Linux hosts and exposes them as Prometheus metrics. The exporter walks the kernel's sysfs tree directly and publishes metrics with `github.com/prometheus/client_golang`.

Overview


Features

• Publishes counters from /sys/class/infiniband///counters and /hw_counters as rdma__total metrics that match NVIDIA's *Understanding mlx5 Linux Counters and Status Parameters* guide (e.g. rdma_port_rcv_data_total, rdma_symbol_error_total, rdma_duplicate_request_total). The sysfs lifespan knob is exported as the gauge rdma_lifespan_milliseconds, not as a _total counter.
• Exposes port metadata (link layer, state, width, speed, netdev, PCI address, VF/PF relationship, etc.) through rdma_port_info.
• Tracks scrape failures with rdma_scrape_errors_total.
• Supports device exclusion (--exclude-devices) to prevent kernel log flooding on firmware-restricted devices (NVIDIA DGX, Umbriel, GB200 systems).
• Ships with an HTTP server that serves /metrics and /healthz and gracefully shuts down on SIGINT/SIGTERM.
• Supports an alternative sysfs root (--sysfs-root) for testing or chroot environments.
• Honors a configurable scrape timeout (--scrape-timeout) to protect long-running sysfs reads.
• Exports RoCEv2 PFC and mlx5 ethtool hardware counters (NIC buffer drops, PCIe stalls, PHY/FEC, IEEE 802.3x global pause, pause storm, vport RDMA) via --collector.ethtool (default on; disable with --no-collector.ethtool).
• Exports mlx5 congestion-control counters (cc_*) and link-wide optional traffic (rdma_optional_{rx,tx}_{bytes,packets}_total) that sysfs omits, via RDMA netlink (--collector.optional-counters, default on). The exporter never enables counters.
• Optionally exports live auto-type QP hardware counters (rdma_qp_*) via a separate RDMA netlink socket (--collector.qp-counters, default off). The dump can exhaust the 5s scrape timeout on dense hosts; the exporter never binds QPs or enables auto mode.

Requirements

• Go 1.27.0 or newer.
• Linux with read access to /sys/class/infiniband (for production use).

Build

go build -o rdma_exporter .

Alternatively, use the provided Makefile helpers:

make build # compiles ./rdma_exporter
make test # runs go test ./...
make lint # runs go vet ./...

Run

./rdma_exporter \
--listen-address=":9879" \
--metrics-path="/metrics" \
--health-path="/healthz"

To exclude specific devices that trigger firmware errors (e.g., on NVIDIA DGX/GB200 systems):

./rdma_exporter --exclude-devices=mlx5_0,mlx5_1

Optional mlx5 congestion-control counters (cc_*) are omitted from sysfs. They are disabled by default and rdma statistic set is not persistent: enablement lives in the in-kernel rdma_hw_stats bitmap (and, on mlx5, in flow-steering rules created at enable time). A reboot, mlx5_ib reload, or device unbind/rebind returns them to disabled. There is no sysfs, module parameter, or mlxconfig knob. Reading them needs Linux 5.16+ (STAT_GET_STATUS). The exporter never calls rdma statistic set.

rdma statistic set link DEV/PORT optional-counters A,B replaces the enabled set. Counters not listed are disabled, including other mlx5 optional counters such as rdma_rx_packets / rdma_tx_bytes. Enabling optional counters allocates mlx5 flow counters and steering rules and may affect datapath performance; measure before leaving them on fleet-wide.

Enable once (CAP_NET_ADMIN), then scrape. Confirm with rdma statistic mode before trusting /metrics:

rdma statistic mode supported link mlx5_0/1
sudo rdma statistic set link mlx5_0/1 optional-counters cc_rx_ce_pkts,cc_rx_cnp_pkts,cc_tx_cnp_pkts
rdma statistic mode link mlx5_0/1
./rdma_exporter

To re-apply after boot or hotplug, use a root oneshot (not the rdma_exporter user). The repository ships [deploy/scripts/rdma-enable-hardware-counters.sh](deploy/scripts/rdma-enable-hardware-counters.sh), which owns the complete optional-counter set as the three cc_* names by default; override RDMA_OPTIONAL_COUNTERS to keep other mlx5 optional names in the same set line. Skip ports that do not advertise cc_*, and do not hide set failures.

Install the script, systemd unit, and udev rule like other RDMA user services (rdma-core udev.md). Do not use udev RUN+= (it blocks the udev queue):

sudo install -Dm0755 deploy/scripts/rdma-enable-hardware-counters.sh \
/usr/local/sbin/rdma-enable-hardware-counters
sudo install -Dm0644 deploy/systemd/rdma-hardware-counters.service \
/etc/systemd/system/rdma-hardware-counters.service
sudo install -Dm0644 deploy/udev/90-rdma-hardware-counters.rules \
/etc/udev/rules.d/90-rdma-hardware-counters.rules
sudo systemctl daemon-reload
sudo systemctl enable rdma-hardware-counters.service

Optional: copy [deploy/systemd/rdma-hardware-counters.env.example](deploy/systemd/rdma-hardware-counters.env.example) to /etc/rdma-hardware-counters.env and uncomment RDMA_ENABLE_QP_COUNTERS=1 when scraping --collector.qp-counters. The script then extends the optional-counter set with rdma_{rx,tx}_{bytes,packets} and runs rdma statistic qp set link ... auto type on optional-counters on on each supported port. Install [deploy/systemd/rdma_exporter-qp-counters.conf.example](deploy/systemd/rdma_exporter-qp-counters.conf.example) as /etc/systemd/system/rdma_exporter.service.d/qp-counters.conf so the...