NVIDIA/NVSentinel v1.16.0

Release v1.16.0

Repository: NVIDIA/NVSentinel

Tag: v1.16.0

Published: 2026-07-27T11:01:37Z

Prerelease: no

Release notes:

Release v1.16.0

This release adds detection of wedged mlx5 NIC drivers (TX timeout followed by a NAPI soft lockup) via the syslog health monitor, automatic nvidia-bug-report collection when a GPU reset fails, node-label actions on fault-quarantine rule matches, and custom recommended actions for Kubernetes Object Monitor policies. It also suppresses false-positive NVLink-down remediations on PCIe GPUs and hardens the NIC monitor against silent misconfiguration.

Major New Features

Detect mlx5 NIC Driver TX Timeout and NAPI Soft Lockup (#1508)

Adds a multi-line detector to the syslog health monitor that catches a wedged mlx5 NIC driver from kernel logs and quarantines the node, closing a gap where a NAPI soft lockup left a node degraded for days with no NVSentinel signal. In the originating incident (#1334), an mlx5 TX timeout was followed ~11 seconds later by BUG: soft lockup - CPU#N stuck with the kernel stack pinned inside the mlx5 NAPI poll loop; the driver's built-in recovery never kicked in and the node slowly failed (CSI socket stalls, pods stuck Pending). Because the generic soft-lockup message and the mlx5 stack frame are separate journal entries, no single-line regex could connect them — the new detector uses a small state machine that arms a 150-line window on the soft-lockup header and confirms on a non-speculative mlx5e_poll_ico_cq / mlx5e_napi_poll stack frame.

Bug-Report Collection on GPU Reset Failure (#1499)

When the GPU reset script observes a failed nvidia-smi --gpu-reset, it now collects an nvidia-bug-report and, when UPLOAD_URL_BASE is configured, uploads it (best-effort) to the file-server endpoint for later diagnosis. The upload URL is wired through the janitor GPU-reset job config and Helm values, defaulting automatically to the in-cluster file server when it is enabled.

Node-Label Actions on Fault-Quarantine Rules (#1506)

Fault-quarantine rulesets can now apply optional Kubernetes node labels when a rule matches. Multiple matching rules for the same label key are resolved by priority (configuration order as tie-breaker), applied labels are recorded in the quarantineHealthEventAppliedLabels annotation and held while any tracked health event remains unhealthy, and only the recorded labels are removed once all tracked events recover (including cleanup on manual uncordon, manual untaint, and stale-state recovery). Metrics are exported for applied and removed labels.

Custom Recommended Actions for KOM Policies (#1521, #1530)

Kubernetes Object Monitor policies can now specify a customRecommendedAction alongside recommendedAction: CUSTOM (ADR-036), routing matched events to arbitrary CR types via fault-remediation's remediationActions config without any additional code — the operator just names the action key they defined in Helm values. The field is wired through to the published HealthEvent proto and validated at config load, so a CUSTOM action with an empty string is caught at startup rather than rejected later at the platform-connector gRPC boundary. The chart's policy TOML template now conditionally renders customRecommendedAction, fixing a startup crash when a policy set CUSTOM without the field appearing in the rendered config.

Bug Fixes & Reliability

• Suppress false-positive `DCGM_FR_NVLINK_DOWN` on PCIe GPUs (#1481): PCIe GPUs (A100 PCIe, A40, L40S, etc.) have no NVLink hardware, but DCGM's GpuNvlinkWatch reported all links as down with isFatal=true and recommendedAction=RESTART_VM, triggering spurious remediations. The metadata collector now records an nvlink_link_count (counting links where NVML GetNvLinkState returns SUCCESS, i.e. hardware present), and the GPU health monitor suppresses DCGM_FR_NVLINK_DOWN only for GPUs with zero NVLink hardware links — conclusive across SKUs since PCIe parts return NOT_SUPPORTED while SXM/NVL return SUCCESS even for broken links.
• NIC monitor hardening: config fail-fast, management-NIC scoping, reboot reconciliation (#1528, #1529): Three fixes so the NIC monitor is never silently wrong. (1) A typo in config.toml (e.g. thresholdType = "velocty") previously made the monitor quietly fall back to defaults — disabling all counter monitoring while still reporting healthy; a malformed config now fails pod startup with a clear error, while a genuinely missing file still falls back to defaults. (2) Counter (degradation) checks now respect management-NIC classification like the state checks do, so a link blip on a management RoCE NIC can no longer emit a fatal REPLACE_VM and cordon the node. (3) Reboots and hardware changes (e.g. NIC renames) can no longer orphan node conditions that nobody can clear.
• Embedded DCGM mode runs privileged (#1509): The embedded DCGM mode added in v1.13.0 could not find NVIDIA devices without elevated privileges; the gpu-health-monitor now runs privileged in embedded mode (DCGM 3.x and 4.x) so local DCGM connectivity works. Also updates the UAT fault-injection scripts to dynamically discover the monitored DCGM endpoint and work across all supported DCGM modes, and documents the embedded-mode privilege requirement in ADR-044.

Documentation

• Documentation fixes (#1526): Spelling and grammar corrections, consistent heading hierarchy across docs, fixes for <> rendering in Fern (replaced with {}), and consolidation of the NIC monitor design docs into the root design folder.

Acknowledgments

This release includes contributions from:

• @AnouarMohamed
• @deesharma24
• @jtschelling
• @KaivalyaMDabhadkar
• @lalitadithya
• @shaq918
• @ymygogogo

Thank you to everyone who contributed code, testing, documentation, design reviews, and feedback! Special thanks to first-time contributors @AnouarMohamed and @ymygogogo....