Closing the Binary Blind Spot: Domain-Knowledge Injection for LLM Malware Triage in npm Packages

Closing the Binary Blind Spot: Domain-Knowledge Injection for LLM Malware Triage in npm Packages | Databricks Blog Skip to main content

Introduction Supply chain attacks on package registries have grown in volume and sophistication [1]. Public research regularly observes malware being deployed to the npm registry by a range of threat actors, from spam campaigns [2] to the steady proliferation of DPRK malware [3, 4, 5], and to the high-impact TeamPCP in 2025 and 2026 [6, 7], with the DPRK and TeamPCP actors having a track record of successfully compromising widely used open source projects for maximized downstream reach [8]. To study this problem directly, we have been collecting npm malware samples since November 2025, and observed that, among the 11,413 package-versions we scanned and corroborated through OSV npm malware advisories [9], most malware on the npm registry hides its payload in the package source or fetches it from an external server, accounting for 96.2% of the corroborated dataset. The remaining 3.8%, 432 package-versions, bundle a binary. Though small in volume, they can carry disproportionate risk during triage. A stripped, opaque binary that carries no source for tools to read can be nested within a facade of clean source and metadata and potentially skew an LLM towards a benign verdict. We evaluate how a panel of LLMs reason over crafted binary payloads at varying obfuscation and ambiguity, identify the failure modes specific to this vector, and close them with a purpose-built, cost-effective encoding of domain knowledge that improves detection quality for both budget and frontier models alike. Background To support the wider research community, we built a scanner that monitors the npm registry in real time and runs a static stage before LLM triage, with YARA [10] on the tarball, Semgrep [11] and the npm registry API on the JavaScript and metadata, and a binary-analysis module over the compiled artifact. It assembles the output into a structured prompt, and the LLM returns a verdict with reasoning, requesting additional files over up to three rounds. Our analysis found packages with bundled binaries to be both the rarest and the hardest for the pipeline to triage. An adversary distributing malicious JavaScript can assume the target already has the runtime to execute the payload, whereas a binary must be compiled for a range of target machines. Across OSV-confirmed npm malware, binary-bearing packages fail to reach a decisive malicious verdict more than twice as often as source-only ones (14.4% vs 6.2%). Prior work instruction-tunes open-source models on expert-authored data, baking the knowledge into the weights [12]. That route cannot reach the closed, hosted models, and it requires retraining as the threat landscape shifts. We take the opposite approach and supply the expertise at inference, as pre-computed static evidence placed in the prompt before the model reasons. Experimental Design Objective We evaluate whether LLMs can detect malicious logic concealed within precompiled binaries in otherwise clean packages. A secondary question is whether the detection failures are systematic rather than random. Third, whether encoding domain knowledge of the binary problem can improve detection accuracy. Adversarial Payloads We built five payload types (Table 1) of native Node.js addons (.node files) covering common supply chain attack patterns. Imports range from defensible (fopen, connect, gethostname) to rarely legitimate (execve with dup2), and each package wraps the binary in a professional package.json, README, JavaScript bindings, and binding.gyp. The sysmon stealer spans an obfuscation ladder from a readable baseline to a hardened build with symbols stripped and strings XOR-encoded, while the other four types appear at a single obfuscation level. We chose single-byte XOR deliberately, since unlike a stronger cipher such as RC4 it leaves no structural markers in disassembly for a model to catch. The set mirrors collected malware. Across the OSV-confirmed npm packages shipping a binary, formats are PE, ELF (all stripped), and Mach-O, with behaviors spanning credential and environment theft, persistence, exfiltration and C2, host reconnaissance, and droppers. Table 1. Adversarial payload types and their import profiles. Package Type Key Imports Ambiguity sysmon-native Credential stealer fopen, connect, send, gethostname HIGH buildchain-native Dropper/backdoor execve, connect, chmod, socket, fwrite LOW netpulse-native Reverse shell execve, dup2, connect, socket LOW hashworker-native Cryptominer stub pthread_create, connect, send, recv MEDIUM taskd-native Persistence system, fopen, fwrite, chmod MEDIUM

Models Tested We evaluated nine models across seven providers (Table 2): seven budget models a team would realistically run across its codebase, four of them open-weight, and two higher-capability baselines as behavioral baseline references at higher price points. We also ran Claude Opus 4.6, the most expensive model, only as a cost reference. Table 2. Models evaluated and their roles. Model Provider Role Type Claude Haiku 4.5 Anthropic Budget Proprietary Google Gemini 3.5 Flash Google Budget Proprietary Gemma 3 12B Google Budget Open-weight GPT-5.4-nano OpenAI Budget Proprietary Qwen3-Next 80B Alibaba Budget Open-weight GLM 5.3 Flash Zhipu Budget Open-weight Llama 3.1 8B Meta Budget Open-weight Claude Sonnet 4.6 Anthropic Baseline behavior Proprietary Kimi K3 Moonshot Baseline behavior Open-weight Claude Opus 4.6 Anthropic Cost baseline Proprietary

To isolate the profiles, each payload was scanned twice per model, once without profiles and once with them. Prompt, context, and binary analysis were identical across both runs. Inference settings were held constant across every model and both conditions: temperature 0, the same system and analysis prompts throughout, and an 8192-token output cap. Capability Profile Architecture A capability profile is a declarative playbook that encodes one malicious technique, such as credential harvesting or persistence, as a set of weighted conditions over the scanner's static evidence. The profiles cover source and binary techniques across npm and PyPI. Figure 1 shows how a profile enters a scan: The static stage extracts evidence from the package. Each profile scores that evidence against its suspicious, malicious, and critical thresholds, a scale separate from the model's own verdict. When a threshold is reached,...