amazon-science/SpIDER

amazon-science/SpIDER

Language: Python

License: Apache-2.0

Stars: 2

Forks: 0

Open issues: 0

Created: 2026-09-21T17:01:52Z

Pushed: 2026-10-08T19:04:51Z

Default branch: main

Fork: no

Archived: no

README:

SpIDER — Spatially Informed Dense Embedding Retrieval

[SpIDER: Spatially Informed Dense Embedding Retrieval for Software Issue Localization](https://arxiv.org/abs/2512.16956)

Shravan Chaudhari, Rahul Thomas Jacob, Jiajun Cao, Shihab Rashid, Mononito Goswami, Christian Bock

*EMNLP 2026 (Main Conference)*

> 🎉 SpIDER has been accepted to the EMNLP 2026 Main Conference. The > SpIDER-Bench graphs are > available on the Hugging Face Hub.

Reference implementation of the SpIDER issue-localization loop from our EMNLP 2026 paper (arXiv:2512.16956), plus the data-preparation chain and the summarizer that produce the reported tables.

SpIDER starts from a dense retrieval ranking over a repository's code graph and uses an LLM for exactly one kind of judgment: *does this graph neighbor itself need to change?* Everything else — which nodes become exploration centers, how the ranked window is rewritten — is deterministic.

Per round, for a window of the top-K ranked functions:

1. Centers = the top-C of the window. No LLM call. 2. Neighbor filtering — for each center, BFS its neighborhood in the code graph (per edge type, at that edge type's own depth), keep the neighbors that fall inside the top-N candidate pool, and ask the LLM to restrictively select the ones that must be modified. C calls. 3. Window rewrite — each selected neighbor is inserted directly below its center; an equal number of bottom-ranked nodes drop out to keep the window at size K.

The final prediction is the top-L of the accumulated window. With the released defaults (K=20 L=20 N=500 C=5 max_steps=1) that is 5 LLM calls per instance.

---

Contents

spider/ the pipeline
pipeline.py the SpIDER loop (steps 1-3 above)
data_loader.py rankings, graphs, BFS, both ground-truth views
prompt_format.py prompt rendering + response parsing
llm_client.py OpenAI / Azure / Azure-Responses / vLLM transports
result_builder.py per-instance result dict, incl. the no-LLM baseline
summary.py aggregation + result serialization
run_api.py runner: hosted LLM endpoint, thread pool
run_vllm.py runner: local models, one vLLM server per GPU
multigpu_utils.py vLLM server launch/teardown
data_prep/ graphs -> embeddings -> scores -> candidate pool
scripts/
download_bench.py fetch SpIDER-Bench from the Hub and rebuild the graphs
materialize_graphs.py parquet -> {instance_id}.pkl (the preprocessing step)
spider_bench_schema.py parquet schema + the graphtables conversion
build_hf_dataset.py producer side: pickle tree -> parquet (run once, by us)
audit_graph_schema.py derives the parquet schema from every graph in the tree
prepare_data.sh run the three data-prep stages for one dataset item
run_spider_api.sh launcher for the hosted-API runner
run_spider.sh launcher for the local-vLLM runner
reproduce_paper_runs.sh every configuration behind the reported tables
summarize.py result JSONs -> SUMMARY.txt / summary.csv

---

Install

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

Python 3.10+. The core runtime is small (openai, httpx, networkx, numpy); requirements.txt marks which extras belong to data preparation (datasets, bm25s, PyStemmer, unidiff), to node embedding (torch, transformers), and to the local-model runner (vllm).

---

Data

SpIDER needs two things per benchmark instance:

| what | where it comes from | |---|---| | repository graph — {instance_id}.pkl, an nx.MultiDiGraph whose nodes carry type / code / start_line / end_line and whose edges carry type ∈ {contains, imports, inherits, invokes} | SpIDER-Bench on the Hub, rebuilt from parquet (below) | | score pickle — the dense ranking over those nodes, plus ground truth | you build it (below), or reuse ours |

Both are Python pickles, which run code when loaded — see [Security](#security) before loading either from a source you don't control.

1. Graphs

SpIDER-Bench is published as parquet, so it loads with datasets and renders in the Hub dataset viewer. The pipeline wants nx.MultiDiGraph pickles, so downloading it is two steps — download, then rebuild. download_bench.py does both:

python scripts/download_bench.py --out data/SpIDER-Bench
# or just what you need:
python scripts/download_bench.py --out data/SpIDER-Bench --subset SWE-PolyBench/python

The rebuild is required. Nothing downstream reads parquet; every stage resolves {graphs_dir}/{instance_id}.pkl. If you fetch the parquet some other way, run the preprocessing step yourself:

python scripts/materialize_graphs.py --out data/SpIDER-Bench # from the Hub
python scripts/materialize_graphs.py --out data/SpIDER-Bench --parquet-dir # from a local copy
python scripts/materialize_graphs.py --out data/SpIDER-Bench \
--instance-ids astropy__astropy-12907 django__django-11099 # a few instances

It downloads only the shards holding what you asked for, skips instances whose pickle already exists, and --verify-against re-checks every rebuilt graph with networkx.utils.graphs_equal.

Layout after the rebuild — what --graphs_dir expects:

data/SpIDER-Bench/SWE-PolyBench/{python,java,javascript,typescript}/{instance_id}.pkl
data/SpIDER-Bench/SWE-bench_Verified/{instance_id}.pkl
data/SpIDER-Bench/Multi-SWE-bench/{java,javascript,typescript}/{instance_id}.pkl

Budget ~7 GB for the parquet and ~73 GB for the rebuilt pickles. download_bench.py --no-keep-parquet drops the former once the latter exist.

Parquet layout, if you want to read the graphs directly

Each graph is stored relationally — one row per node and one row per edge — because a single instance reaches hundreds of MB, too large for a parquet row or the viewer. Three configs: instances (a one-row-per-graph summary), nodes and edges; the latter two carry one split per benchmark+language:

from datasets import load_dataset
nodes = load_dataset("AmazonScience/SpIDER-Bench", "nodes", split="swe_polybench_python")
edges =...