amazon-science/tabpfn-automl2026

amazon-science/tabpfn-automl2026

Language: Python

License: MIT

Stars: 0

Forks: 0

Open issues: 0

Created: 2026-07-22T19:42:58Z

Pushed: 2026-07-31T23:28:05Z

Default branch: main

Fork: no

Archived: no

README:

Zero-Shot Bayesian Optimization with TabPFN

Code, data, and experiment definitions accompanying the paper "Zero-Shot Bayesian Optimization with TabPFN" (AutoML 2026, Methods Track).

> This code is being released solely for academic and scientific reproducibility > purposes, in support of the methods and findings described in the associated > publication. Pull requests are not being accepted in order to maintain the code > exactly as it was used in the paper.

Verify the paper's claims (= 3.10.

Repository structure

| Path | Contents | |------|----------| | run_all.py, tests/, expected_output.txt | Master verification script + pytest suite | | src/ | The TabPFN-direct BO implementation (tabpfn_direct_searcher.py, 1,271 lines) plus baseline searchers | | analysis/ | Statistical analysis scripts behind every paper claim (29 scripts) | | CLAIMS.md | Claim-to-file traceability: every primary numerical claim in the paper mapped to the data file and script that produces it | | DATA-INDEX.md | Generated index of every result file: location, size, producer, consumer, and which nine back the headline claims | | dre_weights/ | Per-space DRE model weights (trained parameters, not experiment output) — see dre_weights/README.md | | EXPERIMENTS.md | Index mapping every data file to its worker and SageMaker job batch (including each batch's run matrix) | | experiments/workers/ | SageMaker worker scripts for every experiment in the paper | | experiments/collectors/ | Result-aggregation scripts (S3 job outputs to JSONL) | | experiments/configs/baselines.md | Exact configuration of every baseline method | | experiments/gitbo/ | GIT-BO replication check: worker, launcher, collector, table generator | | fetch_third_party.py | Fetches third-party code (GIT-BO, ZeroShotOpt, HPO-B, TabDPT, TabICL) from the upstream GitHub repos at the pinned commits used for the paper — this repo does not redistribute third-party source | | results/raw/ | Raw SageMaker output, exactly as collected (one row per run or evaluation) — the single canonical location, indexed in DATA-INDEX.md | | results/rolled-up/ | Aggregates computed from the raw output: statistics, p-values, confidence intervals, merged grids | | REPRODUCE.md | End-to-end reproduction guide (environment, data, jobs, analysis) |

Re-running experiments

The experiments/ layer defines every experiment exactly as it ran. The original runs executed as SageMaker Training Jobs; internal infrastructure identifiers (account IDs, buckets, IAM roles, AWS profiles) are redacted as REDACTED or read from environment variables (HPO_BENCH_BUCKET, HPO_SM_ROLE_ARN, ZSO_CHECKPOINT_URI) — substitute your own values to re-launch, or run the workers directly on any machine (each worker is a self-contained script whose CLI defines the experiment). Benchmarks download from public sources: HPO-B (v3-test) and YAHPO-Gym. Before re-running experiments, fetch the third-party code the workers build on (python fetch_third_party.py — clones GIT-BO, ZeroShotOpt, HPO-B, TabDPT, and TabICL from their upstream GitHub repos at the pinned commits). See REPRODUCE.md and experiments/README.md.

• TabPFN: public tabpfn==7.1.0 package driving the TabPFN v2 weights

(pinned in code via ModelVersion.V2; weights auto-download, ~50 MB).

• Baseline configurations: experiments/configs/baselines.md.

Citation

@inproceedings{tabpfn-zeroshot-bo-2026,
title = {Zero-Shot Bayesian Optimization with TabPFN: Competitive with State-of-the-Art without Per-Task Training},
author = {Theodore Rogers and Srividya Ponnada},
booktitle = {Proceedings of the Fifth International Conference on Automated Machine Learning},
series = {Proceedings of Machine Learning Research},
publisher = {PMLR},
year = {2026}
}

License

Released under the MIT License (see LICENSE). The MIT license covers the authors' code; the experiment results in results/ were produced by evaluating third-party benchmarks whose data carries its own licenses — HPO-B data (CC BY 4.0; Pineda-Arango et al., 2021) and YAHPO-Gym surrogates (data under Apache 2.0; Pfisterer et al., 2022). See DATA_NOTICE.md for full data provenance and attribution.

This repository contains only code written for the paper; it does not redistribute third-party source. Experiment workers that build on third-party projects — GIT-BO, ZeroShotOpt, HPO-B (Pineda-Arango et al., 2021), TabDPT, and TabICL — obtain that code via fetch_third_party.py, which clones each project at the exact commit used for the paper's runs. Each fetched project remains under its own upstream license (see the table in fetch_third_party.py).