amazon-science/SynthAVE

amazon-science/SynthAVE

Description: SynthAVE: multilingual benchmark for attribute-value verification with LLM-arena validation (EMNLP 2026 Industry Track)

License: NOASSERTION

Stars: 0

Forks: 0

Open issues: 0

Created: 2026-09-22T08:31:56Z

Pushed: 2026-09-29T17:02:07Z

Default branch: main

Fork: no

Archived: yes

README:

SynthAVE

> A synthetic multilingual benchmark for attribute-value verification in e-commerce, validated with a diverse LLM arena.

SynthAVE is a benchmark of 15,906 synthetic e-commerce products across five languages, released with per-row labels for the task of attribute-value verification. Each product is paired with an (attribute, value) and a label in {Correct, Incorrect, Unknown}. Labels are produced by a 21-judge LLM arena (7 model families × 3 prompts). The core 12,726-row benchmark across DE / ES / FR / IT follows the paper's full protocol — arena majority plus expert adjudication on the disagreement stratum, with a stratified audit on the agreement stratum. An additional 3,180 English rows are released with arena-majority labels only (see [note](#english-note)*).

> Release note. This dataset and its accompanying code are released solely for academic and scientific reproducibility purposes, in support of the methods and findings described in the associated publication. Pull requests are not accepted, in order to maintain the artifact exactly as it was used in the paper. If you want to build on this work as an ongoing project, please contact the authors and start a new release.

Table of Contents

• [Dataset Description](#dataset-description)
• [Languages](#languages)
• [Dataset Structure](#dataset-structure)
• [Dataset Statistics](#dataset-statistics)
• [Label Provenance](#label-provenance)
• [How to Use](#how-to-use)
• [Dataset Creation](#dataset-creation)
• [Considerations for Using the Data](#considerations-for-using-the-data)
• [Citation](#citation)
• [License](#license)
• [Contact](#contact)

Dataset Description

Task. Attribute-value verification: given a product (title, bullet points, description) and a candidate (attribute, value) pair, decide whether the value is Correct, Incorrect, or Unknown (not determinable from the product text).

Origin. Products are generated with the attribute-aware controlled-generation pipeline of Negri et al. 2025. Attribute values are deliberately manipulated so that all three label types occur in usable proportions. Brand names, model identifiers, and other potentially identifying signals are anonymized during generation.

Validation. Every (product, attribute, value) triple is judged by 21 LLM configurations (7 model families × 3 prompt versions, 267,246 judgments in total). Final labels are:

• Expert adjudication when the arena majority disagrees with the synthetic-pipeline label, and on additional adjudication rounds for DE and IT.
• Arena majority when the arena and synthetic pipeline agree, validated on a 400-sample stratified human audit (Cohen's κ = 0.92 vs. expert; 3.0% overturn rate).

Estimated label accuracy on the released set is 97.9%.

Languages

Spanish (es), French (fr), Italian (it), German (de), English (en*).

Dataset Structure

Files are organised per language:

SynthAVE/
├── german/dataset.json # DE — 3,059 records
├── italian/dataset.json # IT — 3,007 records
├── french/dataset.json # FR — 3,501 records
├── spanish/dataset.json # ES — 3,159 records
├── english/dataset.json # EN — 3,180 records*
└── release_summary.json # aggregate counts and label provenance

Each dataset.json is a JSON array of records. Example (Italian):

{
"id": "it_0",
"language": "it",
"category": "NOTEBOOK_COMPUTER",
"title": "NexusPro Elite 15 R75800H 16GB/750GB W11H",
"bullet_points": [],
"description": "NexusPro Elite 15 R75800H 16GB/750GB W11H",
"attribute": {
"name": "hard_disk.size",
"value": "750.0 GB"
},
"label": "Correct",
"original_label": "Correct",
"label_source": "arena_majority"
}

Fields

| Field | Type | Description | |---|---|---| | id | string | Anonymised product identifier, _ (e.g. it_0) | | language | string | Locale code: de, en, es, fr, or it | | category | string | Product category (one of 229) | | title | string | Product title | | bullet_points | list[string] | Bullet-point features | | description | string | Long-form product description | | attribute.name | string | Attribute being verified (one of 792) | | attribute.value | string | Candidate value for that attribute | | label | string | Final released label: Correct, Incorrect, Unknown, or null on 3 undecisive triage rows | | original_label | string | Label before this cleaning pass (audit trail) | | label_source | string | Provenance tag; see [Label Provenance](#label-provenance) | | triage_note | string, optional | Free-text auditor note when present |

Dataset Statistics

| Locale | Products | Correct | Incorrect | Unknown | |---|---:|---:|---:|---:| | ES | 3,159 | 1,483 | 501 | 1,175 | | FR | 3,501 | 1,670 | 550 | 1,281 | | IT | 3,007 | 1,452 | 442 | 1,113 | | DE | 3,059 | 1,461 | 512 | 1,086 | | EN* | 3,180 | 1,320 | 351 | 1,509 | | Total | 15,906 | 7,386 (46.4%) | 2,356 (14.8%) | 6,164 (38.8%) |

• Product categories: 229 (4-locale set) / 274 (English)
• Unique attributes: 792 (4-locale set) / 727 (English)
• Distinct product–attribute combinations: 2,607 (4-locale set)
• Minimum products per category: 30† (4-locale set)
• Judge configurations behind each label: 21

• English is released as a fifth-locale extension of the paper's four-locale benchmark. Two important differences apply to English rows only: (1) labels come from the arena majority vote alone — no expert adjudication or human triage audit was performed on English, whereas DE / ES / FR / IT went through the paper's full protocol; (2) the arena's Claude family judge is Claude 4 Sonnet, since the paper's Claude 3.5 Sonnet inference profile was retired before English could be run. Estimated label quality for English is therefore lower than the 97.9% figure that applies to the paper's four locales, and should be treated accordingly.

† Target minimum on the 4-locale set. A small number of items were removed at release time because their category could not be reliably determined, leaving 14 of the 229 categories slightly below the target with 26–29 products each. The largest category...