amazon-science/data-turnstile

amazon-science/data-turnstile

Language: Python

License: NOASSERTION

Stars: 0

Forks: 0

Open issues: 0

Created: 2026-07-29T17:59:14Z

Pushed: 2026-08-24T08:28:07Z

Default branch: main

Fork: no

Archived: yes

README:

Data Turnstile

A scalable open framework for generating synthetic function-calling training data. Given API definitions, Turnstile generates multi-turn interactions where an AI assistant reasons about user requests, selects appropriate APIs, and handles real-world complexities — authentication, errors, parallel calls, and multi-step workflows.

Paper: Data Turnstile: A Scalable Open Framework for Function-Calling Data Generation Dataset: amazon/Turnstile-Synthetic-Domains (100,262 interactions, 1,025 APIs)

Features

• Config-driven: YAML controls everything — APIs, templates, personas, sampling params.
• Extensible: Add new domains by providing API definitions + interaction templates.
• Quality-aware: Per-role validation catches hallucinations, schema violations, and structural errors during generation. Failed roles are retried with error context.
• Scalable: Multi-server vLLM support for parallel generation across GPUs.

Quick Start

# Start a vLLM server serving your model (e.g., Qwen2.5-32B-Instruct)
python3 -m vllm.entrypoints.openai.api_server --model {PATH_TO_LOCAL_MODEL} --tensor-parallel-size {TP} --max-model-len 32768 --port {PORT} --served-model-name Qwen2.5-32B-Instruct

# Then generate data:
python run_generation.py configs/template_based_synthetic_domains.yaml

Output is written to ./output// as JSON.

Note: This repository ships with a limited set of 42 demo APIs. The full set of 1,025 APIs used in the paper is available as part of the dataset release on HuggingFace.

Structure

┌───────────────┐ ┌──────────────────┐ ┌──────────────┐
│ YAML Config │────▶│ DataGenerator │────▶│ Output .json │
└───────────────┘ └────────┬─────────┘ └──────────────┘
│
┌──────────────┼──────────────┐
▼ ▼ ▼
┌────────────┐ ┌──────────┐ ┌───────────┐
│ Personas │ │ Templates│ │API Library│
└────────────┘ └────┬─────┘ └───────────┘
▼
┌─────────────┐
│ Builder │ (generates one role at a time)
└──────┬──────┘
▼
┌─────────────┐
│ Validator │ (schema + invariant checks)
└─────────────┘

1. Config specifies API definitions, templates, persona schema, and generation parameters. 2. PersonaGenerator uses the LLM to create user profiles for biasing data generation. 3. Templates define the structure of each interaction — how many turns, which APIs, what patterns. 4. Builder generates the interaction role-by-role (USER → THINKING → API_CALL → API_OBS → ASST), prompting the LLM for each. 5. Validator checks each generated role against API schemas and structural invariants. On failure, the role is retried with error feedback.

For more details, see the paper.

Customization Guide

| Goal | What to modify | |------|---------------| | Use your own APIs | Create api_definitions.json + ApiLibrary subclass | | Add a new interaction pattern | Add a template to template.py (or create new template set) | | Change user diversity | Define a PERSONA_SCHEMA dict and reference via schema_ref in config | | Adjust generation quality | Tune max_retries_per_role, temperature, max_tokens in YAML | | Add a new domain | Create a generator package with builder, template, and config |

Output Format

Each generated interaction follows this schema:

{
"interaction_template_name": "TURNS_1_APIS_2_PARALLEL_MULTIPLE",
"api_names": ["search_flights", "get_weather_forecast"],
"distractors": ["book_hotel", "get_stock_price", "translate_text", "send_email", "get_news"],
"interaction": [
{"SYSTEM": "Today's date is 2023/04/15"},
{"USER": "Find flights from NYC to Tokyo next week and check the weather."},
{"THINKING": "The user wants flights and weather. I need search_flights and get_weather_forecast..."},
{"API_CALL": "search_flights(origin='NYC', destination='Tokyo', date='2023-04-22')"},
{"API_OBS": {"flights": [{"airline": "JAL", "price": 850}]}},
{"API_CALL": "get_weather_forecast(location='Tokyo', days=7)"},
{"API_OBS": {"forecast": [{"day": "Mon", "temp": "18C", "condition": "Sunny"}]}},
{"ASST": "JAL has a flight on April 22nd for $850. Tokyo weather next week looks sunny, around 18°C."}
]
}

Citation

@misc{ramakrishnan2026dataturnstilescalableopen,
title={Data Turnstile: A Scalable Open Framework for Function-Calling Data Generation},
author={Goutham Ramakrishnan and Megha Sharma},
year={2026},
eprint={2607.29250},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2607.29250},
}

Reproducibility Statement

This code is being released solely for academic and scientific reproducibility purposes, in support of the methods and findings described in the associated publication. Pull requests are not being accepted in order to maintain the code exactly as it was used in the paper.

License

This project is licensed under CC-BY-NC-4.0.