QwenLM/BabelArena

QwenLM/BabelArena

Language: Python

License: NOASSERTION

Stars: 1

Forks: 0

Open issues: 0

Created: 2026-10-22T03:05:00Z

Pushed: 2026-10-23T07:11:33Z

Default branch: main

Fork: no

Archived: no

README:

📖 Introduction

BabelArena is a multilingual benchmark for LLM agents: 699 canonical tasks from four agent benchmark families, spanning 13 domains, each adapted into 23 languages for 16,077 task instances. Every language variant keeps the source task identity and evaluation criteria, so agent performance can be compared across languages on exactly the same tasks.

The language variants were built with BabelFlow, an agentic workflow that traces each benchmark's runtime data flow to find the text it can safely translate, translates it consistently across tasks, tools, databases, simulated users and graders, and accepts a variant only after multi-layer verification and human review.

Evaluating five frontier models, we find that no single model leads across all four families, and that the gaps between languages go beyond task success: lower-resource languages show larger shares of tool-use and control-flow errors, consume up to roughly twice the input tokens of English on the same tasks, and lose language consistency, with switches directed overwhelmingly toward English.

✨ Features

• 🧩 Four complementary agent settings. Interactive services, long-horizon

planning, knowledge-grounded tool use and heterogeneous autonomous tasks, each scored by its source benchmark's own evaluator.

| Family | What it tests | Domains | Tasks | |---|---|---|---:| | VitaBench | versatile interactive tasks with a simulated user | delivery, in-store, online travel, cross-domain | 200 | | DeepPlanning | long-horizon planning under verifiable constraints | travel, shopping | 120 | | τ³-Bench (text) | simulated-user interaction, tool calling, knowledge retrieval, policy compliance | airline, retail, telecom, banking | 212 | | ClawEval | service orchestration, multimodal perception and generation, professional dialogue | general, multimodal, multi-turn | 167 | | Total | | 13 | 699 |

• 🌍 23 languages, 15 scripts, three resource levels. The languages vary in

writing direction, word delimiters and glyph shaping.

| High resource (7) | Medium resource (8) | Low resource (8) | |---|---|---| | English en · Chinese zh · Japanese ja · French fr · Russian ru · Arabic ar · Spanish es | Korean ko · Indonesian id · Hindi hi · Thai th · Hebrew he · Turkish tr · Vietnamese vi · Malay ms | Belarusian be · Eastern Panjabi pa · Kazakh kk · Khmer km · Burmese my · Lao lo · Tamil ta · Telugu te |

• 🔧 Every component adapted. System, simulator and evaluation prompts, task

inputs, tool descriptions and environment content are all localized, with shared entities kept consistent across them. Among the multilingual agent benchmarks below, BabelArena adapts every component and has the broadest language and domain coverage.

| Benchmark | Prompt | Task | Tool | Env. | Tasks | Domains | Languages | Instances | |---|:---:|:---:|:---:|:---:|---:|---:|---:|---:| | MAPS | ✗ | ✓ | ✗ | ✗ | 805 | 4 | 12 | 9,660 | | GAIA-v2-LILT | ✓ | ✓ | ✗ | ✗ | 165 | 1 | 6 | 990 | | OmnilingualGAIA2 | ✓ | ✓ | ✗ | ✓ | 640 | 1 | 11 | 7,040 | | MLCL | ✗ | ✓ | ✗ | ✗ | 200 | 1 | 4 | 800 | | SEATauBench | ✓ | ✓ | ✓ | ✓ | 278 | 3 | 6 | 1,668 | | PolyWorkBench | ✓ | ✓ | ✗ | ✓ | 67 | 5 | 10 | 67 | | BabelArena | ✓ | ✓ | ✓ | ✓ | 699 | 13 | 23 | 16,077 |

• ✅ Verified translation quality. On 4,600 sampled translations (200 per

language), mean quality is 4.28 / 4.02 / 3.91 out of 5 for high-, medium- and low-resource languages (human experts for high-resource, inter-annotator Cohen's κ = 0.71). In a controlled ablation, BabelFlow cuts tasks with broken references from 20.0% to 2.5% and behavioral mismatches from 5.0% to 0.0% compared with independent field translation.

• 📏 Task success and language consistency, measured separately. Every task

runs three independent trials and reports Pass^1 (the per-trial success rate) and Pass^3 (all three trials succeed), plus language consistency (LC), the share of trajectories whose user-visible text stays in the target language.

🏆 Main Results

Five frontier models on BabelArena, in percent; higher is better, best in bold.

By benchmark family

| Model | VitaBench Pass^1 | Pass^3 | LC | DeepPlanning Pass^1 | Pass^3 | LC | τ³-Bench Pass^1 | Pass^3 | LC | ClawEval Pass^1 | Pass^3 | LC | |---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| | Claude-Opus-4.8 | 36.9 | 20.1 | 98.5 | 36.2 | 17.2 | 61.8 | 62.6 | 49.5 | 95.0 | 55.8 | 43.0 | 96.3 | | GPT-5.6-Terra | 29.6 | 15.1 | 99.1 | 22.5 | 10.7 | 90.0 | 69.2 | 38.5 | 82.5 | 46.1 | 33.5 | 98.7 | | Gemini-3.1-Pro | 34.3 | 17.7 | 99.9 | 18.6 | 5.6 | 74.3 | 71.4 | 46.9 | 86.2 | 41.5 | 29.3 | 98.8 | | Qwen-3.7-Max | 36.1 | 18.9 | 98.6 | 41.6 | 23.8 | 57.5 | 59.0 | 49.1 | 92.8 | 32.4 | 24.6 | 98.5 | | Qwen-3.8-Max | 42.3 | 25.0 | 94.7 | 46.0 | 26.5 | 75.8 | 67.6 | 57.0 | 87.6 | 40.8 | 27.1 | 88.7 |

By language resource level

| Model | High Pass^1 | Pass^3 | LC | Medium Pass^1 | Pass^3 | LC | Low Pass^1 | Pass^3 | LC | |---|---:|---:|---:|---:|---:|---:|---:|---:|---:| | Claude-Opus-4.8 | 51.3 | 34.3 | 90.4 | 48.1 | 33.9 | 87.2 | 44.8 | 29.6 | 86.7 | | GPT-5.6-Terra | 45.3 | 26.3 | 98.8 | 43.9 | 25.1 | 90.9 | 36.8 | 21.9 | 89.6 | | Gemini-3.1-Pro | 45.9 | 27.4 | 98.6 | 42.4 | 25.8 | 87.5 | 36.4 | 21.0 | 83.8 | | Qwen-3.7-Max | 45.8 | 32.4 | 89.0 | 42.9 | 29.5 | 86.9 | 40.1 | 27.6 | 84.9 | | Qwen-3.8-Max | 51.2 | 34.3 | 94.5 | 48.0 | 33.6 | 84.5 | 44.7 | 29.4 | 81.7 |

No model leads every family, and all five score lower on Pass^1, Pass^3 and LC in low-resource languages than in high-resource ones. See the paper for the failure, token-cost and language-switching analyses.

🛠️ Data

Download the BabelArena dataset from Hugging Face or ModelScope.

The repository carries the code and the frozen standard task lists; the task data (~14 GB) is...