Audio2Tool

Audio2ToolSpeak, Call, Act — A Dataset for Benchmarking Speech Tool Use

Ramit Pahwa ★†, Apoorva Beedu†, Parivesh Priye, Rutu Gandhi, Saloni Takawale, Aruna Baijal, Zengli Yang

Rivian and Volkswagen Group Technologies

★ Research lead

† Equal contribution

Interspeech 2026 · pp. 3453–3457

DOI: 10.21437/Interspeech.2026-2857

A benchmark for speech function calling and audio-native tool use across smart car, smart home, and wearable APIs.

PaperPDFDatasetCodeLeaderboard

Abstract

Voice assistants increasingly rely on Speech Language Models (SpeechLMs) to interpret spoken queries and execute complex tasks, yet existing benchmarks lack domain breadth, acoustic diversity, and compositional reasoning complexity to evaluate tool-calling performance. We introduce Audio2Tool, a large-scale dataset comprising approximately 30,000 queries designed to assess tool-calling capabilities of SpeechLMs across three primary domains: Smart Car, Smart Home, and Wearables. Our benchmark features a multi-tier complexity hierarchy, ranging from simple direct commands to complex multi-intent and needle-in-a-haystack extraction to isolate distinct failure modes. To ensure realism, we employ zero-shot voice cloning text-to-speech synthesis and diverse noise profiles to simulate in-the-wild conditions. Evaluations of state-of-the-art SpeechLMs and ASR-LLM pipelines show strong performance on simple commands but significant degradation under compositional and acoustic challenges.

Benchmark scope

Audio2Tool evaluates spoken tool selection and argument grounding. It supports comparisons of Speech Language Models (SpeechLMs) and speech recognition plus language model (ASR–LLM) pipelines on structured tool-call prediction.

The eight tiers test direct commands, parameter extraction, multiple intents, implicit requests, distracting context, self-corrections, multi-turn dialogue, and overlapping foreground and background speech. Speech is synthesized using reference voices; noise and intent blending probe acoustic robustness. Predicted tool calls do not establish successful execution in a live environment.

Earlier preprint title: “Audio2Tool: Bridging Spoken Language Understanding and Function Calling.” The preprint is available as arXiv:2604.22821.

Tool Taxonomy

Category distribution across the benchmark (Smart Car, Smart Home, Wearables).

Tool taxonomy: category distribution across domains

Dataset & Samples

Explore representative audio samples and ground-truth tool calls for each complexity tier. Select a tier and play samples with transcripts.

Dataset files and release documentation are available on Hugging Face. The paper reports benchmark-level counts; see the dataset card for release contents and subset statistics.

Eight evaluation tiers
TierTaskChallenge
1. DirectSingle direct commandSimple intent recognition.
2. ParametricSlot/value extractionParameter/slot extraction.
3. Multi-IntentMultiple actions in one utterancePredicting multiple tool calls.
4. ImplicitIndirect or implied intentPragmatic inference.
5. NeedleLong speech with tangentsFiltering irrelevant context.
6. CorrectionMid-utterance repairMid-utterance state repair.
7. ConversationMulti-turn dialoguePersistent intent tracking.
8. Intent BlendingMultiple speakers / primary focusSpeaker diarization / primary actor focus.
Tier 1

Direct

Single direct command

Key challenge: Simple intent recognition.

Loading tier data…

Key Results

Loading figures…

Citation

Pahwa, R., Beedu, A., Priye, P., Gandhi, R., Takawale, S., Baijal, A., and Yang, Z. (2026). Audio2Tool: Speak, Call, Act - A Dataset for Benchmarking Speech Tool Use. Interspeech 2026, 3453–3457. doi: 10.21437/Interspeech.2026-2857.

Download BibTeX · Official proceedings record · arXiv preprint

@inproceedings{pahwa26_interspeech,
  title     = {{Audio2Tool: Speak, Call, Act - A Dataset for Benchmarking Speech Tool Use}},
  author    = {Ramit Pahwa and Apoorva Beedu and Parivesh Priye and Rutu Gandhi and Saloni Takawale and Aruna Baijal and Zengli Yang},
  year      = {2026},
  booktitle = {{Interspeech 2026}},
  pages     = {3453--3457},
  doi       = {10.21437/Interspeech.2026-2857},
  issn      = {2958-1796},
  url       = {https://www.isca-archive.org/interspeech_2026/pahwa26_interspeech.html}
}