Audio Interaction Model
Abstract
Domain fit: AI-core · Core AI workload signals detected from paper context and implementation/artifact evidence.
Audio is an inherently interactive modality, yet today's Large Audio Language Models (LALMs) are offline, and streaming audio models each handle only a single task such as streaming ASR or voice chatting. It is time to unify them into one online LALM: a model that, through an always-on perceive-decide-respond loop, listens to sound, environment, and instructions in real time and reacts on the fly. We formalize this regime as the Audio Interaction Model, and realize it with Audio-Interaction, a unified streaming model that retains offline task execution while adding online general audio instruction following, from dialogue to full voice chatting, deciding when to respond from the semantics of the stream. To enable this, we propose SoundFlow, a framework that instantiates the perceive-decide-respond loop end to end, from data to training to deployment, through streaming-native data construction, comprehension-aware training, and asynchronous low-latency inference for stable real-time interaction. We further construct StreamAudio-2M, a 2.6M-item streaming corpus spanning 7 fundamental abilities and 28 sub-tasks, and Proactive-Sound-Bench for evaluating proactive audio intervention. Across 8 benchmarks, Audio-Interaction preserves competitive performance on mainstream audio tasks while unlocking capabilities inaccessible to offline LALMs, including real-time ASR, streaming audio instruction following, and proactive help.
Results and benchmarks
Audio is an inherently interactive modality, yet today's Large Audio Language Models (LALMs) are offline, and streaming audio models each handle only a single task such as streaming ASR or voice chatting.
| Task | Dataset | Metric | Value | Source |
|---|---|---|---|---|
| Machine translation | LibriSpeech | WER | 2.93 | paper-derived |
Audit each benchmark finding before selecting an implementation path. Evidence refs map to the disclosure below.
Evidence graph: 4 refs, 4 links.
Utility signals: depth 95/100, grounding 95/100, status high.
Implementation
Best maintained implementation now
xzf-thu/Audio-Interaction
584 stars · 33 forks · Last push Jun 4, 2026
- License
- CI
- Dependencies
- Docker
Matched via arXiv identifier search · Strong overlap with paper title keywords · Community adoption signal (584 stars)
xzf-thu/Audio-Interaction is the best available implementation candidate based on ranking signals, but recommendation confidence is not yet high. Dependency/environment manifests are present.
Open xzf-thu/Audio-Interaction- License metadata missing
- No CI workflows detected
- Selected xzf-thu/Audio-Interaction as the strongest maintained implementation for new work.
- Includes dependency/environment manifest signals.
- Repository activity is within the last 24 months.
Compare implementation paths
Compare maintenance quality, reproducibility coverage, and evidence confidence before choosing a reproduction baseline.
- Maintenance
- Active
- Confidence
- Low
- Reproducibility
- Strong
- Stars
- 186,523
- Last push
- Aug 20, 2026 (3d)
Strong overlap with paper title keywords · Community adoption signal (186523 stars)
- No Docker setup
- Low confidence match
- Maintenance
- Recently updated
- Confidence
- Medium
- Reproducibility
- Limited
- Stars
- 584
- Last push
- Jun 4, 2026 (81d)
Matched via arXiv identifier search · Strong overlap with paper title keywords
- No CI pipeline detected
- No tagged releases
- No Docker setup
- Maintenance
- Recently updated
- Confidence
- Medium
- Reproducibility
- Limited
- Stars
- 0
- Last push
- Jul 14, 2026 (41d)
Matched via arXiv identifier search · Strong overlap with paper title keywords
- No CI pipeline detected
- No tagged releases
- No Docker setup
Reproduction readiness
Setup required
Dependencies pinned, manual setup needed
- xzf-thu/Audio-Interaction has requirements.txt but requires manual environment setup.
- No Dockerfile, so you will set up the environment manually.
- No CI pipeline, so test coverage is unknown.
Quick start
git clone https://github.com/xzf-thu/Audio-Interaction.git
pip install -r requirements.txt Repositories and ecosystem
No additional verified repositories beyond the primary recommendation.
These repositories had low-confidence matching signals and are hidden by default.
- yt-dlp/yt-dlp
Confidence: Low · 186,523 stars
- coding-sharks/AI-ft
Confidence: Medium · 0 stars
Hugging Face artifacts
No direct paper-linked artifacts were found. Showing strongest curated related artifacts for faster exploration.
Datasets
No trustworthy datasets matches right now.
Search datasets on Hugging FaceSpaces
No trustworthy spaces matches right now.
Search spaces on Hugging FaceResearch context
Tasks
Instruction tuning
Methods
Transformer
Domains
Natural Language Processing, Large Language Models
Open this paper in HFEPX to review benchmark signals, evaluation modes, and human-feedback protocol context.
Open in HFEPXJump to Paper2Code search queries derived from this paper's research context.