RosettaSpeech: Zero-Shot Speech-to-Speech Translation without Parallel Speech

Zhisheng Zheng, Xiaohang Sun, Tuan Dinh, Abhishek Yanamandra, Abhinav Jain, Zhu Liu, Sunil Hadap, Vimal Bhat, Manoj Aggarwal, Gerard Medioni, David Harwath · Nov 26, 2025 · Citations: 0

Automatic Metrics

Open arXiv Find Implementation RSS feed

Abstract

End-to-end speech-to-speech translation (S2ST) systems typically struggle with a critical data bottleneck: the scarcity of parallel speech-to-speech corpora. To overcome this, we introduce RosettaSpeech, a novel zero-shot framework trained exclusively on monolingual speech-text data augmented by machine translation supervision. Unlike prior works that rely on complex cascaded pseudo-labeling, our approach strategically utilizes text as a semantic bridge during training to synthesize translation targets, thereby eliminating the need for parallel speech pairs while maintaining a direct, end-to-end inference pipeline. Empirical evaluations on the CVSS-C benchmark demonstrate that RosettaSpeech achieves state-of-the-art zero-shot performance, surpassing leading baselines by significant margins - achieving ASR-BLEU scores of 25.17 for German-to-English (+27% relative gain) and 29.86 for Spanish-to-English (+14%). Crucially, our model effectively preserves the source speaker's voice without ever seeing paired speech data. We further analyze the impact of data scaling and demonstrate the model's capability in many-to-one translation, offering a scalable solution for extending high-quality S2ST to "text-rich, speech-poor" languages.

HFEPX Relevance Assessment

This paper appears adjacent to HFEPX scope (human-feedback/eval), but does not show strong direct protocol evidence in metadata/abstract.

Eval-Fit Score

0/100 • Low

Treat as adjacent context, not a core eval-method reference.

Human Feedback Signal

Not explicit in abstract metadata

Evaluation Signal

Detected

HFEPX Fit

Adjacent candidate

If you are doing eval pipeline work, start here:

Human Eval Hub LLM-as-Judge Hub Pairwise Preference Hub Tool-Use Eval Hub

Human Data Lens

Uses human feedback: No
Feedback types: None
Rater population: Unknown
Unit of annotation: Unknown
Expertise required: Multilingual
Extraction source: Persisted extraction

Evaluation Lens

Evaluation modes: Automatic Metrics
Agentic eval: None
Quality controls: Not reported
Confidence: 0.35
Flags: low_signal, possible_false_positive

Protocol And Measurement Signals

Benchmarks / Datasets

No benchmark or dataset names were extracted from the available abstract.

Reported Metrics

bleujailbreak success rate

Research Brief

Deterministic synthesis

To overcome this, we introduce RosettaSpeech, a novel zero-shot framework trained exclusively on monolingual speech-text data augmented by machine translation supervision. HFEPX signals include Automatic Metrics with confidence 0.35. Updated from current HFEPX corpus.

Generated Mar 4, 2026, 6:18 AM · Grounded in abstract + metadata only

Key Takeaways

To overcome this, we introduce RosettaSpeech, a novel zero-shot framework trained exclusively on monolingual speech-text data augmented by machine translation supervision.
Empirical evaluations on the CVSS-C benchmark demonstrate that RosettaSpeech achieves state-of-the-art zero-shot performance, surpassing leading baselines by significant margins -…

Researcher Actions

Treat this as method context, then pivot to protocol-specific HFEPX hubs.
Identify benchmark choices from full text before operationalizing conclusions.
Validate metric comparability (bleu, jailbreak success rate).

Caveats

Generated from title, abstract, and extracted metadata only; full-paper implementation details are not parsed.
Low-signal flag detected: protocol relevance may be indirect.

Recommended Queries

human-eval protocol design pairwise preference data quality inter-rater agreement adjudication

Research Summary

Contribution Summary

To overcome this, we introduce RosettaSpeech, a novel zero-shot framework trained exclusively on monolingual speech-text data augmented by machine translation supervision.
Empirical evaluations on the CVSS-C benchmark demonstrate that RosettaSpeech achieves state-of-the-art zero-shot performance, surpassing leading baselines by significant margins - achieving ASR-BLEU scores of 25.17 for German-to-English…

Why It Matters For Eval

Empirical evaluations on the CVSS-C benchmark demonstrate that RosettaSpeech achieves state-of-the-art zero-shot performance, surpassing leading baselines by significant margins - achieving ASR-BLEU scores of 25.17 for German-to-English…

Researcher Checklist

Gap: Human feedback protocol is explicit

No explicit human feedback protocol detected.
Pass: Evaluation mode is explicit

Detected: Automatic Metrics
Gap: Quality control reporting appears

No calibration/adjudication/IAA control explicitly detected.
Gap: Benchmark or dataset anchors are present

No benchmark/dataset anchor extracted from abstract.
Pass: Metric reporting is present

Detected: bleu, jailbreak success rate

Category-Adjacent Papers (Broader Context)

These papers are nearby in arXiv category and useful for broader context, but not necessarily protocol-matched to this paper.

MUSE: A Run-Centric Platform for Multimodal Unified Safety Evaluation of Large Language Models Category Neighbor

Citations: 0 Relevance: 6.50
- Shared arXiv category (eess.AS, cs.CL)
- Shared metric mentions
- Shared terminology (success, rate)
Building a Strong Instruction Language Model for a Less-Resourced Language Category Neighbor

Citations: 0 Relevance: 4.10
- Shared arXiv category (cs.CL, cs.LG)
- Shared terminology (rate, translation)
FlashEvaluator: Expanding Search Space with Parallel Evaluation Category Neighbor

Citations: 0 Relevance: 3.65
- Shared arXiv category (cs.CL, cs.LG)
- Shared terminology (parallel)
From Variance to Invariance: Qualitative Content Analysis for Narrative Graph Annotation Category Neighbor

Citations: 0 Relevance: 3.20
- Shared arXiv category (cs.CL, cs.LG)
Legal RAG Bench: an end-to-end benchmark for legal RAG Category Neighbor

Citations: 0 Relevance: 3.20
- Shared arXiv category (cs.CL, cs.LG)
When Numbers Tell Half the Story: Human-Metric Alignment in Topic Model Evaluation Category Neighbor

Citations: 0 Relevance: 3.20
- Shared arXiv category (cs.CL, cs.LG)
Evaluating Performance Drift from Model Switching in Multi-Turn LLM Systems Category Neighbor

Citations: 0 Relevance: 2.50
- Shared arXiv category (cs.CL)
- Shared terminology (success, rate)
StitchCUDA: An Automated Multi-Agents End-to-End GPU Programing Framework with Rubric-based Agentic Reinforcement Learning Category Neighbor

Citations: 0 Relevance: 2.50
- Shared arXiv category (cs.CL)
- Shared terminology (success, rate)

Need human evaluators for your AI research? Scale annotation with expert AI Trainers.

Post a Job Get a Quote