Return
Benchmarking speech-to-text robustness in noisy emergency medical dialogues: an evaluation of models under realistic acoustic conditions
10.1093/jamiaopen/ooaf147
2025-12-01
0
OA
PDF
AI
Abstract
En 中文
Objectives To evaluate the transcription accuracy of 6 German-capable speech-to-text (STT) systems in simulated emergency medical services (EMS) environments, focusing on clinically relevant performance under noisy and multilingual field conditions.Materials and Methods We generated a corpus of 99 synthetic emergency dialogues and overlaid them with ecologically valid noise types-crowd chatter, traffic, public spaces, and ambulance interiors-at 5 signal-to-noise ratios (SNRs), producing 1980 noisy audio samples. Each was transcribed by 6 STT systems (recapp, Vosk, Whisper v3 variants, and RescueSpeech). We assessed performance using 5 metrics: Word Error Rate (WER), Medical Word Error Rate (mWER), TF-IDF Cosine Similarity, BLEU, and semantic embedding similarity. Statistical models quantified the effects of system, noise, and SNR on transcription fidelity.Results recapp consistently outperformed all other systems across metrics. Among open-source models, Whisper v3 Turbo achieved the lowest mWER and strongest phrase-level accuracy (BLEU), while Whisper v3 Large preserved semantic content best. RescueSpeech and Vosk underperformed. Inside crowded noise had the most degrading impact on performance, while talking noise had minimal effect. Performance degradation was most pronounced at the lowest SNR (-2 dB).Discussion STT model accuracy is highly sensitive to acoustic conditions. Clinically salient transcription errors (mWER) were most frequent under dense environmental noise. Whisper v3 Turbo balances accuracy and efficiency, suggesting strong potential for EMS applications.Conclusion This study introduces a clinically grounded, noise-robust benchmark for STT evaluation in EMS settings. It highlights the importance of domain-specific metrics and acoustic realism for deploying STT systems where transcription errors carry safety-critical consequences. This study tests how well 6 German speech-to-text (STT) systems turn spoken emergency conversations into written text when there is real-world noise. We focus on emergency medical services (EMS), where paramedics work in loud settings such as crowded stations, traffic, and inside ambulances. We created 99 realistic emergency dialogues and mixed them with 4 common noise types at 5 loudness levels, then compared each system's transcripts with the correct text.Overall, one commercial system (recapp) was most accurate. Among open-source options, Whisper v3 Turbo offered the best balance of accuracy and speed, while Whisper v3 Large preserved overall meaning well. Systems called RescueSpeech and Vosk were less accurate. The most harmful noise was inside crowded public-space noise; simple background talking had the smallest effect. Accuracy dropped sharply in very challenging conditions where noise was louder than speech.We also measured errors on medical terms (eg, drug names or procedures) because mistakes there can affect patient safety. Our results provide a practical benchmark to choose STT tools for EMS and show that testing must reflect real acoustic conditions, not just quiet rooms.
Keywords:
speech recognition
emergency medical services
speech-to-text
word error rate
clinical documentation
AI Summary
Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.
Journal
J
IF:
3.4
Papers: 251
・
Citations: 1.7K
Researchers
M
Moser, Denis
H-index:
0
Papers: 1
・
Citations: 0
S
Stanic, Nikola
H-index:
0
Papers: 1
・
Citations: 0
S
Sariyar, Murat
H-index:
0
Papers: 1
・
Citations: 0
Organization
No organization information available
Cited Papers