logo
科言猫学术研究的AI总结
language
手机端手机端
  • Phone Feature微信小程序
简体中文简体中文
notice消息通知
博客博客
delete
arrow
返回

Cross-Modal Diffusion on Pretrained Alignment Codebook for Multimodal Machine Translation

delete2026-01-01
delete0
PRE
AI
G
Guojing Liu
D
Ding, Xiangqian
X
Xiangyu Qu
Z
Zhenyu Yang *
H
Huili Gong *
DOI：10.1109/TASLPRO.2026.3658943delete
delete原文链接
delete原文求助
delete分享
delete收藏
摘要

摘要

En 中文
多模态机器翻译旨在整合辅助信息（如图像）以提升文本翻译质量。尽管现有方法通过多种跨模态融合技术实现了有前景的性能，但它们均忽略了推理阶段解码速度的迫切需求，从而限制了其在实际场景中的应用。本文提出了一种新颖的多模态扩散翻译框架MDT，该框架在性能与效率之间取得平衡，能够并行生成高质量的目标翻译。具体而言，我们设计了一个具有共享语义的对齐码本，通过潜在对齐和跨模态对齐预训练任务，促进图像与文本输入间的有效关联。在此基础上构建跨模态扩散模型，利用视觉特征高效恢复潜在文本码字。我们还开发了Bézier噪声调度策略，使KL散度测量的去噪难度随时间步长稳健增长。在两个数据集的五种双语语言对上的实验结果表明，MDT相比先进方法更为有效，实现了更快的推理加速。
Keyword:
Translation
Diffusion models
Visualization
Transformers
Schedules
Machine translation
Noise reduction
Decoding
Semantics
Feature extraction
Inference speedup
alignment codebook
cross-modal diffusion model
multimodal machine translation

期刊

I
IEEE Transactions on Audio Speech and Language Processing
IF:
0
论文数:
151
被引数:
0

机构

Q
qilu university of technology
学者数:
2.1K
论文数: 642
被引数: 0
O
ocean university of china
学者数:
3.1W
论文数: 2.0W
被引数: 21
S
shandong university
学者数:
9.5W
论文数: 6.4W
被引数: 94
学者 查看更多机构
引用论文

引用论文

Distill The Image to Nowhere: Inversion Knowledge Distillation for Multimodal Machine Translation
errProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
IF0
err2022-01-01
err0
errOAAI
errRu Peng; Yawen Zeng; Jake Zhao
err分享
err收藏
BLEU布卢
errProceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02
IF0
err2001-01-01
err0
errOAAI
errKishore Papineni; Salim Roukos; Todd Ward; Wei-Jing Zhu
err分享
err收藏
Generative Imagination Elevates Machine Translation
errProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
IF0
err2021-01-01
err0
PREAI
errLong,Quanyu; Wang,Mingxuan; Li,Lei
err分享
err收藏
Post-Training Quantization on Diffusion Models
err2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
IF0
err2023-06-01
err0
errOAAI
errYuzhang Shang; Zhihang Yuan; Bin Xie; Bingzhe Wu; Yan Yan
err分享
err收藏
Multi30K: Multilingual English-German Image Descriptions
errProceedings of the 5th Workshop on Vision and Language
IF0
err2016-01-01
err0
errOAAI
errDesmond Elliott; Stella Frank; Khalil Sima'an; Lucia Specia
err分享
err收藏
WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning
errProceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval
IF0
err2021-07-11
err0
errOAAI
errKrishna Srinivasan; Karthik Raman; Jiecao Chen; Michael Bendersky; Marc Najork
err分享
err收藏
Dynamic Context-guided Capsule Network for Multimodal Machine Translation
errProceedings of the 28th ACM International Conference on Multimedia
IF0
err2020-10-12
err0
errOAAI
errHuan Lin; Fandong Meng; Jinsong Su; Yongjing Yin; Zhengyuan Yang; Yubin Ge; Jie Zhou; Jiebo Luo
err分享
err收藏
Findings of the 2014 Workshop on Statistical Machine Translation
errProceedings of the Ninth Workshop on Statistical Machine Translation
IF0
err2014-01-01
err0
errOAAI
errOndrej Bojar; Christian Buck; Christian Federmann; Barry Haddow; Philipp Koehn; Johannes Leveling; Christof Monz; Pavel Pecina; Matt Post; Herve Saint-Amand; Radu Soricut; Lucia Specia; Aleš Tamchyna
err分享
err收藏
Multimodal Neural Machine Translation Using Synthetic Images Transformed by Latent Diffusion Model
errProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)
IF0
err2023-01-01
err0
PREAI
errYuasa,Ryoya; Tamura,Akihiro; Kajiwara,Tomoyuki; Ninomiya,Takashi; Kato,Tsuneo
err分享
err收藏
Towards Multimodal Simultaneous Neural Machine Translation
errProceedings of the Fifth Conference on Machine Translation
IF0
err2020-01-01
err0
PREAI
errImankulova,Aizhan; Kaneko,Masahiro; Hirasawa,Tosho; Komachi,Mamoru
err分享
err收藏
学者 查看更多内容