Cross-Modal Diffusion on Pretrained Alignment Codebook for Multimodal Machine Translation

Guojing Liu

doi:10.1109/TASLPRO.2026.3658943

Cross-Modal Diffusion on Pretrained Alignment Codebook for Multimodal Machine Translation

2026-01-01

0

PRE

AI

DOI：10.1109/TASLPRO.2026.3658943

Original

Original request for help

Save

Abstract

En 中文

Multimodal machine translation aims to integrate auxiliary information, such as images, to enhance the quality of text translation. While existing methods deliver promising performance through various cross-modal fusion techniques, they all ignore the urgent need for decoding speed during inference, which limits their applicability in real-world scenarios. In this paper, we propose a novel multimodal diffusion translation framework called MDT, which balances performance and efficiency to generate high-quality target translation in parallel. Specifically, we design an alignment codebook with shared semantics, which facilitates efficient correlation between image and text inputs through latent alignment and multimodal alignment pretraining tasks. A cross-modal diffusion model is built on the codebook to recover latent text codewords efficiently using visual features. We also develop a B & eacute;zier noise schedule, which allows the denoising difficulty measured by KL-divergence to grow robustly with the time step. Experimental results on two datasets with five bilingual language pairs demonstrate the effectiveness of MDT over advanced methods, achieving superior inference speedup.

Keywords:

Translation

Diffusion models

Visualization

Transformers

Schedules

Machine translation

Noise reduction

Decoding

Semantics

Feature extraction

Inference speedup

alignment codebook

cross-modal diffusion model

multimodal machine translation

Cross-Modal Diffusion on Pretrained Alignment Codebook for Multimodal Machine Translation

Abstract

Journal

Organization