Reparameterizing and dynamically quantizing image features for image generation

Mingzhen Sun

doi:10.1016/j.patcog.2023.109962

2024-02-01

PRE

DOI：10.1016/j.patcog.2023.109962

原文链接

原文求助

摘要

En 中文

For autoregressive image generation, vector-quantized VAEs (VQ-VAEs) quantize image features with discrete codebook entries and reconstruct images from quantized features. However, they treat each codebook entry separately, which causes losses of image details. In this paper, we propose to reparameterize image features with weight vectors to treat all codebook entries as an entity, and present a novel dynamically vector quantized VAE (DVQ-VAE) to quantize reparameterized image features. Specifically, each image feature corresponds to a weight vector and we sum weighted codebook entries to obtain values of image features. In this way, image features can incorporate information from different codebook entries. Additionally, a novel continuous weight regularization loss is proposed to improve the reconstruction of image details. Our method achieves competitive results with prior state-of-the-art works for image generation and extensive experiments are conducted to take a deep insight into our DVQ-VAE.

Keyword:

Vector quantization

Variational auto-encoder

Unconditional image generation

Text-to-image generation

Autoregressive generation

期刊

Pattern Recognition

IF:

7.6

论文数:

1.3W

被引数:

4.5W

机构

chinese academy of sciences

学者数:

56.7W

论文数: 45.0W

被引数: 704

引用论文

ImageNet Large Scale Visual Recognition Challenge ImageNet大规模视觉识别挑战

INTERNATIONAL JOURNAL OF COMPUTER VISION

IF9.3

2015-04-11

2.7W

PREAI

Russakovsky, Olga; Deng, Jia; Su, Hao; Krause, Jonathan; Satheesh, Sanjeev; Ma, Sean; Huang, Zhiheng; Karpathy, Andrej; Khosla, Aditya; Bernstein, Michael; Berg, Alexander C.; Fei-Fei, Li

A New Perspective on Stabilizing GANs Training: Direct Adversarial Training 稳定GANs训练的新视角: 直接对抗训练

IEEE TRANSACTIONS ON EMERGING TOPICS IN COMPUTATIONAL INTELLIGENCE

IF6.5

2023-02-01