DP-DocLDM: Differentially Private Document Image Generation Using Latent Diffusion Models

Saifullah Saifullah

doi:10.1007/978-3-032-04627-7_6

Return

DP-DocLDM: Differentially Private Document Image Generation Using Latent Diffusion Models

2026-01-01

PRE

Saifullah Saifullah *

Stefan Agne

Andreas Dengel

Sheraz Ahmed

DOI：10.1007/978-3-032-04627-7_6

Original

Original request for help

Save

Abstract

En 中文

As deep learning-based, data-driven information extraction systems become increasingly integrated into modern document processing workflows, one primary concern is the risk of malicious leakage of sensitive private data from these systems. While some recent works have explored Differential Privacy (DP) to mitigate these privacy risks, DP-based training is known to cause significant performance degradation and impose several limitations on standard training procedures, making its direct application to downstream tasks both difficult and costly. In this work, we aim to address the above challenges within the context of document image classification by substituting real private data with a synthetic counterpart. In particular, we propose to use conditional latent diffusion models (LDMs) in combination with differential privacy (DP) to generate class-specific synthetic document images under strict privacy constraints, which can then be utilized to train a downstream classifier following standard training procedures. We investigate our approach under various pretraining setups, including unconditional, class-conditional, and layout-conditional pretraining, in combination with multiple private training strategies such as class-conditional and per-label private fine-tuning with DPDM and DP-Promise algorithms. Additionally, we evaluate it on two well-known document benchmark datasets, RVL-CDIP and Tobacco3482, and show that it can generate useful and realistic document samples across various document types and privacy levels (epsilon is an element of{1, 5, 10}). Lastly, we show that our approach achieves substantial performance improvements in downstream evaluations on small-scale datasets, compared to the direct application of DP-Adam. To the best of the authors' knowledge, this is the first work to explore differentially private document generation via latent diffusion for document analysis. The source code for this work has been made publicly available at: https://github.com/saifullah3396/dpdocldm.git.

Keywords:

Differential Privacy

Document Image Classification

Latent Diffusion Models

Private Document Image Generation

Differentially Private Diffusion Models