SwapController: Toward Improving Identity and Attribute Control for Diffusion-Based Face Swapping

F. Liu

doi:10.1109/tvcg.2026.3702263

SwapController: Toward Improving Identity and Attribute Control for Diffusion-Based Face Swapping

2026-09-01

0

PRE

AI

DOI：10.1109/tvcg.2026.3702263

Original

Original request for help

Save

Abstract

En 中文

Face swapping efforts strive to achieve high-fidelity and well-controlled generation effects. Owing to the remarkable generative capabilities, diffusion models deliver promising high-fidelity solutions. However, their intrinsic stochastic properties complicate the accurate modeling of facial representations, introducing new challenges for identity and attribute consistency of the generated faces. In this paper, we introduce a novel diffusion-based face-swapping framework, named SwapController, which achieves high-fidelity generation via careful facial identity and attribute modeling. Specifically, our facial modeling mainly involves facial structure and facial texture. For structure modeling, 3D facial priors are leveraged to provide explicit structure supervision, enabling accurate head structure control. On this basis, two novel components are proposed to deeply mine facial textural representations from identity and attribute aspects. To enhance identity control, multi-grained source identity embeddings are obtained from various functional encoders to convey critical global identity and fine-grained identity details. To improve attribute modeling, identity-shifted attribute embeddings are derived by applying identity modulation to the most salient textural attribute features of the target face. Moreover, in line with the diffusion denoising characteristics, a timestep-aware identity optimization objective is introduced to optimize identity consistency guidelines and overall fidelity. Extensive experiments demonstrate the effectiveness of our SwapController in generating identity-consistent portrait images while faithfully preserving target attributes, which obtains a 98.32 ID Retrieval, exceeding the SOTA DiffSFSR by 7.32 up arrow.

Keywords:

Faces

Modeling

Noise reduction

Computers

Conferences

Training

Visualization

Computer vision

Videos

Optimization

Face swapping

diffusion models

identity transfer

SwapController: Toward Improving Identity and Attribute Control for Diffusion-Based Face Swapping

Abstract

Journal

Organization

Cited Papers