Bài báo quốc tế
Kho tri thức
/
Bài báo quốc tế
/
LiveNeRF: Efficient face replacement through Neural Radiance Fields integration
LiveNeRF: Efficient face replacement through Neural Radiance Fields integration
Vũ Sơn Tùng
Synthesizing photorealistic talking head videos from single facial images and speech audio presents significant challenges in modeling natural head motion dynamics, ensuring temporal coherence, and preserving subject identity under computational constraints. Current diffusion-based approaches face deployment limitations due to computationally intensive training and large-scale dataset requirements, while efficient NeRF-based methods require hours of person-specific training per subject. This paper introduces LiveNeRF, a structural redesign of Efficient Region-aware Neural Radiance Fields (ER-NeRF) that integrates face replacement capabilities directly within the model architecture. We position this as an architectural contribution: rather than proposing new algorithmic components, we demonstrate that restructuring an existing ER-NeRF model by removing its person-specific output head and replacing it with face manipulation modules (a motion extractor, keypoint-based warping network, and identity decoder) yields measurable practical benefits over both the original model and naive two-stage combinations of the same components. The resulting architecture is a single, cohesive neural model in which the ER-NeRF backbone produces a pre-output-head intermediate representation
that retains audio-conditioned motion information from volume rendering; this representation is then directly processed by the appended face manipulation modules within the same forward pass, bypassing the original output head that would produce a person-specific image. This structural redesign eliminates the need for person-specific training by leveraging pretrained motion priors, enabling zero-shot synthesis from a single reference image. Our evaluation demonstrates competitive visual quality (PSNR: 33.05 dB, LPIPS: 0.0315, FID: 10.65), effective lip synchronization (Sync: 5.680, LMD: 2.765), and real-time performance (33 FPS), establishing LiveNeRF as a practical solution for talking head synthesis.
Xuất bản trên:
LiveNeRF: Efficient face replacement through Neural Radiance Fields integration
Ngày đăng:
2026
Nhà xuất bản:
Computer Vision and Image Understanding
Địa điểm:
Từ khoá:
Talking head synthesis; Neural Radiance Fields; Real-time rendering; Face replacement
Bài báo liên quan
MIST: A Multilingual Dataset and Benchmark for Fine-Grained Audio Inpainting Tampering Localization
Vũ Sơn TùngNF-DCL: Enhancing video anomaly detection with synthetic normal features and Debiased Contrastive Learning
Nguyễn Thu Nga