SARS: A Novel Face and Body Shape and Appearance Aware 3D Reconstruction System extends Morphable Models

Morphable Models (3DMMs) are a type of morphable model that takes 2D images as inputs and recreates the structure and physical appearance of 3D objects, especially human faces and bodies. 3DMM combines identity and expression blendshapes with a basic face mesh to create a detailed 3D model. The variability in the 3D Morphable models can be controlled by tuning diverse parameters. They are high-level image descriptors, such as shape, texture, illumination, and camera parameters. Previous research in 3D human reconstruction concentrated solely on global face structure or geometry, ignoring face semantic features such as age, gender, and facial landmarks characterizing facial boundaries, curves, dips, and wrinkles. In order to accommodate changes in these high-level facial characteristics, this work introduces a shape and appearance-aware 3D reconstruction system (named SARS by us), a c modular pipeline that extracts body and face information from a single image to properly rebuild the 3D model of the human full body.

💡 Research Summary

This paper presents SARS (Shape and Appearance-aware Reconstruction System), a novel modular pipeline for reconstructing a detailed 3D human avatar from a single 2D image. The work addresses key limitations in existing 3D Morphable Model (3DMM) and body reconstruction methods, which often fail to capture high-level semantic facial attributes (like age, gender, and fine wrinkles) or struggle to balance detail between the face and body in a unified model.

The SARS framework consists of four core modules. The first is a Face Feature Extraction Module, which utilizes a multi-task learning convolutional neural network (based on a shared ResNet backbone with lightweight dedicated branches) to simultaneously predict facial landmarks (68 points), age category, and gender from an input face crop.

The second, the 3D Face Reconstruction Module, enhances a traditional 3DMM. It generates standard identity and expression blendshape coefficients, a displacement map, and a signed distance field (SDF) from the input image. Crucially, it then encodes these elements along with the semantic features (age, gender, landmarks) from the first module into a latent space. This combined representation is decoded to produce a refined 3D face mesh that preserves identity while incorporating age, gender, and expression-driven details like wrinkles and muscle deformations.

The third module is the Body Reconstruction Module. It follows an optimization-based approach using the SMPLify method. It estimates initial body pose, shape, and camera parameters from the image, refines them, and then uses the Skinned Multi-Person Linear (SMPL) model to generate a 3D body mesh.

Finally, the Integration Module performs a fusion operation, seamlessly replacing the coarse face region of the 3D body mesh with the high-fidelity 3D face mesh generated by the second module. This results in a complete, unified 3D human avatar.

The paper positions SARS within the literature, reviewing regression-based, parametric, deep learning, single-image, and modular approaches. It argues that SARS adopts a hybrid strategy, leveraging the strengths of parametric models (3DMM, SMPL), deep learning for feature extraction, single-image input, and a modular architecture. This modular design allows each component (face-specific and body-specific reconstruction) to be optimized independently using state-of-the-art techniques for its respective domain, leading to improved accuracy and controllability.

Experiments conducted on datasets such as 3DHumans demonstrate the system’s promising performance in accurately reconstructing facial identity, expression, age, and body shape simultaneously. The paper concludes by highlighting the benefits of this decoupled, specialized approach for creating realistic full-body avatars and suggests future work on more sophisticated fusion techniques and evaluation under even more challenging in-the-wild conditions.