Skip to content

Latest commit

 

History

History
executable file
·
133 lines (109 loc) · 5.99 KB

README.md

File metadata and controls

executable file
·
133 lines (109 loc) · 5.99 KB

DECA: Detailed Expression Capture and Animation (SIGGRAPH2021)

input image, aligned reconstruction, animation with various poses & expressions

Open In Colab

This is the official Pytorch implementation of DECA.

DECA reconstructs a 3D head model with detailed facial geometry from a single input image. The resulting 3D head model can be easily animated. Please refer to the arXiv paper for more details.

The main features:

  • Reconstruction: produces head pose, shape, detailed face geometry, and lighting information from a single image.
  • Animation: animate the face with realistic wrinkle deformations.
  • Robustness: tested on facial images in unconstrained conditions. Our method is robust to various poses, illuminations and occlusions.
  • Accurate: state-of-the-art 3D face shape reconstruction on the NoW Challenge benchmark dataset.

Getting Started

Clone the repo:

git clone https://github.com/YadiraF/DECA
cd DECA

Requirements

  • Python 3.7 (numpy, skimage, scipy, opencv)
  • PyTorch >= 1.6 (pytorch3d)
  • face-alignment (Optional for detecting face)
    You can run 
    pip install -r requirements.txt
    Or use virtual environment by runing 
    bash install_conda.sh
    For visualization, we use our rasterizer that uses pytorch JIT Compiling Extensions. If there occurs a compiling error, you can install pytorch3d instead and set --rasterizer_type=pytorch3d when running the demos.

Usage

  1. Prepare data
    run script:

    bash fetch_data.sh

    (Optional for Albedo)
    follow the instructions for the Albedo model to get 'FLAME_albedo_from_BFM.npz', put it into ./data

  2. Run demos
    a. reconstruction

    python demos/reconstruct_cmd.py -i TestSamples/examples --saveDepth True --saveObj True

    to visualize the predicted 2D landmanks, 3D landmarks (red means non-visible points), coarse geometry, detailed geometry, and depth.

    You can also generate an obj file (which can be opened with Meshlab) that includes extracted texture from the input image.

    Please run python demos/demo_reconstruct.py --help for more details.

    b. expression transfer

    python demos/transfer_emotions_cmd.py

    Given an image, you can reconstruct its 3D face, then animate it by tranfering expressions from other images. Using Meshlab to open the detailed mesh obj file, you can see something like that:

    (Thank Soubhik for allowing me to use his face ^_^)

    Note that, you need to set '--useTex True' to get full texture.

Training

  1. Prepare Training Data

    a. Download image data
    In DECA, we use VGGFace2, BUPT-Balancedface and VoxCeleb2

    b. Prepare label
    FAN to predict 68 2D landmark
    face_segmentation to get skin mask

    c. Modify dataloader
    Dataloaders for different datasets are in decalib/datasets, use the right path for prepared images and labels.

  2. Download face recognition trained model
    We use the model from VGGFace2-pytorch for calculating identity loss, download resnet50_ft, and put it into ./data

  3. Start training

    Train from scratch:

    python train.py --cfg configs/release_version/deca_pretrain.yml 
python train.py --cfg configs/release_version/deca_coarse.yml 
python train.py --cfg configs/release_version/deca_detail.yml

    In the yml files, write the right path for 'output_dir' and 'pretrained_modelpath'.
    You can also use released model as pretrained model, then ignor the pretrain step.

Related works:

  • for better emotion prediction: EMOCA
  • for better skin estimation: TRUST

Citation

If you find our work useful to your research, please consider citing:

@inproceedings{DECA:Siggraph2021,
  title={Learning an Animatable Detailed {3D} Face Model from In-The-Wild Images},
  author={Feng, Yao and Feng, Haiwen and Black, Michael J. and Bolkart, Timo},
  journal = {ACM Transactions on Graphics, (Proc. SIGGRAPH)}, 
  volume = {40}, 
  number = {8}, 
  year = {2021}, 
  url = {https://doi.org/10.1145/3450626.3459936} 
}

License

This code and model are available for non-commercial scientific research purposes as defined in the LICENSE file. By downloading and using the code and model you agree to the terms in the LICENSE.