Lei Yang2 Ziwei Liu1+
[Project Page] •
[arXiv] •
[Video] •
[Colab Demo] •
[Hugging Face Demo]
Accepted to ICCV 2023
Abstract: 3D human motion generation is crucial for creative industry. Recent advances rely on generative models with domain knowledge for text-driven motion generation, leading to substantial progress in capturing common motions. However, the performance on more diverse motions remains unsatisfactory. In this work, we propose ReMoDiffuse, a diffusion-model-based motion generation framework that integrates a retrieval mechanism to refine the denoising process.
Pipeline Overview: ReMoDiffuse is a retrieval-augmented 3D human motion diffusion model. Benefiting from the extra knowledge from the retrieved samples, ReMoDiffuse is able to achieve high-fidelity on the given prompts. It contains three core components: a) Hybrid Retrieval database stores multi-modality features of each motion sequence. b) Semantics-modulated transformer incorporates several identical decoder layers, including a Semantics-Modulated Attention (SMA) layer and an FFN layer. The SMA layer will adaptively absorb knowledge from both retrived samples and the given prompts. c) Condition Mxture technique is proposed to better mix model's outputs under different combinations of conditions.
[09/2023] Add a 🤗Hugging Face Demo!
[09/2023] Add a Colab Demo!
[09/2023] Release code for ReMoDiffuse and MotionDiffuse
If you find our work useful for your research, please consider citing the paper:
@article{zhang2023remodiffuse,
title={ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model},
author={Zhang, Mingyuan and Guo, Xinying and Pan, Liang and Cai, Zhongang and Hong, Fangzhou and Li, Huirong and Yang, Lei and Liu, Ziwei},
journal={arXiv preprint arXiv:2304.01116},
year={2023}
}
@article{zhang2022motiondiffuse,
title={MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model},
author={Zhang, Mingyuan and Cai, Zhongang and Pan, Liang and Hong, Fangzhou and Guo, Xinying and Yang, Lei and Liu, Ziwei},
journal={arXiv preprint arXiv:2208.15001},
year={2022}
}
# Create Conda Environment
conda config --add channels nvidia
conda create -n mogen python=3.10 -y
conda activate mogen
# C++ Environment
export PATH=/mnt/lustre/share/gcc/gcc-8.5.0/bin:$PATH
export LD_LIBRARY_PATH=/mnt/lustre/share/gcc/gcc-8.5.0/lib:/mnt/lustre/share/gcc/gcc-8.5.0/lib64:/mnt/lustre/share/gcc/gmp-4.3.2/lib:/mnt/lustre/share/gcc/mpc-0.8.1/lib:/mnt/lustre/share/gcc/mpfr-2.4.2/lib:$LD_LIBRARY_PATH
# Install Mamba
conda install mamba -n base -c conda-forge
# Install scikit-learn
mamba install scikit-learn -y
export MAMBA_NO_LOW_SPEED_LIMIT=1 && export MAMBA_SOLVER_NO_LOW_SPEED_LIMIT=1
ulimit -v unlimited
ulimit -m unlimited
# Install Pytorch
mamba install pytorch=1.12.1 torchvision=0.13.1 torchaudio=0.12.1 cudatoolkit=11.3 numpy=1.21.0 scipy diffusers transformers -c pytorch -c conda-forge -y
# mamba install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.8 numpy=1.21.0 diffusers transformers -c pytorch -c nvidia
# Install MMCV
# source /etc/network_turbo
pip install "mmcv-full>=1.4.2,<=1.9.0" -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.12.1/index.html
# unset http_proxy && unset https_proxy
# Install Pytorch3d
mamba install -c bottler nvidiacub -y
mamba install -c fvcore -c iopath -c conda-forge fvcore iopath -y
mamba install pytorch3d -c pytorch3d -y
# Install other requirements
# source /etc/network_turbo
pip install -r requirements.txt
python -c "from transformers import pipeline; print(pipeline('sentiment-analysis')('we love you'))"
# unset http_proxy && unset https_proxyNote: If you use pytorch in 1.12.1, you need to install xformers manually.
cd ..
git clone --branch v0.0.16 https://github.com/facebookresearch/xformers.git
cd xformers
git submodule update --init --recursive
pip install -r requirements.txt
pip install -e .
cd ReMoDiffuseDownload data files from google drive link or Baidu Netdisk link(access code: vprc). Unzipped all files and arrange them in the following file structure:
ReMoDiffuse
├── mogen
├── tools
├── configs
├── logs
│ ├── motiondiffuse
│ ├── remodiffuse
│ └── mdm
└── data
├── database
├── datasets
├── evaluators
└── glove
PYTHONPATH=".":$PYTHONPATH python tools/train.py ${CONFIG_FILE} ${WORK_DIR} --no-validateNote: The provided config files are designed for training with 8 gpus. If you want to train on a single gpu, you can reduce the number of epochs to one-fourth of the original.
./tools/slurm_train.sh ${PARTITION} ${JOB_NAME} ${CONFIG_FILE} ${WORK_DIR} ${GPU_NUM} --no-validateCommon optional arguments include:
--resume-from ${CHECKPOINT_FILE}: Resume from a previous checkpoint file.--no-validate: Whether not to evaluate the checkpoint during training.
Example: using 8 GPUs to train ReMoDiffuse on a slurm cluster.
./tools/slurm_train.sh my_partition my_job configs/remodiffuse/remodiffuse_kit.py logs/remodiffuse_kit 8 --no-validatePYTHONPATH=".":$PYTHONPATH python tools/test.py ${CONFIG} --work-dir=${WORK_DIR} ${CHECKPOINT}./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} ${CONFIG} ${WORK_DIR} ${CHECKPOINT}Example:
./tools/slurm_test.sh my_partition test_remodiffuse configs/remodiffuse/remodiffuse_kit.py logs/remodiffuse_kit logs/remodiffuse_kit/latest.pthNote: Run full evaluation for HumanML3D dataset is very slow. You can change replication_times in human_ml3d_bs128.py to
PYTHONPATH=".":$PYTHONPATH python tools/visualize.py ${CONFIG} ${CHECKPOINT} \
--text ${TEXT} \
--motion_length ${MOTION_LENGTH} \
--out ${OUTPUT_ANIMATION_PATH} \
--device cpuExample:
PYTHONPATH=".":$PYTHONPATH python tools/visualize.py \
configs/remodiffuse/remodiffuse_t2m.py \
logs/remodiffuse/remodiffuse_t2m/latest.pth \
--text "a person is running quickly" \
--motion_length 120 \
--out "test.gif" \
--device cpuThis study is supported by the Ministry of Education, Singapore, under its MOE AcRF Tier 2 (MOE-T2EP20221-0012), NTU NAP, and under the RIE2020 Industry Alignment Fund – Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s).
The visualization tool is developed on top of Generating Diverse and Natural 3D Human Motions from Text