Repository navigation
Friction log from new user #24
Description
Hey all! I'm writing a friction log as a new user and someone with 0 experience with diffusion models. I hope this is useful.
Overall, I loved that the README had a definition section, it was very quick to get things going and I was able to have some result with existing pipelines very quickly. As a first step I just went through the README, I'll kick off some training tomorrow. Awesome work everyone
1. Going through the README
I1. (first code example)
When I see https://huggingface.co/fusing/ddpm-lsun-church, the repo shows as transformers. Two questions arise
- Do we want a new tag for diffusers?
- What about adding a code snippet?
DiffusionPipeline.from_pretrained(model_id)
(both should be easy and happy to help with it)
I2 (first code example)
I tried the code snippet suggested at https://huggingface.co/fusing/ddpm-lsun-church, but it fails
model_id = "fusing/ddpm-lsun-church"
ddpm = DiffusionPipeline.from_pretrained(model_id)with
AttributeError: module 'diffusers' has no attribute 'GaussianDDPMScheduler'
(same for second exampke)
This also makes me wonder what's the difference between just using the diffusion pipeline directly as in the model card vs using DDPMScheduler + UNetModel approach as in the README. Is the pipeline approach just a wrapper of both?
I3 (second code example)
Should num_inference_steps be len(noise_scheduler) as in first example?
I4
I just quickly browsed through the RL guide, but it seems the pretrained model is downloaded from Dropbox 😅 should we add it to the Hub?
I5
The short examples in section 2 are 🔥. They are short, clear, concise and nice results!
I6
It would be nice to have metadata in the model repos to be able to filter models depending what they can do (tts, text-to-image, etc).
Activity
osanseviero commentedon Jun 19, 2022
As a second step I trained a model with a new dataset based on the example script. It was very straightforward 🔥 and everything worked together very nicely, except for my GPU not allowing larger models, so I had to decrease resolution. I would have loved a
push_to_hubmethod to upload itpatil-suraj commentedon Jun 19, 2022
Thanks a lot for the feedback !
I1. (first code example):
diffuserstag would be awesome.I2 (first code example):
The example is failing because of renaming.
GaussianDDPMScheduleris nowDDPMScheduler. These will be fixed once we settle on names. Fixed this example for now :)We are aiming for two APIs here:
press a buttonAPI with pipeline, where user could just load the pipeline and use it as it is for inference. This is more abstract and black-box API to play with pre-trained pipelines.barebone/follow-the-equation: Which offers more control , allows to mix and match different schedulers and models, write your own denoising loop. This is close to how these models are presented in research papers.I3 (second code example)
some schedulers like DDIM allows you to use different
num_inference_stepsthan what was used during the training to allow fast inference. So this is intended,num_inference_stepsandlen(noise_scheduler)can be different.I4 and I6:
The models were added in a fast hacky way to get the initial version rolling. All of these will improved before formal release.
@anton-l is already working on it :)
osanseviero commentedon Jun 19, 2022
I1
I2
I3
*Modelclasses andpipelineintransformers. Sounds exciting!For push to Hub, feel free me to tag me in the PR once opened, it would be cool to have metrics and other metadata out-of-the-box
patrickvonplaten commentedon Jul 21, 2022
Thanks for the feedback here @osanseviero - think we applied almost all of it now :-)
Merge pull request huggingface#24 from yuyanpeng-google/yuyan-dev
Flux3 image (#15046)