automatic/DIFFUSERS.md

Diffusers WiP

initial support merged into dev branch

git clone https://github.com/vladmandic/automatic -b dev diffusers
cd diffusers
webui --debug --backend diffusers

default sd 1.5 model will be downloaded automatically to models/Diffusers

on first startup, disable controlnet and multi-diffusion extensions as right now they are not compatible with diffusers
lora support is not compatible with setting Use LyCoris handler for all Lora types, make sure its disabled

to update repo, do not use --upgrade flag, use manual git pull instead

Test

Standard

goal is to test standard workflows (so not diffusers) to ensure there are no regressions
so diffusers code can be merged into master and we can continue with development there

• run with webui --debug --backend original

Diffusers

whats implemented so far?

• simple model downloader for huggingface models: tabs -> models -> hf hub
• use huggingface models
• extra networks ui
• use safetensor models with diffusers backend
• lowvram and medvram equivalents for diffusers
• standard workflows:
  • txt2img, img2img, inpaint, outpaint, process
  • hires fix, restore faces, etc?
• textual inversion
  yes, this applies to standard embedddings, don't need ones from huggingface
• lora
  yes, this applies to standard loras, don't need ones from huggingface
  but seems that diffuser lora support is somewhat limited, so quite a few loras may not work
  you should see which lora loads without issues in console log
• system info tab with updated information
• kandinsky model
  works for me

Experimental

• cuda model compile in settings -> compute settings
  diffusers recommend reduce overhead, but other methods are available as well
  it seems that fullgraph is possible (with sufficient vram) when using diffusers
• deepfloyd
  in theory it should work, but its 20gb model so cant test it just yet
  note that access is gated, so you'll need to download using your huggingface credentials
  (you can still do it from sdnext ui, just need access token)

Todo

• sdxl model

Limitations

even if extensions are not supported, runtime errors are never nice
will need to handle in the code before we get out of alpha

• controlnet sd_model.model?.diffusion_model?
• multi-diffusion
  sd_model.first_stage_model?.encoder?
• lycoris lyco_patch_lora

Issues

• seed vs batch size?

Notes for HF

• removed quicksettings alternative completely
• added simple model downloader in ui: tabs -> models -> huggingface
• redone textual inversion support, core is now in modules/textual_inversion/textual_inversion.py:load_diffusers_embedding()
  the point is that sdnext pre-loads all compatible embeddings on model load so they are available in prompt context
• added support for diffuser models in safetensors/ckpt format
  btw, when i use: diffusers.StableDiffusionPipeline.from_ckpt
  first time it downloads something - what is that?

  Downloading (…)lve/main/config.json: 4.55k
  Downloading pytorch_model.bin: 1.22G
  and in general, loading safetensors model is quite slow, is that expected?
  for example, 2sec vs 18sec

• in modules/modelloader.py:download_diffusers_model() i get unknown property for hf.model_info(hub_id).cardData can you double-check if this is linter issue or actual problem?
• redone lora support, core is now in modules/lora_diffusers.py
• question on pipe.load_lora_weights does it support loading multiple loras? i don't see any notes on that in docs also, lora strength is specified using cross_attention_kwargs={"scale": x} during pipeline execution
  which means if there are multiple loras, they all have the same strength?