add optional external post-processing toolchain, add brief chinese readme

2023-01-20 18:21:38 +08:00 · 2023-01-20 18:21:38 +08:00 · cda0dff372
parent 1b46078b6a
commit cda0dff372
8 changed files with 340 additions and 12 deletions
--- a/.gitignore
+++ b/.gitignore
@ -0,0 +1,9 @@
+# meta
+.vscode/
+__pycache__/
+
+# third party tools
+tools/*
+!tools/README.md
+!tools/*.cmd
+!tools/busybox.exe
--- a/README.md
+++ b/README.md
@ -34,6 +34,7 @@ Try interpolating on the hidden vectors of conditioning prompt to make seemingly

 ⚪ Features

+- 2023/01/20: `v2.0` add optional external [post-processing pipeline](#post-processing-pipeline) to highly boost up smoothness, greate thx to [Real-ESRGAN](https://github.com/xinntao/Real-ESRGAN) and [RIFE](https://github.com/nihui/rife-ncnn-vulkan)!!
 - 2023/01/16: `v1.5` add upscale options (issue #12); add 'embryo' genesis, reproducing idea of [stable-diffusion-animation](https://replicate.com/andreasjansson/stable-diffusion-animation) except [FILM](https://github.com/google-research/frame-interpolation) support (issue #11)
 - 2023/01/12: `v1.4` remove 'replace' & 'grad' mode support, due to webui's code change
 - 2022/12/11: `v1.3` work in a more 'successive' way, idea borrowed from [deforum](https://github.com/deforum-art/deforum-for-automatic1111-webui) ('genesis' option)
@ -58,6 +59,7 @@ Try interpolating on the hidden vectors of conditioning prompt to make seemingly
  - freeze all other settings (`steps`, `sampler`, `cfg factor`, `seed`, etc.)
  - note that only the major `seed` will be forcely fixed through all processes, you can still set `subseed = -1` to allow more variances
 - export a video!
+  - follow [post-processing pipeline](#post-processing-pipeline) to get much better result 👌

 ⚪ Txt2Img

@ -73,13 +75,18 @@ Try interpolating on the hidden vectors of conditioning prompt to make seemingly
 | Eular a | ![i2i-f-euler_a](img/i2i-f-euler_a.gif) | ![i2i-s-euler_a](img/i2i-s-euler_a.gif) | ![i2i-e-euler_a](img/i2i-e-euler_a.gif) |
 | DDIM    | ![i2i-f-ddim](img/i2i-f-ddim.gif)       | ![i2i-s-ddim](img/i2i-s-ddim.gif)       | ![i2i-e-ddim](img/i2i-e-ddim.gif)       |

-Reference image for img2img:
+post-processing pipeline (case `i2i-f-ddim`):

-![i2i-ref](img/i2i-ref.png)
+| w/o. post-processing | w/. post-processing |
+| :-: | :-: |
+| ![i2i-f-ddim](img/i2i-f-ddim.gif) | ![i2i-f-ddim-pp](img/i2i-f-ddim-pp.gif) |

-Embryo image decoded (case `i2i-e-euler_a` with `embryo_step=8`):
+other stuff:
+
+| reference image for img2img | embryo image decoded <br/> case `i2i-e-euler_a` with `embryo_step=8` |
+| :-: | :-: |
+| ![i2i-ref](img/i2i-ref.png) | ![embryo](img/embryo.png) |

-![embryo](img/embryo.png)

 Example above run configure:

@ -106,7 +113,7 @@ Hypernet: (this is my secret :)
 - prompt: (list of strings)
 - negative prompt: (list of strings)
  - input multiple lines of prompt text
-  - we call each line of prompt a stage, usually you need at least 2 lines of text to starts travel (unless in 'grad' mode)
+  - we call each line of prompt a stage, usually you need at least 2 lines of text to starts travel
  - if len(positive_prompts) != len(negative_prompts), the shorter one's last item will be repeated to match the longer one
 - steps: (int, list of int)
  - number of images to interpolate between two stages
@ -116,8 +123,8 @@ Hypernet: (this is my secret :)
  - `fixed`: starts from pure noise in txt2img pipeline, or from the same ref-image given in img2img pipeline
  - `successive`: starts from the last generated image (this will force txt2img turn to actually be img2img from the 2nd frame on)
  - `embryo`: starts from the same half-denoised image, see [=> How does it work?](https://replicate.com/andreasjansson/stable-diffusion-animation#readme) 
-    - (experimental): it only processes 2 lines of prompts, and does not interpolate on negative_prompt, and with no FILM postprocessing :(
- genesis_extra_params:
+    - (experimental) it only processes 2 lines of prompts, and does not interpolate on negative_prompt :(
+- genesis_extra_params
  - denoise_strength: (float), denoise strength in img2img pipelines (for `successive`)
  - embryo_step: (int or float), steps to hatch the common embryo (for `embryo`)
    - if >= 1, taken as step cout
@ -143,6 +150,38 @@ Manual install:
 2. (Optional) Restart the webui


+### Post-processing pipeline
+
+There are still two steps away from a really smooth and high resolution animation, namely image **super-resolution** & video **frame interpolation** (see `third-party tools` below).  
+⚠ Media data processing is intrinsic resource-exhausting, and it's also not webui's work or duty, hence we separated it out. 😃
+
+#### setup once
+
+⚪ auto install
+
+- run `tools/install.cmd`
+- if you got any errors like `Access denied.`, try run it again until you see `Done!` without errors 😂
+- you will have three components: [Real-ESRGAN](https://github.com/xinntao/Real-ESRGAN-ncnn-vulkan), [RIFE](https://github.com/nihui/rife-ncnn-vulkan) and [FFmpeg](https://ffmpeg.org/) installed under the [tools](tools) folder
+
+⚪ manually install
+
+- understand the `tools` folder layout => [tools/README.txt](tools/README.txt)
+  - if you indeed wanna put the tools elsewhere, modify paths in [tools/link.cmd](tools/link.cmd) and run `tools/link.cmd` 😉
+- download [Real-ESRGAN](https://github.com/xinntao/Real-ESRGAN/releases) (e.g.: `realesrgan-ncnn-vulkan-20220424-windows.zip`)
+  - (optional) download interesting seperated model checkpoints (e.g.: `realesr-animevideov3.pth`)
+- download [rife-ncnn-vulkan](https://github.com/nihui/rife-ncnn-vulkan/releases) bundle (e.g.: `rife-ncnn-vulkan-20221029-windows.zip `)
+- download [FFmpeg](https://ffmpeg.org/download.html) binary (e.g.: `ffmpeg-release-full-shared.7z` or `ffmpeg-git-full.7z`)
+
+#### run each time
+
+- check params in [postprocess.cmd](postprocess.cmd)
+- pick one way to start 😃
+  - run `postprocess.cmd path/to/<image_folder>` from command line
+  - drag & drop any image folder over `postprocess.cmd` icon
+
+ℹ Once processing finished, the explorer will be auto lauched to locate the generated file named with `synth.mp4`
+
+
 ### Related Projects

 ⚪ extensions that inspired this repo
@ -155,18 +194,23 @@ Manual install:
 - deforum (img2img + depth model): [https://github.com/deforum-art/deforum-for-automatic1111-webui](https://github.com/deforum-art/deforum-for-automatic1111-webui)
 - seed-travel (varying seed): [https://github.com/yownas/seed_travel](https://github.com/yownas/seed_travel)

-⚪ third-party gives us power
+⚪ third-party tools

 - image super-resoultion
-  - Real-ESRGAN: [https://github.com/xinntao/Real-ESRGAN](https://github.com/xinntao/Real-ESRGAN)
-  - ESRGAN: [https://github.com/xinntao/ESRGAN](https://github.com/xinntao/ESRGAN)
+  - ESRGAN:
+    - ESRGAN: [https://github.com/xinntao/ESRGAN](https://github.com/xinntao/ESRGAN)
+    - Real-ESRGAN: [https://github.com/xinntao/Real-ESRGAN](https://github.com/xinntao/Real-ESRGAN)
+    - Real-ESRGAN-ncnn-vulkan (recommended): [https://github.com/xinntao/Real-ESRGAN-ncnn-vulkan](https://github.com/xinntao/Real-ESRGAN-ncnn-vulkan)
 - video frame interpolation
-  - FILM: [https://github.com/google-research/frame-interpolation](https://github.com/google-research/frame-interpolation)
+  - FILM (recommended): [https://github.com/google-research/frame-interpolation](https://github.com/google-research/frame-interpolation)
  - RIFE:
    - ECCV2022-RIFE: [https://github.com/megvii-research/ECCV2022-RIFE](https://github.com/megvii-research/ECCV2022-RIFE)
-    - rife-ncnn-vulkan: [https://github.com/nihui/rife-ncnn-vulkan](https://github.com/nihui/rife-ncnn-vulkan)
+    - rife-ncnn-vulkan (recommended): [https://github.com/nihui/rife-ncnn-vulkan](https://github.com/nihui/rife-ncnn-vulkan)
    - Squirrel-RIFE: [https://github.com/Justin62628/Squirrel-RIFE](https://github.com/Justin62628/Squirrel-RIFE)
    - Practical-RIFE: [https://github.com/hzwer/Practical-RIFE](https://github.com/hzwer/Practical-RIFE)
+- GNU tool-kits
+  - busybox: [https://www.busybox.net/](https://www.busybox.net/)
+  - ffmpeg: [https://ffmpeg.org/](https://ffmpeg.org/)

 ⚪ my other experimental toy extensions

--- a/README.zh_CN.md
+++ b/README.zh_CN.md
@ -0,0 +1,91 @@
+# 提示词跃迁
+
+    在模型隐层旅行以制作伪动画，项目 AUTOMATIC1111/stable-diffusion-webui 的插件。
+
+----
+
+对语言理解模型 CLIP 的输出进行插值，从而实现多条提示词之间的语义过渡，产生看似连续的图像序列，或者说伪动画。😀  
+
+⚠ 我们成立了插件反馈 QQ 群: 616795645 (赤狐屿)，欢迎出建议、意见、报告bug等 (w  
+
+ℹ 实话不说，我想有可能通过这个来做ppt童话绘本<del>甚至本子</del>……  
+ℹ 聪明的用法：先手工盲搜两张好看的图 (只有提示词差异)，然后再尝试在其间跃迁 :lolipop:  
+
+
+### 使用方法 & 它如何工作
+
+- 在提示词/负向提示词框里输入**多行**文本，每一行被称作一个**阶段**
+- 逐帧生成图像，在每个阶段内，所使用的提示词向量是经过插值运算的
+  - 为了保证某种连续性，所有其他参数将被固定
+  - 虽然所有图的主随机数种子将被统一固定，但你仍然可以启用 `subseed` 去增加随机性
+- 导出视频！
+  - 使用额外的 [后处理流程](#post-processing-pipeline) 可以获得更好的画质和流畅度 👌
+
+
+### 参数选项
+
+- 提示词: (多行文本)
+- 反向提示词: (多行文本)
+  - 就是提示词和反向提示词的输入框，但是你必须输入多行文本，每一行是一个阶段
+  - 如果提示词和反向提示词的阶段数量不一致，少的那一方会被重复到对齐多的一方
+- 插帧数/steps: (整数，或者逗号分隔的多个整数)
+  - 每个阶段之间插帧的数量
+  - 若为单个整数，每个阶段使用相同的插帧数量
+  - 若为西文逗号分隔的多个整数，每个阶段使用不同的插帧数量，比如有4个阶段则可给出3个独立步数：`12, 24, 36`
+- 起源/genesis: (选项), 每张图像的内容先验
+  - `固定/fixed`: 在 txt2img 流程中，始终从高斯噪声开始降噪；在 img2img 流程中，始终从给定的参考图开始降噪
+  - `连续/successive`: 从上一帧的内容开始降噪 (这会导致 txt2img 流程从第二步开始强制转为 img2img 流程)
+  - `胚胎/embryo`: 从某个已部分降噪的公共先祖胚胎开始降噪，参考 [=> 原理](https://replicate.com/andreasjansson/stable-diffusion-animation#readme) 
+    - (该功能为实验性质) 只支持两个阶段跃迁，并且不能为逆向提示词插值 :(
+- 起源的额外参数
+  - 降噪强度: (浮点数), 在 img2img 流程中所用的降噪强度 (仅对 `连续/successive` 模式)
+  - 胚胎步数: (整数或浮点数), 产生公共胚胎的预降噪步数 (仅对 `胚胎/embryo` 模式)
+    - 如果 >= 1，解释为采样步数
+    - 如果 < 1，解释为占总采样步数的比例
+- 视频相关
+  - 帧率/fps: (浮点数), 导出视频的帧率，设置为 `0` 将禁用导出
+  - 文件格式/fmt: (选项), 导出视频的文件格式
+  - 首尾填充/pad: (整数), 重复首尾帧 `N` 次以留出一段入场/退场时间
+  - 帧选择器/pick: (切片器), 使用 [Python切片语法](https://www.pythoncentral.io/how-to-slice-listsarrays-and-tuples-in-python) 精心选择所需要导出的帧，注意切片发生在填充之前 (例如：设为 `::2` 将只使用偶数帧 , 设为 `:-1` 将去除最后一帧)
+- 调试开关: (逻辑值)
+  - 是否在控制台显示详细日志
+
+
+### 后处理流程
+
+单凭CLIP模型自身能实现语义插值就已经到达能力天花板了，但我们距离高清丝滑的动画还差两步： **图像超分辨率** 和 **视频插帧**。 
+⚠ 多媒体数据的处理是非常消耗资源的，我们不能指望 webui 去做这件事。实际上，我们将其从宿主和插件中分离，出来作为一个可选的外部工具。 😃
+
+#### 安装依赖
+
+⚪ 自动安装
+
+- 运行 `tools/install.cmd`
+- 如果遇到诸如 `访问被拒绝` 之类的错误，多次运行直到提示 `Done!` 无错误退出 😂
+- 你将安装好 [Real-ESRGAN](https://github.com/xinntao/Real-ESRGAN-ncnn-vulkan), [RIFE](https://github.com/nihui/rife-ncnn-vulkan), [FFmpeg](https://ffmpeg.org/) 这三个组件在 [tools](tools) 目录下
+
+⚪ 手动安装
+
+- 参照 [README.md](README.md#post-processing-pipeline)
+- 我寻思你既然都想着手动安装了，也不至于不肯咬一口英文罢…… 🤔
+
+#### 运行任务
+
+- 检查 [postprocess.cmd](postprocess.cmd) 中的默认参数
+- 你有两种方式启动后处理任务 😃
+  - 从命令行运行 `postprocess.cmd path/to/<image_folder>`
+  - 鼠标拖拽任意图片文件夹到 `postprocess.cmd` 的文件图标上然后释放
+
+ℹ 任务完成后，资源浏览器将被自动打开并定位到导出的 `synth.mp4` 文件~
+
+
+插件直出和加入后处理对比 (配置为 `img2img-fixed-ddim`)：
+
+| 插件直出 | 加入后处理 |
+| :-: | :-: |
+| ![i2i-f-ddim](img/i2i-f-ddim.gif) | ![i2i-f-ddim-pp](img/i2i-f-ddim-pp.gif) |
+
+
+----
+by Armit
+2023/01/20 
--- a/img/i2i-f-ddim-pp.gif
+++ b/img/i2i-f-ddim-pp.gif
--- a/postprocess.cmd
+++ b/postprocess.cmd
@ -0,0 +1,82 @@
+@REM Test script for post-process pipeline
+@ECHO OFF
+SETLOCAL
+
+REM =================================================
+REM Real-ESRGAN model ckpt
+SET RESR_MODEL=realesr-animevideov3
+REM image upscale rate, must choose from [2, 3, 4]
+SET RESR_UPSCALE=2
+REM RIFE model ckpt
+SET RIFE_MODEL=rife-v4
+REM interpolated frame count, set 0 to simply Nx2
+SET RIFE_INTERP=0
+REM rendered video fps, higher value requires more interpolations
+SET FPS=20
+REM =================================================
+
+
+IF NOT EXIST "%~1" (
+  ECHO Usage: %~nx0 path/to/^<image_folder^> [-k]
+  ECHO    -k     keep cache data for debug
+  PAUSE
+  GOTO :EOF
+)
+
+SET RESR_HOME=%~dp0tools\realesrgan-ncnn-vulkan
+SET RIFE_HOME=%~dp0tools\rife-ncnn-vulkan
+SET FFMPEG_HOME=%~dp0tools\ffmpeg
+
+SET RESR_BIN=realesrgan-ncnn-vulkan.exe
+SET RIFE_BIN=rife-ncnn-vulkan.exe
+SET FFMPEG_BIN=ffmpeg.exe
+
+PATH %RESR_HOME%;%PATH%
+PATH %RIFE_HOME%;%PATH%
+PATH %FFMPEG_HOME%\bin;%FFMPEG_HOME%;%PATH%
+
+SET IMAGE_FOLDER=%~1
+SET RESR_FOLDER="%IMAGE_FOLDER%\resr"
+SET RIFE_FOLDER="%IMAGE_FOLDER%\rife"
+SET OUT_FILE="%IMAGE_FOLDER%\synth.mp4"
+
+ECHO ==================================================
+
+ECHO [1/5] image super-resolution
+IF EXIST %RESR_FOLDER% GOTO skip_resr
+MKDIR %RESR_FOLDER%
+%RESR_BIN% -v -s %RESR_UPSCALE% -n %RESR_MODEL% -i %IMAGE_FOLDER% -o %RESR_FOLDER%
+IF ERRORLEVEL 1 GOTO die
+:skip_resr
+ECHO ==================================================
+
+ECHO [2/5] video frame-interpolation
+IF EXIST %RIFE_FOLDER% GOTO skip_rife
+MKDIR %RIFE_FOLDER%
+SET NFRAMES=%RESR_FOLDER%
+
+%RIFE_BIN% -v -n %RIFE_INTERP% -m %RIFE_MODEL% -i %RESR_FOLDER% -o %RIFE_FOLDER%
+IF ERRORLEVEL 1 GOTO die
+:skip_rife
+ECHO ==================================================
+
+ECHO [3/5] render video
+%FFMPEG_BIN% -y -framerate %FPS% -i %RIFE_FOLDER%\%%08d.png -crf 20 -c:v libx264 -pix_fmt yuv420p %OUT_FILE%
+IF ERRORLEVEL 1 GOTO die
+ECHO ==================================================
+
+IF NOT "%2"=="-k" (
+  RMDIR /S /Q %RESR_FOLDER%
+  RMDIR /S /Q %RIFE_FOLDER%
+)
+
+explorer.exe /e,/select,%OUT_FILE%
+
+ECHO ^>^> Done!
+ECHO.
+GOTO :eof
+
+:die
+ECHO ^<^< errorlevel: %ERRORLEVEL%
+
+:eof
--- a/tools/busybox.exe
+++ b/tools/busybox.exe
--- a/tools/install.cmd
+++ b/tools/install.cmd
@ -0,0 +1,84 @@
+@REM Auto download and setup post-process tools
+@ECHO OFF
+SETLOCAL
+
+TITLE Install tools for post-process...
+
+SET CURL_BIN=curl.exe -L -C -
+SET UNZIP_BIN=busybox.exe unzip
+
+SET RESR_URL=https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesrgan-ncnn-vulkan-20220424-windows.zip
+SET RESR_ZIP=realesrgan-ncnn-vulkan.zip
+SET RESR_DIR=realesrgan-ncnn-vulkan
+
+SET RIFE_URL=https://github.com/nihui/rife-ncnn-vulkan/releases/download/20221029/rife-ncnn-vulkan-20221029-windows.zip
+SET RIFE_ZIP=rife-ncnn-vulkan.zip
+SET RIFE_DIR=rife-ncnn-vulkan
+SET RIFE_RDIR=rife-ncnn-vulkan-20221029-windows
+
+SET FFMPEG_URL=https://github.com/GyanD/codexffmpeg/releases/download/5.1.2/ffmpeg-5.1.2-full_build-shared.zip
+SET FFMPEG_ZIP=ffmpeg.zip
+SET FFMPEG_DIR=ffmpeg
+SET FFMPEG_RDIR=ffmpeg-5.1.2-full_build-shared
+
+SET DOWNLOAD_DIR=.download
+IF NOT EXIST %~dp0%DOWNLOAD_DIR% MKDIR %~dp0%DOWNLOAD_DIR%
+ATTRIB +H %~dp0%DOWNLOAD_DIR%
+
+ECHO ==================================================
+
+ECHO [1/3] install Real-ESRGAN
+IF EXIST %~dp0%RESR_DIR% GOTO skip_resr
+IF EXIST %DOWNLOAD_DIR%\%RESR_ZIP% GOTO skip_dl_resr
+ECHO ^>^> download from %RESR_URL%
+%CURL_BIN% %RESR_URL% -o %DOWNLOAD_DIR%\%RESR_ZIP%
+IF ERRORLEVEL 1 GOTO die
+:skip_dl_resr
+ECHO ^>^> uzip %RESR_ZIP%
+MKDIR %~dp0%RESR_DIR%
+%UNZIP_BIN% %DOWNLOAD_DIR%\%RESR_ZIP% -d %~dp0%RESR_DIR%
+IF ERRORLEVEL 1 GOTO die
+:skip_resr
+ECHO ==================================================
+
+ECHO [2/3] install RIFE
+IF EXIST %~dp0%RIFE_DIR% GOTO skip_rife
+IF EXIST %DOWNLOAD_DIR%\%RIFE_ZIP% GOTO skip_dl_rife
+ECHO ^>^> download from %RIFE_URL%
+%CURL_BIN% %RIFE_URL% -o %DOWNLOAD_DIR%\%RIFE_ZIP%
+IF ERRORLEVEL 1 GOTO die
+:skip_dl_rife
+ECHO ^>^> uzip %RIFE_ZIP%
+%UNZIP_BIN% %DOWNLOAD_DIR%\%RIFE_ZIP% -d %~dp0
+IF ERRORLEVEL 1 GOTO die
+RENAME %~dp0%RIFE_RDIR% %RIFE_DIR%
+:skip_rife
+ECHO ==================================================
+
+ECHO [3/3] install FFmpeg
+IF EXIST %~dp0%FFMPEG_DIR% GOTO skip_ffmpeg
+IF EXIST %DOWNLOAD_DIR%\%FFMPEG_ZIP% GOTO skip_dl_ffmpeg
+ECHO ^>^> download from %FFMPEG_URL%
+%CURL_BIN% %FFMPEG_URL% -o %DOWNLOAD_DIR%\%FFMPEG_ZIP%
+IF ERRORLEVEL 1 GOTO die
+:skip_dl_ffmpeg
+ECHO ^>^> uzip %FFMPEG_ZIP%
+%UNZIP_BIN% %DOWNLOAD_DIR%\%FFMPEG_ZIP% -d %~dp0
+IF ERRORLEVEL 1 GOTO die
+RENAME %~dp0%FFMPEG_RDIR% %FFMPEG_DIR%
+:skip_ffmpeg
+ECHO ==================================================
+
+ECHO ^>^> Done!
+ECHO.
+
+ATTRIB -H %~dp0%DOWNLOAD_DIR%
+RMDIR /S /Q %~dp0%DOWNLOAD_DIR%
+
+PAUSE
+GOTO :eof
+
+:die
+ECHO ^<^< errorlevel: %ERRORLEVEL%
+
+:eof
--- a/tools/link.cmd
+++ b/tools/link.cmd
@ -0,0 +1,18 @@
+@REM Make soft links to post-process tools
+@ECHO OFF
+SETLOCAL
+
+SET RESR_HOME=D:\tools\realesrgan-ncnn-vulkan
+SET RIFE_HOME=D:\tools\rife-ncnn-vulkan
+SET FFMPEG_HOME=D:\tools\ffmpeg
+
+@ECHO ON
+
+MKLINK /J %~dp0realesrgan-ncnn-vulkan %RESR_HOME%
+MKLINK /J %~dp0rife-ncnn-vulkan       %RIFE_HOME%
+MKLINK /J %~dp0ffmpeg                 %FFMPEG_HOME%
+
+ECHO ^>^> Done!
+ECHO.
+
+PAUSE