controlnet inpainting supported

2023-04-18 03:33:23 +08:00 · 2023-04-18 03:33:23 +08:00 · e76a212314
parent 5e1b28c221
commit e76a212314
4 changed files with 37 additions and 39 deletions
--- a/README.md
+++ b/README.md
@ -7,6 +7,7 @@ This extension aim for helping [stable diffusion webui](https://github.com/AUTOM
 - `2023/04/12`: [Feature] Mask expansion released by [@jordan-barrett-jm](https://github.com/jordan-barrett-jm)!
 - `2023/04/15`: [Feature] [GroundingDINO](https://github.com/IDEA-Research/GroundingDINO) support released! Check [Note about GroundingDINO](https://github.com/continue-revolution/sd-webui-segment-anything#note-about-groundingdino), [How to Use](#how-to-use) and [Demo](#demo) for more detail.
 - `2023/04/15`: [Feature] API support released by [@jordan-barrett-jm](https://github.com/jordan-barrett-jm)! Check [API Support](#api-support) for more detail.
+- `2023/04/18`: [Feature] [ControlNet V1.1](https://github.com/lllyasviel/ControlNet-v1-1-nightly) inpainting support released! Note that you **must** update [ControlNet extension](https://github.com/Mikubill/sd-webui-controlnet) to the most up-to-date version to use it. Check [How to Use](#how-to-use) for more detail. ControlNet demo will be released after semantic segmentation is supported.

 ## Plan

@ -15,7 +16,7 @@ Thanks for suggestions from [GitHub Issues](https://github.com/continue-revoluti
 - [ ] Support color inpainting as mentioned in [#21](https://github.com/continue-revolution/sd-webui-segment-anything/issues/22)
 - [ ] Support automatic mask generation for hierarchical image segmentation and SD animation
 - [ ] Support semantic segmentation for batch process, ControlNet segmentation and SD animation
- [ ] Connect to [ControlNet](https://github.com/Mikubill/sd-webui-controlnet) inpainting and segmentation
+- [ ] Connect to [ControlNet](https://github.com/Mikubill/sd-webui-controlnet) segmentation
 - [ ] Support WebUI older commits (e.g. `a9fed7c364061ae6efb37f797b6b522cb3cf7aa2`)

 Not all plans may ultimately be implemented. Some ideas might not work and be abandoned. Support for old commits has low priority, so I would encourage you to update your WebUI as soon as you can.
@ -54,18 +55,22 @@ To give you a reference, [vit_h](https://dl.fbaipublicfiles.com/segment_anything

 ### Step 3:

- Launch webui and switch to img2img mode.
-
 #### Single Image
 - Upload your image
 - Optionally add point prompts on the image. Left click for positive point prompt (black dot), right click for negative point prompt (red dot), left click any dot again to cancel the prompt. You must add point prompt if you do not wish to use GroundingDINO.
 - Optionally check `Enable GroundingDINO`, select GroundingDINO model you want, write text prompt and pick a box threshold. You must write text prompt if you do not wish to use point prompts. Note that GroundingDINO models will be automatically downloaded from [HuggingFace](https://huggingface.co/ShilongLiu/GroundingDINO/tree/main). If your terminal cannot visit HuggingFace, please manually download the model and put it under `${sd-webui-sam}/models/grounding-dino`.
 - Optionally enable previewing GroundingDINO bounding box and click `Generate bounding box`. You must write text prompt to preview bounding box. After you see the boxes with number marked on the left corner, uncheck all the boxes you do not want. If you uncheck all boxes, you will have to add point prompts to generate masks.
 - Click `Preview Segmentation` button. Due to the limitation of SAM, if there are multiple bounding boxes, your point prompts will not take effect when generating masks.
- Choose your favorite segmentation and check `Copy to Inpaint Upload`
- Optionally check `Expand Mask` and specify the amount, then click `Update Mask`
+- Choose your favorite segmentation.
+- Optionally check `Expand Mask` and specify the amount, then click `Update Mask`.
+
+##### img2img Inpainting
+- Check `Copy to Inpaint Upload`. Note that you **must** be at img2img tab to use this functionality.
 - Click `Switch to Inpaint Upload` button. There is no need to upload another image or mask, just leave them blank. Write your prompt, configurate and click `Generate`.

+##### ControlNet Inpainting
+- Check `Copy to ControlNet Inpaint` and select the ControlNet panel for inpainting if you want to use multi-ControlNet. You can be either at img2img tab or at txt2img tab to use this functionality.
+
 #### Batch Process
 - Choose your SAM model, GroundingDINO model, text prompt, box threshold and mask expansion amount. Enter the source and destination directories of your images. **The source directory should only contain images**.
 - `Output per image` gives you a choice on configurating the number of masks per bounding box. I would highly recommend choosing 3, since some mask might be wierd.
--- a/javascript/sam.js
+++ b/javascript/sam.js
@ -80,7 +80,7 @@ function samCreateDot(sam_image, image, coord, label) {
    const y = coord.y;
    const realCoord = samGetRealCoordinate(image, coord.x, coord.y);
    if (realCoord[0] >= 0 && realCoord[0] <= image.naturalWidth && realCoord[1] >= 0 && realCoord[1] <= image.naturalHeight) {
-        const isPositive = label == (samTabPrefix() + "sam_positive");
+        const isPositive = label == (samTabPrefix() + "positive");
        const circle = document.createElement("div");
        circle.style.position = "absolute";
        circle.style.width = "10px";
@ -95,16 +95,16 @@ function samCreateDot(sam_image, image, coord, label) {
        circle.addEventListener("click", e => {
            e.stopPropagation();
            circle.remove();
-            if (gradioApp().querySelectorAll("." + samTabPrefix() + "sam_positive").length == 0 &&
-                gradioApp().querySelectorAll("." + samTabPrefix() + "sam_negative").length == 0) {
-                disableRunButton();
+            if (gradioApp().querySelectorAll("." + samTabPrefix() + "positive").length == 0 &&
+                gradioApp().querySelectorAll("." + samTabPrefix() + "negative").length == 0) {
+                samChangeRunButton();
            } else {
                if (samIsRealTimePreview()) {
                    samImmediatelyGenerate();
                }
            }
        });
-        enableRunButton();
+        samChangeRunButton();
        if (samIsRealTimePreview()) {
            samImmediatelyGenerate();
        }
@ -114,7 +114,7 @@ function samCreateDot(sam_image, image, coord, label) {
 function samRemoveDots() {
    const sam_image = gradioApp().getElementById(samTabPrefix() + "input_image");
    if (sam_image) {
-        ["." + samTabPrefix() + "sam_positive", "." + samTabPrefix() + "sam_negative"].forEach(cls => {
+        ["." + samTabPrefix() + "positive", "." + samTabPrefix() + "negative"].forEach(cls => {
            const dots = sam_image.querySelectorAll(cls);
    
            dots.forEach(dot => {
@ -152,15 +152,15 @@ function submit_sam() {
    let res = create_submit_sam_args(arguments);
    let positive_points = [];
    let negative_points = [];
-    const sam_image = gradioApp().getElementById(samTabPrefix() + "sam_input_image");
+    const sam_image = gradioApp().getElementById(samTabPrefix() + "input_image");
    const image = sam_image.querySelector('img');
-    const classes = ["." + samTabPrefix() + "sam_positive", "." + samTabPrefix() + "sam_negative"];
+    const classes = ["." + samTabPrefix() + "positive", "." + samTabPrefix() + "negative"];
    classes.forEach(cls => {
        const dots = sam_image.querySelectorAll(cls);
        dots.forEach(dot => {
            const width = parseFloat(dot.style["left"]);
            const height = parseFloat(dot.style["top"]);
-            if (cls == "." + samTabPrefix() + "sam_positive") {
+            if (cls == "." + samTabPrefix() + "positive") {
                positive_points.push(samGetRealCoordinate(image, width, height));
            } else {
                negative_points.push(samGetRealCoordinate(image, width, height));
@ -196,8 +196,8 @@ function dinoCanSubmit() {
 }

 function samCanSubmit() {
-    return (gradioApp().querySelectorAll("." + samTabPrefix() + "sam_positive").length > 0 ||
-        gradioApp().querySelectorAll("." + samTabPrefix() + "sam_negative").length > 0)
+    return (gradioApp().querySelectorAll("." + samTabPrefix() + "positive").length > 0 ||
+        gradioApp().querySelectorAll("." + samTabPrefix() + "negative").length > 0)
 }

 function samHasImageInput() {
@ -209,22 +209,26 @@ function dinoOnChangePreviewBoxesSelection() {
    samChangeRunButton(arguments[0].length > 0)
 }

-prevImg = null
+samPrevImg = {
+    "txt2img_sam_": null,
+    "img2img_sam_": null,
+}

 onUiUpdate(() => {
    const sam_image = gradioApp().getElementById(samTabPrefix() + "input_image")
    if (sam_image) {
        const image = sam_image.querySelector('img')
-        if (image && prevImg != image.src) {
+        if (image && samPrevImg[samTabPrefix()] != image.src) {
+            console.log("remove points 1")
            samRemoveDots();
-            prevImg = image.src;
+            samPrevImg[samTabPrefix()] = image.src;

            image.addEventListener("click", event => {
                const rect = event.target.getBoundingClientRect();
                const x = event.clientX - rect.left;
                const y = event.clientY - rect.top;

-                samCreateDot(sam_image, event.target, { x, y }, samTabPrefix() + "sam_positive");
+                samCreateDot(sam_image, event.target, { x, y }, samTabPrefix() + "positive");
            });

            image.addEventListener("contextmenu", event => {
@ -233,22 +237,24 @@ onUiUpdate(() => {
                const x = event.clientX - rect.left;
                const y = event.clientY - rect.top;

-                samCreateDot(sam_image, event.target, { x, y }, samTabPrefix() + "sam_negative");
+                samCreateDot(sam_image, event.target, { x, y }, samTabPrefix() + "negative");
            });

            const observer = new MutationObserver(mutations => {
                mutations.forEach(mutation => {
                    if (mutation.type === 'attributes' && mutation.attributeName === 'src' && mutation.target === image) {
+                        console.log("remove points 2")
                        samRemoveDots();
-                        prevImg = image.src;
+                        samPrevImg[samTabPrefix()] = image.src;
                    }
                });
            });

            observer.observe(image, { attributes: true });
        } else if (!image) {
+            console.log("remove points 3")
            samRemoveDots();
-            prevImg = null;
+            samPrevImg[samTabPrefix()] = null;
        }
    }

--- a/scripts/sam.py
+++ b/scripts/sam.py
@ -493,10 +493,7 @@ class Script(scripts.Script):
                p.init_images = [input_image]
                p.image_mask = image_mask
            if enable_copy_cn_inpaint and cn_num < self.max_cn_num():
-                image_mask_np = np.array(image_mask.convert("L"))
-                input_image_np = np.array(input_image).astype(np.int32)
-                input_image[image_mask_np > 127] = -255
-                self.set_p_value(p, 'control_net_input_image', cn_num, input_image_np)
+                self.set_p_value(p, 'control_net_input_image', cn_num, {"image": input_image, "mask": image_mask.convert("L")})
        
    def set_p_value(self, p: StableDiffusionProcessing, attr: str, idx: int, v):
        value = getattr(p, attr, None)
--- a/style.css
+++ b/style.css
@ -1,19 +1,9 @@
-#sam_run_button {
+#txt2img_sam_run_button, #img2img_sam_run_button, #txt2img_sam_dino_run_button, #img2img_sam_dino_run_button {
    position: absolute;
    display: none;
    width: 100%;
 }

-#sam_run_button:hover {
+#txt2img_sam_run_button:hover, #img2img_sam_run_button:hover, #txt2img_sam_dino_run_button:hover, #img2img_sam_dino_run_button:hover {
   background:  #b4c0cc;
 }
-
-#dino_run_button {
-    position: absolute;
-    display: none;
-    width: 100%;
-}
-
-#dino_run_button:hover {
-   background:  #b4c0cc;
-}