Location>code7788 >text

ComfyUI Plugin: ComfyUI-BrushNet Node

Popularity:462 ℃/2024-08-05 12:01:02

Preface:

Learning ComfyUI is a constant battle, and ComfyUI-BrushNet is the most recent local repainting node, which contains two main nodes, BrushNet and Powerpaint, of which BrushNet has two versions, SD1.5 and SDXL, and PowerPaint is only available for 1.5 models, learn the plugin, you can complete the local repainting of images and products to change the background and other multiple workflows. Wish you all learn well and become a master of ComfyUI soon!

catalogs

I. BrushNet Loader Node

II. BrushNet nodes

III. Blend Inpaint nodes

IV. Cut For Inpaint Node

V. PowerPaint nodes

I. BrushNet Loader Node

This node focuses on loading BrushNet models for use in subsequent image processing workflows. By loading the BrushNet model, its powerful image processing capabilities can be utilized to perform a variety of detailed editing and enhancement tasks.

1

Important parameters:

dtype → This parameter sets the model's runtime data type

Note: It is recommended to set the dtype to float32 to run, if this is float16, then when the CUDA memory is not enough, part of the data will be put into the CPU to perform the model calculation, but the CPU doesn't support float16 precision data, which will generate an error message.

The node is currently offered for loading in three models

The first is the PowerPaint model: as shown in the figure, the PowerPaint model must be used with a large SD model of version 1.5 with the Clip model required for PowerPaint loaded. The function of this model is to remove the areas of the image that are covered by the mask.

2

The second is the BrushNet model, which uses the random_mask model. In my experiments, I used the SDXL large model. The image below shows the effect of a localized repair using the random_mask model. It can be seen that the final repaired image is not filled strictly according to the contents of the mask, for example, the size of the bottle is significantly smaller than the mask. Therefore, the model is mainly used to randomly select masks for restoration.

3

The third is the segmentation model used by BrushNet. As can be seen in the figure, the final generated image is drawn strictly to the masked area, and the size of the bottles exactly matches the mask. However, it is mentioned in the original BrushNet paper that the segmentation model may introduce potential inaccuracies in terms of interpolation operations, as the operation of resizing the mask to match the potential space may lead to potential inaccuracies". That is, when the mask information is coded error via VAE.4

Usage Scenarios:

- Image Editing: Utilize the BrushNet model for detailed image editing tasks such as denoising, repairing, and enhancing.

- Image Enhancement: Enhance the detail and quality of images with BrushNet models to improve visualization.

- Automated Processing: Efficient and accurate image processing using BrushNet models in an automated image processing process.

By using the BrushNet Loader node, efficient BrushNet model loading and application can be realized in the image processing workflow to enhance the accuracy and effect of image processing.

II. BrushNet nodes

This node focuses on using the BrushNet model for various image processing, including denoising, restoration, enhancement, and more. By configuring and using the BrushNet model, high-quality image processing results can be achieved.

5

Important parameters:

mask → indicates the mask you want to upload ** if the mask is not aligned with the image, the result may be poor**

scale → directly translated means scale, which is the threshold for controlling the size, the model encodes the raw data and adds it to the unet before multiplying it with scale to control the intensity of the model's action.

Example: The following figure shows the BrushNet batch repainting method

6

Example: The following figure shows the basic repainting method of BrushNet

7

Usage Scenarios:

- Image denoising: Use the BrushNet model to remove noise from images and improve image quality.

- Image Repair: Repair blemishes and damaged areas in an image with the BrushNet model.

- Image Enhancement: Enhances the details and visual effects of an image to make it clearer and more appealing.

- Automated Processing: Efficient and accurate image processing using BrushNet models in an automated image processing process.

By using BrushNet nodes, it is possible to realize efficient BrushNet model applications in image processing workflows to enhance the accuracy and effectiveness of image processing.

III. Blend Inpaint nodes

This node specializes in image restoration and filling. By using advanced image restoration algorithms, missing parts of an image can be filled in, or new image content can be seamlessly blended into an existing image.

8

Important parameters:

inpaint → localized image after repainting

original → Original Image

origin → Cropped data from the original image

kernel → Setting the degree of edge blending for pasted images

sigma → set the transparency of the image

Usage Scenarios:

Image Restoration: Repairing damaged or missing parts of an image in image processing.

Image Fill: Used in scenes where new image content needs to be seamlessly blended into an existing image.

Automated Processing: Efficient image processing through image restoration and filling algorithms in an automated image processing flow.

By using the Blend Inpaint node, you can achieve efficient image restoration and filling in your image processing workflow, improving the accuracy and effectiveness of image processing.

IV. Cut For Inpaint Node

This node focuses on preparing image data for use in subsequent repair and fill tasks. By cutting and processing specific areas of the image, the parts of the image to be repaired or filled and their corresponding masks can be generated.

9

Important parameters:

image → input original image

mask → Enter the mask area to be redrawn

width → set cropping width

height → set crop height

Example: The following figure shows the combined use of the Cut For Inpaint node and the Blend Inpaint node, mainly used in the case where our original image is very large and the area we want to redraw is very small, in this case, we will zoom in to fix the corresponding area to enable the model to better understand the context and thus perform a more integrated redraw.

10

Usage Scenarios:

- Image Repair Preparation: In image processing, preparing the portion of the image that needs to be repaired or filled to provide suitable data for subsequent processing nodes.

- Image Fill Prep: Prepares the image area to be filled in a scene where new image content needs to be seamlessly blended into an existing image.

- Automated processing: Efficient area preparation and optimization by cutting and processing image areas in an automated image processing flow.

By using the Cut For Inpaint node, efficient image region preparation can be realized in the image processing workflow to provide suitable input data for subsequent restoration and filling tasks, thus enhancing the precision and effect of image processing and meeting various complex image processing needs.

V. PowerPaint nodes

This node focuses on complex image restoration and enhancement tasks, enabling high-quality image editing through the use of advanced image processing algorithms.

11

Important parameters:

clip → PowerPaint has a corresponding CLIP model

fitting → PowerPaint fit

function → PowerPaint function

There are five options in function, text guided, shape guided, object removal, context aware and image outpainting, which correspond to the functions "use text as a guide", "use shape as a guide", "object removal", "context aware" and "image outpainting" respectively. ", "shape guided", "object removal", "context aware" and "image outpainting ".

Example: The workflow shown in the following figure is to expand the original image, select the image outpainting option, you need to pad the image before expansion.

12

Usage Scenarios:

- Image Repair: Repairing damaged or missing parts of an image in image processing.

- Image Enhancement: Make images clearer and more appealing by enhancing their details and visual effects.

- Image Fill: Used in scenes where new image content needs to be seamlessly blended into an existing image.

- Automated Processing: Efficient image processing through image restoration and enhancement algorithms in an automated image processing process.

The PowerPaint node is a powerful image editing tool specialized for complex image repair and enhancement tasks. This node is designed to perform highly accurate image editing by providing advanced image processing algorithms and flexible configuration options.

**To go beyond oneself is to strive for excellence. Perseverance is the key to success. **