Preface:
Learning ComfyUI is a protracted battle, ComfyUI_Noise is a plug-in library to control the noise in ComfyUI, the library can complete the image noise backward push, and through the sampling and then return to the original image in a virtually lossless manner, through the use of the library can better help the image to restore the original appearance, very suitable for generating video in the field of the character as a figure to use. Wish you all learn well, as soon as possible to become a master of ComfyUI!
catalogs
I. Installation method
II. BNK_NoisyLatentImage node
III. BNK_SlerpLatent node
IV. BNK_GetSigma node
V. Inject Noise node
VI. BNK Unsampler nodes
I. Installation method
Inside the ComfyUI home directory type CMD and enter.
Type git clone xxx in the CMD command line that pops up to start the download.
Enter the following line of code in the terminal to start the download
git clone /BlenderNeko/ComfyUI_Noise.git
II. BNK_NoisyLatentImage node
This node focuses on generating latent images with noise in the latent space. This is particularly useful in image generation tasks, such as in Generative Adversarial Networks (GANs) or other latent space-based generative models, where the introduction of noise can increase the diversity of images or enhance the robustness of the model.
Important parameters:
source → where the noise is generated - selectable CPU or GPU
Example:The differences between different images are compared through the Image Compare (mtb) node, which compares the noisy graph generated by the CPU and the noisy graph generated by the GPU, respectively, and denoises the raw graph on top of that graph, and finally compares the differences between the two raw graphs.
From the results it can be seen that there is a difference between the noise generated by the CPU and the GPU, but the difference is not small enough to completely affect the quality or composition of the final image, so it can be alternated in terms of selection.
Usage Scenarios:
- Image Generation Enhancement: In the image generation process, the diversity of images is increased by introducing noise to avoid the generated images being too similar.
- Model Robustness Test: The robustness and stability of the model is evaluated by adding noise when testing the generated model.
- Data augmentation: generating diverse training data in the latent space to enhance the generalization ability of the model.
By using the BNK_NoisyLatentImage node, efficient noise addition can be implemented in the image generation and processing workflow, enhancing the diversity of the generated images and the robustness of the model.
III. BNK_SlerpLatent node
This node focuses on performing spherical linear interpolation (Slerp) in the latent space, generating intermediate vectors between the two latent vectors, thus enabling smooth transitions in image generation.
Important parameters:
factor → latent space image mixing ratio, can be interpreted as transparency
Example:
Usage Scenarios:
- Image Generation Transition: generates a sequence of smooth transitions from one image to another by interpolating between two potential vectors in an image generation task.
- Latent space exploration: exploring transitions between different vectors in latent space through interpolation to understand the latent space structure of the generated model.
- Animation generation: By generating multiple interpolation points, it is possible to create smooth animation effects from one image to another.
By using the BNK_SlerpLatent node, smooth interpolations and transitions in the latent space can be implemented in the image generation workflow, exploring the latent space structure and creating a smooth and coherent image generation effect.
IV. BNK_GetSigma node
This node is used to extract or calculate the standard deviation (Sigma) of latent variables in the latent space. The standard deviation (Sigma) is very important in image generation and processing tasks, especially when dealing with noise or latent variables, and knowing and adjusting the Sigma value can affect the quality and characterization of the generated image.
Important parameters:
model → select the model to predict
Example:The following figure shows the initial usage of this node, the understanding is not deep enough to come up with a better way of using it, and it may require a deeper study to discover the real meaning of this node.
Usage Scenarios:
- Latent space analysis: analyze the distributional properties of the latent space and understand the behavior of the model by calculating the Sigma values of the latent variables.
- Noise Adjustment: in Generative Adversarial Networks (GANs) or other latent variable models, noise is adjusted according to the Sigma value to control the characteristics of the generated image.
- Image processing optimization: Sigma values are used to optimize the parameter settings of image processing algorithms to improve the quality of image generation.
By using the BNK_GetSigma node, Sigma values in the latent space can be efficiently computed and utilized in the image generation and processing workflow, thus improving the control of the model and the quality of image generation.
V. Inject Noise node
This node focuses on introducing random noise into an image or latent vector. By configuring the intensity and type of noise, the injection of noise can be flexibly controlled to affect the characteristics of the generated or processed image.
Important parameters:
latents → empty latent space images
mask → injected noise mask area
This node can be used in conjunction with the previous node, see the diagram of the previous node for an example.
Usage Scenarios:
- Generated image diversity: increase the diversity of the generated image by injecting noise to make the generated image richer.
- Model Robustness Test: Injecting noise into an image or latent vector to test the model's performance in handling noisy data.
- Simulate real-world scenarios: simulate real-world uncertainties through noise injection to improve the generalization ability of the model when training or testing the model.
By using the Inject Noise node, noise can be effectively controlled and utilized in image processing and generation tasks to increase data diversity, test model robustness, and enhance the realism and richness of generated images.
VI. BNK Unsampler nodes
This node focuses on performing backsampling operations in the process of generating images. It can convert the sampled or processed latent space representation back into an image or other form of output, which is useful for the training and inference process of deep learning models especially Generative Adversarial Networks (GANs) and others.
Important parameters:
model → need to select the corresponding model for noise prediction
cfg → 1 is recommended for noise backpropagation, and 1 is also recommended for raw maps
Example:As shown in the figure, we first upload an original image, and then load a Vincennes workflow, through the node for noise prediction, and then use the predicted latent space image, using the same configuration, such as VAE, sampler, conditional information, etc. for noise removal, and ultimately generate the same image as the original image, through the IMAGE comparison node, you can see that no difference occurs.
Usage Scenarios:
- Potential space transformation: converting a representation in potential space (e.g., sampled potential vectors) back into an image or other form of output for further analysis or presentation.
- Inverse operations during image generation: in models such as Generative Adversarial Networks (GANs) or Variational Auto-Encoders (VAEs), inverse sampling techniques are used to recover or generate images.
- Complex Image Processing Workflow: As part of an image generation or processing workflow, backsampling is a key step from potential representation to actual image generation.
By using BNK Unsampler nodes, it is possible to realize the conversion from potential spatial representations to actual outputs in image generation and processing tasks, to complete complex image generation workflows, and to meet the needs of various deep learning applications.
**To go beyond oneself is to strive for excellence. Perseverance is the key to success. **