Location>code7788 >text

LoRA's big model fine-tuning boon

Popularity:483 ℃/2024-09-04 09:31:23

LoRA models are small Stable Diffusion models that make minor adjustments to checkpoint models. They are typically 10 to 100ths of the size of a checkpoint model. Because of their small size and effectiveness, LORA models are used to a high degree.

This is a tutorial for beginners who have never used LoRA models before. You will learn what LoRA models are, where to find them and how to use them in the AUTOMATIC1111 GUI. Then, you will find some demos of LoRA models at the end of the article.

What is the LoRA model?

LoRA (Low-Rank Adaptation) is a training technique for fine-tuning Stable Diffusion models.

Although we already have training techniques like Dreambooth and text inversion. So what's the point of LoRA?

LoRA can actually be seen as a compromise between Dreambooth and Text Reversal embeddings.Dreambooth is powerful but the model files can be quite large (2-7 GB). Text inversion is very small (~100 KB) but can do less.

LoRA is somewhere in between. It has a more manageable file size (2-200 MB) and pretty good trainability.

Because checkpoint is relatively large, hard disk space storage can be a problem. And LoRA is an excellent solution to the storage problem.

Like text inversion.You can't use the LoRA model alone. It must be used in conjunction with a model checkpoint file. the LoRA changes the style by making small changes to the companion model file.

LoRA is a great way to customize AI art models without taking up too much local storage space.

How does LoRA work?

LoRA makes minor modifications to the most critical parts of the Stable Diffusion model:cross-cutting attention span. The researchers found that fine-tuning this part of the model was enough to achieve good training. The cross-attention layer is the yellow part of the Stable Diffusion model architecture below.

交叉注意力层

The weights of the cross-attention layers are expressed in terms ofmatricesThe matrix is just a bunch of numbers arranged in columns and rows, just like on an Excel spreadsheet. Matrices are just a bunch of numbers arranged in columns and rows, just like on an Excel spreadsheet.The LoRA model fine-tunes the model by adding its own weights to these matrices.

If LoRA models need to store the same number of weights, how can their files be smaller?The trick with LoRA is to decompose the matrix into two smaller (low-rank) matrices.By doing so, it can store fewer numbers. Let's go through the following example.

Suppose the model has a matrix with 100 rows and 100 columns. That's 10,000 numbers (100x100) to be stored in the model file.LoRA breaks the matrix into a 1000x2 matrix and a 2x100 matrix. That's only 400 numbers (100 x 2 + 2 x 100, and the LoRA model will be reduced even more if the matrix is large enough. This is why LoRA files are much smaller.

image-20240703105705024

In this example, the matrixorderlinessis 2. It is much lower than the original size, so they are calledlow-rank matrix (math.). The rank can be as low as 1.

This kind of operation is bound to result in missing data, but in the case of the cross-attention layer, these losses are harmless. So Lora's approach is feasible.

Where do I find LoRA models?

Civitai

image-20240703110017448

The top place to look for LoRA is Civitai. this site hosts a large collection of LoRA models. Apply the LORA filter to view only LoRA models. Most of the models in there though are: female portraits, anime, realist illustration styles, and more.

Hugging Face

image-20240703110326192

Hugging Face is another good source for LoRA libraries. You'll find a wider variety of LoRA models. But there may not be as many LoRA models there as at Station C, and they are less intuitive. Because there is no image preview.

How do I use LoRA?

Here I will describe how to use the LoRA model in the AUTOMATIC1111 Stable Diffusion GUI.

AUTOMATIC1111 supports LoRA natively. you don't need to install any extensions.

Step 1: Install the LoRA model

To install the LoRA model in the AUTOMATIC1111 webui, place the model files in the following folder.

stable-diffusion-webui/models/Lora

Step 2: Use LoRA in prompts

To add LoRA with weights to the AUTOMATIC1111 Stable Diffusion WebUI, use the following syntax in the prompt or negative prompt:

<lora: name: weight>

nameis the name of the LoRA model. It can be different from the file name.weightis the weight applied to the LoRA model. The default value is 1. Setting it to 0 will disable the model.

So how do you know what Lora's name is?

We don't actually need to manually type in the lora name, we just need to click on the lora tab below:

image-20240703111227834

You should see a list of installed LoRA models. Click on the model you want to use.

The LoRA phrase will be inserted into the prompt.

That's it!

Precautions for using LoRA

You can.Adjustment multiplierto enhance or adjust the effect. Setting the multiplier to 0 will disable the LoRA model. You can adjust the style effect between 0 and 1.

Some LoRA models are trained using Dreambooth. You need to include aTrigger KeywordsOnly then you can use the LoRA model. You can find the trigger keyword on the model page.

Similar to embedding, you can use bothMultiple LoRA models. You can also use them with embeds.

In AUTOMATIC1111, the LoRA phrase is not part of the prompt. It is removed after the LoRA model is applied. This means that you can't use them with a cue syntax like [keyword1:keyword2: 0.8].

Practical LoRA modeling

Here are some Lora that are more useful in real applications.

add_detail

image-20240703112744179

Who doesn't want more detail in an AI image? The Detail Adjuster allows you to increase or decrease the detail in your images. Now you can adjust the amount of detail you want.

Use positive LoRA weights to increase detail and negative weights to decrease detail.

add_saturation

This lora can add some saturation to the picture with the following effect:

image-20240703113008727

add_brightness

This lora can be used to control the brightness of the image:

image-20240703134833039

summarize

LoRA models are small modifiers for checkpoint models. You can easily use them in AUTOMATIC1111 by including a phrase in the prompt.

I'll show you how to train a LoRA model yourself in a future post.

Click on me for more highlights.