Decrypting attention mechanism: Why is Flash Attention so cool?

Background Review: What is a Big Language Model (LLM)?

Before entering the details of the attention mechanism, let’s first look at what a large language model (LLM) is. Simply put, LLM is a large-scale neural network model trained through deep learning techniques for processing and generating natural language. LLM can be applied to various tasks, such as text generation, machine translation, question and answer systems, etc.

The reason why LLM is so powerful is inseparable from its huge number of parameters and complex architecture. Encoder and Decoder are two core components of LLM, which process input data and generate outputs respectively. On this basis, the introduction of attention mechanism further improves the performance and expression ability of LLM.

Basic concepts of encoding and decoding

In large language models, the encoder and the decoder are two core components that process input data and generate outputs respectively. Generally speaking, encoders and decoders in LLM use the Transformer architecture, and the following are their basic concepts:

Encoder (Encoder)

Function: The encoder is responsible for converting an input sequence (for example, a piece of text) into an internal representation that can be understood by the decoder. It processes input data through a multi-layer neural network, allowing the model to capture context and word-to-word relationships.

Application stage: Input encoding stage. At this stage, attention mechanisms help the model understand the importance of each word in the input sequence and the relationship between them, thereby better capturing context information and understanding the meaning of the sentence.

Decoder

Function: The decoder generates an output sequence (e.g., translated text) from the internal representation generated by the encoder. When the decoder generates each output word, it refers to the input sequence and the partial output sequence that has been generated.

Application stage: Output generation stage. When generating outputs, attention mechanisms help the model focus on key parts of the input sequence, thereby generating high-quality, coherent output.

By understanding the role of encoders and decoders, readers can better understand the key role of attention mechanisms in large language models.

The stages and significance of the attention mechanism application

Input Encoding: At this stage, attention mechanisms help the model understand the importance of each word in the input sequence and the relationship between them. By calculating the attention weight of each word, the model can better capture context information and understand the meaning of the sentence.

Intermediate Layers: In each intermediate layer of the large language model, the attention mechanism plays a role in global information transmission and integration. The self-attention mechanism allows each word to “see” other words in the entire input sequence, thus providing a more comprehensive understanding of the context. This is especially important when dealing with long sequence data, because the model needs to capture the dependencies between long-distance words.

Output Generation: When generating output, attention mechanism helps the model focus on key parts of the input sequence, thereby generating high-quality, coherent output. This is particularly important for tasks such as machine translation and text generation. For example, in machine translation, the model needs to generate translations of the target language based on different parts of the source language sentence, and the attention mechanism is the key to achieving this.

Various attention mechanisms, their advantages and disadvantages and historical background

Additive Attention

Adding attention is like picking the most relevant reference materials when writing an article. It is not only calculated accurately, but also suitable for queries and keys in different dimensions. But sometimes, it can appear a bit slow to heat up. It was proposed by Bahdanau et al. in 2014, opening a new era of neural machine translation.

Scaled Dot-Product Attention

Biting attention is like you are watching a fierce basketball game, focusing only on the best players on the court. The calculation efficiency is super high, but be careful to scale the score in time to avoid falling into the quagmire of gradient disappearing or explosion. Vaswani et al. introduced it into the Transformer model in 2017, making it a mainstream choice quickly.

Self-Attention

Self-attention is like you are recalling an unforgettable trip, and every detail comes to your mind. It can capture global context information, but the demand for memory and computing resources cannot be underestimated. Also proposed by Vaswani et al. in 2017, self-attention has become the core of Transformer.

Multi-Head Attention

The bulls' attention is like you are organizing a grand party, you need to pay attention to different details at the same time. It enhances the expression ability of the model, but the computational complexity also increases. Also proposed by Vaswani et al. in 2017, multi-head attention gives the model the superpower of multitasking.

Flash Attention

Flash Attention is a superhero in the attention mechanism, which can quickly find key information and is memory efficient. Although it is a bit complicated to implement, it relies on underlying hardware optimization, making the computing speed very fast. This mechanism is designed to address performance bottlenecks in traditional attention mechanisms when dealing with long sequences.

Selection of embedded dimensions

Choosing an embedded dimension is like choosing a suitable pair of glasses for the model. It determines how detailed the semantic information the model can see. The higher the embedding dimension, the stronger the expressiveness of the model, but the computational burden will also increase. Conversely, lower embedding dimensions are computed faster, but may not capture complex semantic relationships. This important design decision needs to be trade-offs and optimized based on specific tasks and data characteristics.

A concrete example: vectorization process
Suppose we have a saying: "My deskmate and I made an appointment to play games at his house tomorrow."

1. Word Embedding
First, each word in the sentence is converted into a fixed-length vector. For example, suppose we choose the embed dimension to be 4:

"I" -> [0.1, 0.2, 0.3, 0.4]
"and" -> [0.5, 0.6, 0.7, 0.8]
"Tablemate" -> [0.9, 1.0, 1.1, 1.2]
"Have an appointment" -> [1.3, 1.4, 1.5, 1.6]
"It's" -> [1.7, 1.8, 1.9, 2.0]
"tomorrow" -> [2.1, 2.2, 2.3, 2.4]
"exist" -> [2.5, 2.6, 2.7, 2.8]
"His home" -> [2.9, 3.0, 3.1, 3.2]
"Play" -> [3.3, 3.4, 3.5, 3.6]
"game" -> [3.7, 3.8, 3.9, 4.0]

These vectors capture the semantic information of each word.

2. Positional Encoding
Since word order in sentences is important, we need to add position information to the vector representation. Assume that the position encoding vector is as follows:

Location1 -> [0.01, 0.02, 0.03, 0.04]
 Position 2-> [0.05, 0.06, 0.07, 0.08]
...

We add word embedding vectors and position coded vectors to get a new word vector:

"I" -> [0.1+0.01, 0.2+0.02, 0.3+0.03, 0.4+0.04] -> [0.11, 0.22, 0.33, 0.44]
"and" -> [0.5+0.05, 0.6+0.06, 0.7+0.07, 0.8+0.08] -> [0.55, 0.66, 0.77, 0.88]
...

3. Multi-Head Attention
In the multi-head attention mechanism, we divide the vector into multiple subspaces (heads) and calculate attention in parallel on each subspace.

For example, we use three for example, we use three subspaces to demonstrate:
Head 1: Calculate the dot product of the query and key and scale it, generate attention weights, and then weight sum the value (Value).
Head 2: Repeat the above process, but in another subspace.
Head 3: Repeat the above process again.

Assume that the calculated attention weights for each subspace are as follows:

Head 1:
"I" -> 0.3, "and" -> 0.2, "Tablemate" -> 0.4, ...

Head 2:
"I" -> 0.1, "and" -> 0.5, "Tablemate" -> 0.3, ...

Head 3:
"I" -> 0.4, "and" -> 0.3, "Tablemate" -> 0.1, ...

The weighted summing results of each subspace are as follows:

Head 1:
"I" -> [0.11*0.3, 0.22*0.2, 0.33*0.4, 0.44*0.1] -> [0.033, 0.044, 0.132, 0.044]

Head 2:
"I" -> [0.11*0.1, 0.22*0.5, 0.33*0.3, 0.44*0.1] -> [0.011, 0.11, 0.099, 0.044]

Head 3:
"I" -> [0.11*0.4, 0.22*0.3, 0.33*0.1, 0.44*0.2] -> [0.044, 0.066, 0.033, 0.088]

"I" -> [0.033, 0.044, 0.132, 0.044, 0.011, 0.11, 0.099, 0.044, 0.044, 0.066, 0.033, 0.088]

                      +-------------------------+
                      | Enter a sentence：              |
                      | "My deskmate and I made an appointment to play games at his house tomorrow"  |
                      +-------------------------+
                                |
                                v
                  +--------------------------------+
                  | Word embedding（Word Embedding）       |
                  | Convert each word into a fixed length vector     |
                  +--------------------------------+
                                |
                                v
                  +--------------------------------+
                  | Position encoding（Positional Encoding）|
                  | Word vectors that add position information           |
                  +--------------------------------+
                                |
                                v
                  +---------------------------+
                  | Bulls' attention（Multi-Head Attention）  |
                  | Multiple subspaces compute attention in parallel    |
                  +---------------------------+
                                |
                                v
                  +--------------------------------------+
                  | Final vector representation                       |
                  | The result of vectors of each word combined with multiple attention heads |
                  +--------------------------------------+