This blog post is going to be a part of a series of posts related to LLM concepts. I’m writing it primarily for myself so I have a place to come back to whenever I need a refresher. Frankly I got the topic inspiration from one of Daniel Lee‘s LinkedIn post where he posted a list of questions asked in Google, Nvidia, Meta, Intuit and Adobe’s interview questions. I figured I forgot most of these concepts, I should do a refresher course myself and the best way to do it is teach myself again.
I will try and make this blog post as readable as possible without using unnecessary technical jargons because I admit that I myself need to dig through documentations whenever I come across technical terms I don’t regularly use. I hope this series makes these concepts a little easier to understand.
Ill cover the transformer architecture with attention mechanism in this post.
Transformer Architecture: Training
Transformer Architecture is a deep learning model. In attention mechanism, the goal is to create a relationship between the sequence of words. It basically lets us predict what word comes next in a sentence. Input text is broken up into tokens. Consider a token to be a word (sometimes they are part of the word and not the full word). Tokens are converted to vectors in high dimensional space. The tokens in the high dimensional space corresponds to a certain semantic meaning. For example if the distance between 2 words are close, most likely they can mean the same thing or at least refer to similar things, for example car and truck refers to vehicles.
Now the question is how are the distances similar? I mean how does the model know that they belong closely together? This is because the words appear in similar context millions of times during training. Below I briefly describe what happens in training phase.
Step 1: of training the model starts with random vectors
| Word | Initial Vector |
| car | [0.12, -0.23, 0.45 ……] |
| truck | [0.32, 0.54, -0.12 ……] |
| ship | [-0.33, -0.28, 0.78 ……] |
| airplane | [0.65, -0.88, -0.29 ……] |
Here all the words are vehicles but none of the words are close in the high dimensional space.
Step 2: the model reads billions of sentences
The car is parked outside the house.
The truck is parked outside the warehouse.
The ship is docked outside the port.
The airplane is parked at the airport gate
She traveled by car to the next city.
The goods were delivered by truck.
The cargo was transported by ship.
The passengers flew by airplane.
Although a car, truck, ship, and airplane look very different, they all share concepts like transportation, travel, moving people, and moving goods.
Step 3: Training adjusts the vectors
The model is trained to predict the next or missing token in a sequence. Every time it makes a mistake, it calculates how wrong its prediction was using a loss function. It then uses gradient descent to update its parameters in a way that reduces this loss.
This process is repeated billions of times across enormous amounts of text. With each iteration, the model gradually learns which words and concepts are related. As a result, tokens that frequently appear in similar context such car, truck, ship, and airplane end up with embeddings that are close together in distance in the high dimension.
Over time, these embeddings naturally move closer together in the high-dimensional vector space, while unrelated tokens move farther apart. This is how the model learns semantic relationships between words without anyone explicitly telling it that a car and a ship are both vehicles.
Attention Mechanism
The above description is how the training process works in the transformer architecture. Now we will know what about it is the attention mechanism.
Now we will use this example: The car is parked outside the house to explain transformer layer 1
The sentence is first tokenized
The | car | is | parked | outside | the | house
Each token is converted into an embedding
“The” → [0.12, -0.43, …]
“car” → [0.91, 0.27, …]
“is” → [0.08, -0.15, …]
“parked” → [0.55, 0.73, …] …
At this point, the embedding for car simply represents the general concept of a car. It doesn’t yet know that this particular car is parked outside a house.
Now each token pays attention to every token. For example, car is looking at ‘parked’, ‘outside’, ‘house’. You might be wondering about words like the and is. The model looks at those too, but it doesn’t necessarily pay equal attention to every word. During training, it learns that content words such as car, parked, outside, and house often contribute more to understanding the meaning of the sentence, while function words like the, is, a, and an mainly provide grammatical structure. As a result, in many contexts the model assigns more attention to the content words. However, this isn’t a fixed rule the attention weights are calculated dynamically for each sentence, so even words like is can receive high attention when they’re important to the meaning.
After the attention scores are calculated based on the weights of each of the token, the embedding for car is updated by combining information from the other tokens. You can think of it as enriching the original embedding with the information it has gathered from the words it paid attention to. Instead of representing only the generic concept of a car, the new embedding now represents a car that is parked outside a house.
The same process happens for every other token. For example, the embedding for parked is also updated by looking at car, outside, house, and the remaining words in the sentence. By the end of the first transformer layer, every token has a richer, more context-aware representation than it had initially.
This process is repeated in the next transformer layer, and then again in the following layers. Each layer takes the updated embeddings from the previous layer and refines them even further, allowing the model to build an increasingly deeper understanding of the sentence.
An example of the calculation for each token to update its embedding in each transformer layer will look something like this.
| Token | Embedding | Attention weight |
| the | [0.34, 0.23….] | 0.02 |
| car | [0.21, 0.12….] | 0.3 |
| is | [0.42, 0.88….] | 0.02 |
| parked | [0.48, -0.32….] | 0.35 |
| outside | [0.62, -0.13….] | 0.35 |
| the | [-0.98, 0.21….] | 0.02 |
| house | [0.12, 0.29….] | 0.27 |
the new embedding for the car in the first transformer layer will be
(0.3 x car) + (0.02 x the) + (0.02 x is )+ (0.35 x parked) + (0.35 x outside) + (0.02 x the) + (0.27 x house)
If we calculate it
0.3 x [0.21, 0.12….] + 0.02 x [0.34, 0.23….] + 0.02 x [0.42, 0.88….] + 0.35 x [0.48, -0.32….] + 0.35 x [0.62, -0.13….] + 0.02 x [0.34, 0.23….] + 0.27 x [0.12, 0.29….]
So, after this the original embedding of car [0.21, 0.12….] will be updated to a new embedding. Same will happen for the other tokens in the same layer. Tt will move on to the next layer and start the same process again to update the embedding of each of the tokens.
Wrapping up
I tried as best as I can to explain the mechanism without using technical jargon that would send you or myself down a rabbit hole of searching for more definitions just to understand this explanation. I hope you got some insights and if you would like more articles like this please subscribe and stay tuned 🙂
Leave a Reply