The transformer works in inference; its nature is autoregressive
During training, the model learns its parameters. During inference, it uses those learned parameters to generate the next token.

Every request passes through these stages, and inference optimizations are done around it, whether it's memory optimization or compute optimization

I ran Phi-4 Mini Reasoning locally on my laptop using an NVIDIA GeForce GTX 1050 Ti (4 GB VRAM). While the model used around 3.0 GB of VRAM during inference, I never encountered an OOM error. Instead, the practical limitation was the 4096-token context length.

  1. Prefill

  2. Decode

Prefill [Process the Prompt]

Imagine you sent the Prompt "hey, what is the continuous batching"
It doesn't immediately generate the token, as we see in any chatbot; when we pass the user prompt, it generates the output. In actuality, first, these user prompt KV pairs are calculated in parallel, and then the first tokens are generated. We also assume it as the ingestion or thinking phase, just all the tokens ingested at a time

By definition, we can say that prefill is where it user entire prompt gets encoded, builds up its understanding, prepares the response, and then generates the first token

figure 1.1 Prefill Stage

The prefill phase has higher Arithmetic intensity, as here, the chunk of data is processed, and it performs thousands of calculations. In calculations, we have weights that are frozen at inference that we do MatMul with the Prompt Tokens, so that it is also called "compute bound" or "FLOP-bound". Here, the utilization of the GPU is 100%

figure 1.2 Prefill evals

As shown in Figure 1.2, Phi-4 Mini Reasoning run processed a 50-token prompt during the prefill phase in 233.94 ms, achieving a throughput of approximately 213.73 tokens/s.

Prompt eval time = 233.94 ms / 50 tokens

This is the prefill phase.

Prompt length: 50 tokens

Prefill speed: 213.73 tokens/s

Birth of "KV" Cache

Once we process the user prompt, it leads to the output of the very first generated token. Now, to process the second token (word2), as LLMs are autoregressive, the model needs the context of the user prompt + the first word to generate the 2nd token(word2). Every time to generate the new word, recalculate the whole process [ context of user prompt + word1 + word2 +......]
Recalculating the entire user prompt math for the whole prompt every time, and generating a single new word is higlly inefficient
Here comes the KV cache in the picture. During the prefill phase, the K and V for every token, instead of flushing them, were saved in the GPU Memory (VRAM), which we called the KV Cache, and it acts like short-term memory. Instead of recalculating it, we just compute a new token and append it to the existing KV cache. This saves a lot of compute, but since the cost is precious VRAM

The formula to calculate the KV Cache size is :

Maximum KV Cache Memory (Full 4096-token Context)

Decode Stage[ Generation phase]

In this stage, the model generates the output tokens autoregressively, one at a time, until the stopping criteria are met. This stage is also called Memory Bound, meaning it's limited by how fast the GPU can move data around.
It uses the cache state from prefill and generates the next token, like the absolute best next word based on all available context, repeating token by token

figure 1.2 Decode stage


To connect the theory with a real inference run, here's the timing summary from llama.cpp after generating 512 tokens.

figure 1.3 Phi-4 logs

eval time = 24323.65 ms / 512 tokens

This is the decode phase.

Generated Tokens: 512 tokens

Decode speed: 21.05 tokens/s