Transformer Architecture and LLM Fundamentals — NVIDIA NCA Gen AI LLMs practice questions

Transformer self-attention computes:

1. softmax(QKᵀ/√d_k)V
2. Q + K + V
3. Convolution of inputs
4. Recurrent state update

Answer: A — softmax(QKᵀ/√d_k)V

A) Correct — scaled dot-product attention: Q · Kᵀ → scale by √d_k → softmax → weight V. B) Wrong operation. C) CNNs, not transformers. D) RNNs.

Which statement about Mixture-of-Experts (MoE) transformers is correct?

1. A router activates only a few expert FFNs per token
2. Every parameter is used for every token processed
3. MoE removes attention layers from the stack entirely
4. MoE is incompatible with autoregressive decoding

Answer: A — A router activates only a few expert FFNs per token

B) That describes dense models. A) Correct — sparse routing activates a small subset of experts per token, giving a high total parameter count at much lower per-token compute. C) Attention layers remain. D) MoE works fine with decoders such as Mixtral.

ReAct pattern alternates between:

1. Training and test phases of a supervised learning run
2. Encoder and decoder passes inside a seq2seq model stack
3. Thought, Action (tool call), and Observation (result)
4. Two separate models exchanging generated messages

Answer: C — Thought, Action (tool call), and Observation (result)

A) Those are training stages. C) Correct — ReAct loops: the model thinks, chooses an action, executes it, observes the result, and continues until the goal is met. Foundation of agents. B/D) Unrelated.

'Lost in the middle' problem in LLMs:

1. An encryption failure in the model serving pipeline
2. A hardware fault in GPU memory during long runs
3. A tokenizer bug that silently drops mid-sequence tokens
4. Mid-context information is attended to less than ends

Answer: D — Mid-context information is attended to less than ends

A) Unrelated. D) Correct — Liu et al. (2023) showed a serial-position effect: models use information at the start and end of the context far better than material in the middle. Mitigate by structuring prompts, summarizing and reranking. B/C) Not the phenomenon.

GPT-style models use which transformer variant?

1. Encoder-only stack with bidirectional self-attention
2. Encoder-decoder stack with cross-attention to a source
3. Decoder-only stack, autoregressive with causal masking
4. Convolutional stack with fixed-width filters over tokens

Answer: C — Decoder-only stack, autoregressive with causal masking

A) BERT-style. C) Correct — GPT uses a decoder-only stack with causal masking, generating one token at a time. B) T5, BART. D) Not a transformer variant.

Layer normalization in modern transformers is typically applied:

1. Before attention and before the FFN, called Pre-LN
2. Only at the input embedding, before block one
3. Only after the final output projection of the stack
4. Inside the attention softmax, applied to raw logits

Answer: A — Before attention and before the FFN, called Pre-LN

B) Insufficient for stability. A) Correct — Pre-LN is the de facto choice in GPT/Llama-style stacks and trains more stably than Post-LN. C) That is too late to help. D) The softmax is not normalized this way.

Rotary Position Embeddings (RoPE) primarily encode position information by:

1. Rotating query and key vectors by position-dependent angles
2. Adding learned absolute position vectors to embeddings
3. Applying sinusoidal position embeddings only at layer zero
4. Replacing self-attention with depthwise convolution blocks

Answer: A — Rotating query and key vectors by position-dependent angles

B) That is the original absolute scheme used by GPT-2. A) Correct — RoPE applies a position-dependent rotation to Q and K, giving relative-position behavior in the dot product. C) Sinusoidal encoding is a different scheme. D) Not how RoPE works.

FlashAttention improves performance primarily by:

1. Approximating attention with low-rank factorization
2. Replacing the attention softmax with a ReLU nonlinearity
3. Tiling attention in on-chip SRAM to cut HBM traffic
4. Quantizing query and key projections to INT4 precision

Answer: C — Tiling attention in on-chip SRAM to cut HBM traffic

A) That is Linformer-style approximation. C) Correct — FlashAttention is exact but I/O-aware, tiling and recomputing scores in fast on-chip SRAM to reduce HBM reads and writes. B) A different research direction. D) Quantized attention is separate work.

A support assistant must produce consistent, factual answers to identical policy questions. The BEST decoding configuration is:

1. Temperature near 1.0 with top-p 0.95
2. A low temperature with greedy or near-greedy selection
3. Temperature above 1.5 to broaden coverage of possible phrasings
4. Beam search with a large beam width and a high temperature

Answer: B — A low temperature with greedy or near-greedy selection

B) Correct — flattening randomness maximizes reproducibility and factual adherence for policy answers. A) A near-default sampling setup introduces run-to-run variation this use case cannot tolerate. C) Raising temperature further increases diversity and hallucination risk. D) Adding high temperature to beam search reintroduces the variability beam search was meant to reduce.

Which architecture choice is associated with Llama-style models?

1. Sinusoidal absolute positions, Post-LN, and ReLU
2. Learned positions, GELU, and an encoder-only stack
3. RoPE, RMSNorm, SwiGLU, and a decoder-only stack
4. ALiBi biases, LayerNorm, and sigmoid attention

Answer: C — RoPE, RMSNorm, SwiGLU, and a decoder-only stack

A) That is original GPT-2 style. C) Correct — the Llama family uses RoPE positions, RMSNorm, SwiGLU activations and a decoder-only stack. B) That describes BERT-ish models. D) Mixes unrelated choices.

Why use multiple attention heads instead of one wide attention operation?

Each head projects into its own lower-dimensional subspace, so different heads can specialize on different relationships — syntactic dependency, coreference, positional locality — in parallel. Their outputs are concatenated and linearly projected, giving the layer several distinct relational views at the same total compute cost as one wide head.

Position bias / 'Lost in the middle'?

LLMs attend more to start + end of context. Important info in middle may be ignored. Mitigate: structure prompts, reranking.

Zero-shot vs Few-shot prompting?

Zero-shot: task description only. Few-shot: include 2-10 examples in prompt to demonstrate format.

Contrast greedy decoding, beam search, top-k, and top-p sampling.

Greedy takes the single highest-probability token each step — fast but repetitive. Beam search keeps several partial sequences and picks the best total likelihood, favoring fluent but generic text. Top-k samples from the k most probable tokens. Top-p, or nucleus sampling, samples from the smallest set of tokens whose cumulative probability exceeds p, so the candidate pool adapts to how confident the model is.