Experimentation — Fine-Tuning and Alignment — NVIDIA NCA Gen AI LLMs practice questions

LoRA (Low-Rank Adaptation) does which?

1. Updates all base model weights during fine-tuning
2. Distills a large teacher model into a smaller student
3. Learns a new subword tokenizer for the target domain
4. Trains small low-rank matrices inside frozen layers

Answer: D — Trains small low-rank matrices inside frozen layers

A) Full fine-tuning is the alternative LoRA avoids. D) Correct — LoRA freezes the base model and trains low-rank decomposition matrices inserted into attention/FFN layers, typically about 0.1–1% of parameters. Lightweight and composable. B/C) Different techniques.

Continued pretraining (a.k.a. domain-adaptive pretraining) is best used when:

1. You hold only a small labeled instruction set
2. You need to benchmark the model against MMLU
3. You need to stand up a production inference server
4. You hold large unlabeled in-domain text corpora

Answer: D — You hold large unlabeled in-domain text corpora

D) Domain-adaptive pretraining continues the language modeling objective on bulk in-domain text so the base model absorbs the domain distribution before instruction tuning. A) A small labeled set calls for SFT instead. B) Benchmarking is evaluation, not a training stage. C) Serving is downstream of training entirely.

KTO (Kahneman-Tversky Optimization) is best described as a preference-tuning method that:

1. Requires strictly paired chosen/rejected preference data
2. Replaces the language modeling head with a regression head
3. Uses unpaired good/bad labels via a prospect-theory loss
4. Trains only the embedding layer and freezes the rest

Answer: C — Uses unpaired good/bad labels via a prospect-theory loss

C) KTO applies a prospect-theory-inspired utility loss to unpaired desirable/undesirable examples. A) Pairing is exactly the requirement KTO drops. B) The LM head is retained. D) KTO is not restricted to the embedding layer.

QLoRA's principal advantage over standard LoRA is that it:

1. Trains adapters over a 4-bit quantized frozen base to cut memory
2. Eliminates the need for a separate validation split
3. Removes the low-rank constraint on the adapter update
4. Applies quantization to the adapter weights during training

Answer: A — Trains adapters over a 4-bit quantized frozen base to cut memory

A) Correct — quantizing the frozen base to 4 bits is what lets large models be tuned on a single GPU while adapters train in higher precision. B) Validation discipline is unchanged. C) The update remains low-rank; that is still LoRA. D) The adapters are kept in higher precision precisely so gradients remain usable.

Catastrophic forgetting during fine-tuning is best mitigated by:

1. Using a very high learning rate on a narrow dataset
2. Skipping evaluation entirely and shipping the model fast
3. Mixing in general data, using PEFT and low learning rates
4. Training for a single epoch on a very small dataset

Answer: C — Mixing in general data, using PEFT and low learning rates

C) Replaying general-domain data, constraining the update with PEFT or small learning rates, and regularizing toward the base model all limit drift. A) A large learning rate on narrow data accelerates forgetting. B) Skipping evaluation hides the problem rather than mitigating it. D) Shrinking the run limits learning without protecting prior capabilities.

Which sequence describes the most common modern LLM training pipeline from scratch to chat assistant?

1. Supervised fine-tuning on instructions first, then pretraining, then RLHF
2. RLHF on preference pairs, then pretraining, then supervised fine-tuning
3. Pretraining on unlabeled text, then instruction SFT, then RLHF or DPO alignment
4. Pretraining on unlabeled text, then quantization, then prompt engineering

Answer: C — Pretraining on unlabeled text, then instruction SFT, then RLHF or DPO alignment

C) Pretrain on broad unlabeled text, then supervised fine-tune on instructions, then align on preferences — that ordering is the standard recipe. A) Reverses pretraining and fine-tuning. B) Puts alignment before the model can even model language. D) Quantization and prompting are deployment steps, not a substitute for alignment.

Which mitigation MOST directly reduces catastrophic forgetting during domain fine-tuning?

1. Mixing a replay sample of general-domain data into training
2. Raising the learning rate so the model adapts to the domain faster
3. Training for many more epochs on the domain corpus alone
4. Increasing the batch size while keeping the same data

Answer: A — Mixing a replay sample of general-domain data into training

A) Correct — replay keeps gradient signal for the original distribution, preserving general ability. B) A higher learning rate accelerates overwriting. C) More epochs on the narrow corpus deepens the specialization. D) Batch size changes gradient noise, not the data distribution being fit.

Which is the BEST description of LoRA adapter merging at deployment?

1. Base weights and LoRA BA can be merged into one tensor
2. Adapters must remain separate at runtime to function at all
3. Merging adapters requires retraining the whole base model
4. Merging only works with prefix-tuning style adapters

Answer: A — Base weights and LoRA BA can be merged into one tensor

A) Because the update is a plain additive low-rank product, it can be folded into the base tensor and served with no adapter overhead — at the cost of easy multi-adapter swapping. B) Separation is optional, not required. C) Merging is arithmetic and needs no retraining. D) Prefix tuning is a different PEFT family.

Instruction tuning is BEST characterized as supervised training that:

1. Optimizes a scalar preference reward using policy gradients
2. Compresses a large teacher model into a smaller student
3. Continues next-token pretraining on unlabeled domain documents
4. Teaches a base model to follow task directions given as prompts

Answer: D — Teaches a base model to follow task directions given as prompts

D) Correct — instruction tuning pairs directions with desired responses so the base model learns the instruction-following format. A) That describes RLHF with PPO. B) That describes distillation. C) That describes continued pretraining, which is unlabeled.

LoRA (Low-Rank Adaptation)?

Insert low-rank matrices into transformer layers. Train only those (~0.1-1% of params). Lightweight + composable.

What does QLoRA add to LoRA, and why does that combination work?

QLoRA quantizes the frozen base model to 4-bit and trains LoRA adapters in higher precision on top of it, with gradients backpropagated through the quantized weights. Base weights are never updated so their quantization error is fixed and harmless, while the small high-precision adapters carry all the learning — cutting fine-tuning memory enough to tune large models on a single GPU.

Instruction tuning?

Supervised fine-tuning on (instruction, response) pairs. Aligns base model to follow instructions.

Catastrophic forgetting?

Fine-tuning on narrow domain causes loss of general capabilities. Mitigate: lower learning rate, mixed domains, PEFT.