Home › Certifications › NVIDIA NCA Gen AI LLMs › Experimentation — Evaluation and Experiment Design
30 multiple-choice questions and 11 flashcards on Experimentation — Evaluation and Experiment Design, about 10% of the NVIDIA NCA Gen AI LLMs bank. Every one carries a written rationale.
Experimentation — Evaluation and Experiment Design is one of 8 chapters in CoStudy's NVIDIA NCA Gen AI LLMs bank, and it holds 30 of the bank's 310 multiple-choice questions — roughly 10% of the total. That proportion is not arbitrary: chapters follow the certifying body's published exam outline, and the number of questions in each is set by that domain's published weight, so the share of your practice time this chapter takes matches the share of the real exam it accounts for.
Studying by chapter is worth doing once you have a diagnostic score. A single overall percentage tells you whether you are close; it does not tell you which domain is dragging. Working a weak chapter in isolation, and re-testing it in isolation, is the fastest way to move a score that has stalled — and it is why the mock exams in CoStudy report by domain rather than as one number.
10 questions drawn from this chapter, with the full rationale shown — the controlling principle behind the right answer, and why each wrong option tempts and fails.
A team uses a strong LLM to score outputs from several candidate systems, one of which shares the judge's model family. The MOST serious threat to validity is:
Answer: B — Self-preference bias favoring outputs stylistically like the judge's own
B) Correct — LLM judges systematically over-score text resembling their own distribution, biasing the ranking. A) Cost is a budget concern, not a validity threat. C) Judge verbosity affects readability of rationales, not score validity. D) Diversity across judged systems is not what compromises the comparison here.
Which evaluation choice is most appropriate for measuring an LLM's broad knowledge across academic subjects?
Answer: B — MMLU or a HELM-style broad multi-subject benchmark suite
B) Correct — these suites span many academic subjects, which is what broad knowledge requires. A) BLEU only scores translation. C) Perplexity on training data is uninformative about generalization. D) ROUGE only scores summarization.
A team wants an offline metric that predicts whether users will prefer a new assistant. The BEST approach among these is:
Answer: C — Build a task-specific rubric evaluation validated against human preferences
C) Correct — an offline proxy is trustworthy only when it has been calibrated against the human judgments it is meant to predict. A) General-corpus perplexity is far removed from assistant preference. B) BLEU against a handful of gold replies penalizes valid alternative phrasings. D) Response length is a superficial signal that can be gamed trivially.
After running fifty hyperparameter configurations and selecting the best by validation score, the reported validation score is likely:
Answer: C — Optimistically biased due to selection over many trials
C) Correct — picking the maximum over many noisy estimates captures favorable noise, so a held-out test set is needed for an honest number. A) Selection on that same split destroys unbiasedness. B) The bias runs upward, not downward. D) Nothing forces validation and test scores to match.
Which statement about BERTScore is most accurate?
Answer: B — It uses contextual embeddings to score semantic similarity
B) Correct — comparing contextual embeddings often correlates better with human judgment than n-gram overlap on generation tasks. A) That describes BLEU and ROUGE. C) It does not rank retrieved documents. D) It says nothing about speed.
Which practice contributes MOST to reproducibility of an LLM fine-tuning experiment?
Answer: D — Recording seeds, library versions, data snapshot, and full config
D) Correct — the full provenance of code, data, and configuration is what lets another team regenerate the result. B) A checkpoint alone cannot be re-derived or audited. C) A recorded score is a claim, not a reproduction path. A) Batch size is one hyperparameter and does not by itself confer reproducibility.
A benchmark suite aggregates many subtasks into one headline score. The MAIN risk of relying on that score alone is that it:
Answer: B — Masks large per-subtask regressions behind an average
B) Correct — averaging can hide a serious drop in one capability offset by gains elsewhere, so per-task breakdowns matter. A) Most benchmark suites score automatically. C) Weight availability is irrelevant to score aggregation. D) Aggregate benchmarks apply across architecture families.
Two models report perplexity of 12.4 and 15.8 on different corpora tokenized with different tokenizers. The MOST defensible conclusion is:
Answer: C — The comparison is invalid without a shared corpus and tokenizer
C) Correct — perplexity depends on the evaluation text and the token segmentation, so cross-setup numbers are not comparable. A) Different corpora make any generalization claim unsupported. B) Perplexity carries no information about hallucination rate. D) Declaring equivalence from an arbitrary threshold on incomparable numbers is unfounded.
ROUGE-L is most commonly used for summarization because it captures:
Answer: A — The longest common subsequence shared with the reference
A) Correct — ROUGE-L rewards in-order overlap via longest common subsequence, which suits content-coverage evaluation. B) Fixed-order bigram precision describes a different ROUGE-N variant and ignores subsequence structure. C) Edit distance underlies metrics such as TER, not ROUGE-L. D) Embedding similarity is BERTScore's mechanism.
When designing a human evaluation of summary quality, the MOST important step for interpreting the results is to:
Answer: B — Define a rubric and report inter-annotator agreement
B) Correct — without an explicit rubric and an agreement statistic, scores cannot be interpreted or reproduced. A) Scale without a rubric produces high-volume noise. C) Fixed presentation order introduces the very ordering bias randomization prevents. D) Free-form comments are useful supplements but do not yield comparable measurements.
4 cards from the 11 in this chapter.
Perplexity?
Exponentiated negative log-likelihood. Lower = model is more confident on the test data. Common LLM eval metric.
You change the base model, the prompt, and the retrieval chunk size at once and quality improves. What is wrong and what should you do?
You have confounded three factors, so you cannot attribute the gain, cannot revert the harmful one, and cannot tell whether two changes partially cancelled. Run an ablation: change one factor at a time against a fixed evaluation set with fixed seeds and decoding parameters, report the effect of each, and only then combine. Also check that the improvement exceeds run-to-run variance before calling it real.
LLM-as-judge?
Use a strong LLM (GPT-4, Claude) to grade outputs against rubrics. Scalable + consistent if calibrated.
BLEU score?
Translation/generation metric. N-gram precision against reference. Higher = better. Limited for open-ended tasks.
These are a sample. The full Experimentation — Evaluation and Experiment Design chapter runs 41 items with per-chapter progress tracking, on the web and in the iOS app.
Open NVIDIA NCA Gen AI LLMs in CoStudy →