Home › Certifications › Databricks Generative AI Engineer › Evaluation and Monitoring
34 multiple-choice questions and 21 flashcards on Evaluation and Monitoring, about 11% of the Databricks Generative AI Engineer bank. Every one carries a written rationale.
Evaluation and Monitoring is one of 8 chapters in CoStudy's Databricks Generative AI Engineer bank, and it holds 34 of the bank's 300 multiple-choice questions — roughly 11% of the total. That proportion is not arbitrary: chapters follow the certifying body's published exam outline, and the number of questions in each is set by that domain's published weight, so the share of your practice time this chapter takes matches the share of the real exam it accounts for.
Studying by chapter is worth doing once you have a diagnostic score. A single overall percentage tells you whether you are close; it does not tell you which domain is dragging. Working a weak chapter in isolation, and re-testing it in isolation, is the fastest way to move a score that has stalled — and it is why the mock exams in CoStudy report by domain rather than as one number.
10 questions drawn from this chapter, with the full rationale shown — the controlling principle behind the right answer, and why each wrong option tempts and fails.
An endpoint must enforce per-request token limits + provider routing. Right Databricks feature?
Answer: B — AI Gateway routes with rate limits and provider fallbacks
B) AI Gateway is the policy layer in front of endpoints: it defines routes, enforces usage and rate limits, and abstracts the underlying provider. A) DLT expectations validate pipeline data, not request policy. C) Genie does not front model endpoints. D) Auto Loader ingests files after the fact.
An agent calls three tools. To debug a misbehaving run you should:
Answer: D — Use MLflow Tracing to inspect spans for LLM, retrieval, and tool calls
D) Correct - tracing records each step as a span with its inputs and outputs, so you can see which call went wrong. A) Manual log correlation is slow and loses the parent-child structure. B) Retraining is an expensive response to what is probably an orchestration bug. C) Ablation removes capability and still does not show what the tool received.
A team wants to cap the spend a single application can incur against a shared foundation model endpoint and attribute usage per team. The MOST appropriate control is
Answer: A — AI Gateway rate limits and usage tracking on the endpoint
A) Correct — the gateway is the enforcement and accounting point for rate limits, usage tracking and payload logging. B) Token caps reduce per-call cost but impose no ceiling on volume. C) Table grants control data access, not model spend. D) Time windows shift load rather than bounding it.
When evaluating an agent end-to-end vs component-wise, you should:
Answer: C — Both: components localize faults, end-to-end shows UX
C) Component metrics tell you which stage failed while end-to-end metrics reflect what the user actually experienced, so both are needed. A) End-to-end alone leaves you unable to diagnose. B) Component alone can look healthy while the answer is poor. D) Complaints are a lagging and biased sample.
Two RAG quality dimensions to evaluate are:
Answer: B — Retrieval quality and answer correctness
B) Correct — measure retrieval separately with recall and MRR, then measure generation with correctness and groundedness, so you know which stage failed. A) Security properties, not answer quality. C) Disaster-recovery targets. D) Front-end concerns.
A regression slips into production. Best first action:
Answer: C — Roll back to the prior alias and inspect stored traces
C) Restore known-good behavior first by repointing the alias, then diagnose offline using the captured request and response traces. A) Waiting extends user impact with no plan. B) A full shutdown is a heavier outage than a rollback. D) Re-training is premature before the cause is known.
Lakehouse Monitoring on inference data helps you detect:
Answer: A — Input drift, output quality trends, and SLA regressions
A) Lakehouse Monitoring profiles inference data over time, surfacing distribution shift, quality movement, and service regressions. B) Facility events are cloud provider concerns. C) Training metrics live with the MLflow run, not the inference table. D) The monitoring is managed, not hand-rolled.
An AI Gateway is used to:
Answer: A — Proxying LLM calls with rate limits and auth
A) Correct — AI Gateway fronts internal and external model calls so rate limiting, credentials, redaction, and logging are enforced in one place. B) Training runs elsewhere. C) That is Vector Search. D) The gateway serves no UI layer.
LLM-as-judge results should be:
Answer: B — Calibrated against human ratings and watched for drift
B) A judge is itself a model with bias and version drift, so its scores need anchoring to human labels and ongoing monitoring. A) Blind trust turns judge error into silent product error. C) Evaluation exists to guide engineering decisions. D) Random scores destroy the signal you are measuring.
LLM-as-judge evaluation provides:
Answer: D — Rubric-based grading by a strong LLM
D) Correct — a strong judge model scores outputs against rubrics repeatably, which scales far past hand grading. A) Judging is an evaluation step, not training. B) It substitutes for human labeling rather than making it free, and it still costs tokens. C) It removes the manual labeling, not adds it.
4 cards from the 21 in this chapter.
Inference Tables?
Auto-logged Delta tables of every serving request + response. Foundation for monitoring + replay.
Why use LLM-as-judge?
Scalable, consistent evaluation without exhaustive human labeling. Calibrate vs human ratings to validate judge.
Cost monitoring?
Track token usage per endpoint, per user, per query. Identify expensive prompts; consider prompt compression.
Mosaic AI Agent Evaluation?
LLM-as-judge based evaluation framework. Includes built-in judges for relevance, groundedness, harmfulness, plus custom rubrics.
These are a sample. The full Evaluation and Monitoring chapter runs 55 items with per-chapter progress tracking, on the web and in the iOS app.
Open Databricks Generative AI Engineer in CoStudy →