Home › Certifications › AWS Certified Generative AI Developer AIP-C01 › Testing, Validation, and Troubleshooting
16 multiple-choice questions and 7 flashcards on Testing, Validation, and Troubleshooting, about 11% of the AWS Certified Generative AI Developer AIP-C01 bank. Every one carries a written rationale.
Testing, Validation, and Troubleshooting is one of 5 chapters in CoStudy's AWS Certified Generative AI Developer — Professional (AIP-C01) bank, and it holds 16 of the bank's 150 multiple-choice questions — roughly 11% of the total. That proportion is not arbitrary: chapters follow the certifying body's published exam outline, and the number of questions in each is set by that domain's published weight, so the share of your practice time this chapter takes matches the share of the real exam it accounts for.
Studying by chapter is worth doing once you have a diagnostic score. A single overall percentage tells you whether you are close; it does not tell you which domain is dragging. Working a weak chapter in isolation, and re-testing it in isolation, is the fastest way to move a score that has stalled — and it is why the mock exams in CoStudy report by domain rather than as one number.
1 questions drawn from this chapter, with the full rationale shown — the controlling principle behind the right answer, and why each wrong option tempts and fails.
A team is evaluating a summarization model in Amazon Bedrock and needs to score a subjective quality -- whether the tone of each summary is appropriate for an executive audience -- across hundreds of generated outputs. Which evaluation approach is MOST appropriate?
Answer: C — Configure a Bedrock human evaluation job with a defined work team and a rubric that scores tone appropriateness on an anchored scale
Right -- tone appropriateness is a nuanced, subjective judgment that automatic metrics cannot reliably score, so a human evaluation job with a defined rubric and work team is the controlling choice; the binding constraint is that only human raters can judge audience-appropriate tone consistently. The toxicity metric is a real automatic metric, but it measures harmful content, not tone fit, so it is true but irrelevant here. ROUGE measures n-gram overlap with a reference text and says nothing about tone, a common misconception about what lexical-overlap metrics can capture. Ad hoc spot-checking by one person is not a repeatable or scalable evaluation process and lacks any defined rubric.
4 cards from the 7 in this chapter.
What does the 'faithfulness' metric measure in RAG evaluation?
Whether the generated answer is factually consistent with and fully supported by the retrieved source documents, flagging cases where the model adds unsupported claims not present in the context.
How should an application respond when it receives a ThrottlingException from Bedrock, and how does load testing help prevent it?
The application should retry using exponential backoff and jitter, and consider requesting a service quota increase or Provisioned Throughput; load testing beforehand helps identify realistic throughput limits and surfaces throttling behavior before it affects production users.
What is automatic model evaluation in Amazon Bedrock?
A built-in evaluation method that scores model outputs against benchmark datasets using programmatic metrics (such as accuracy, robustness, or toxicity), without requiring human reviewers, useful for fast iterative comparisons.
When should you use human-based evaluation instead of automatic evaluation in Bedrock?
When judging subjective qualities like coherence, helpfulness, or brand tone that automated metrics cannot reliably capture; human evaluators score or compare model outputs using defined rating criteria.
These are a sample. The full Testing, Validation, and Troubleshooting chapter runs 23 items with per-chapter progress tracking, on the web and in the iOS app.
Open AWS Certified Generative AI Developer AIP-C01 in CoStudy →
All AWS Certified Generative AI Developer AIP-C01 practice questions →