Securing AI Systems — Controls and Guardrails — CompTIA SecAI+ practice questions

Which statement about system prompts and user prompts is MOST accurate for a securely designed application?

1. The user prompt should be concatenated ahead of the system prompt
2. The system prompt outranks user text and must never embed user input
3. Both carry identical trust since the model sees one token stream
4. The system prompt is only needed once tools have been enabled

Answer: B — The system prompt outranks user text and must never embed user input

B) Correct - the prompt hierarchy exists so developer-set policy outranks user content, and that boundary collapses the moment user text is interpolated into the system message. A) Ordering the user text first inverts the hierarchy the design depends on. C) They share a token stream but the application must still enforce a trust distinction. D) Policy, tone and refusal rules are needed with or without tools.

An agent acts on behalf of many users against a document store. Which approach BEST preserves least privilege?

1. Grant a static high-privilege token and filter results after
2. Exchange the session for a short-lived delegated token
3. Cache each user's credentials in the agent for reuse later
4. Let the agent request elevation whenever a query fails

Answer: B — Exchange the session for a short-lived delegated token

B) Correct - per-request delegated credentials keep the data layer enforcing the user's own entitlements, so an injected instruction cannot exceed them. A) Post-hoc filtering means over-privileged data already entered the context. C) Storing credentials creates a high-value secret store and no scoping benefit. D) Automatic elevation on failure is privilege escalation by design.

A vision model is brittle to small rotations and lighting changes that attackers exploit. Which training-time measure MOST directly improves robustness?

1. Reducing the number of training epochs used
2. Raising the classification decision threshold
3. Augmenting with realistic and adversarial samples
4. Switching the training loss function to accuracy

Answer: C — Augmenting with realistic and adversarial samples

C) Correct - augmenting with the variations the model must tolerate, adversarial ones included, teaches invariance rather than patching symptoms. A) Fewer epochs generally reduces fitting quality. B) Threshold tuning trades precision against recall without adding robustness. D) Accuracy is a metric, not a differentiable training loss.

Prompt injection mitigations include:

1. Treat user-supplied text as trusted instructions
2. Input sanitization, output filters, role split
3. Increase model parameter count to resist injection
4. Disable the LLM whenever untrusted input is present

Answer: B — Input sanitization, output filters, role split

B) Correct — defense in depth: validate input, filter output, and keep system and user roles separated in the chat template. A) Trusting user text is the vulnerability itself. C) Scale does not remove the instruction-following weakness. D) Turning the feature off is avoidance, not a control.

Output validation for an LLM API should include:

1. Trusting the model output as returned by the API
2. Logging every response without inspecting content
3. Schema, PII, and grounding checks on responses
4. Sampling a random subset of responses for review

Answer: C — Schema, PII, and grounding checks on responses

C) Correct — output validation layers structural checks with content filtering for sensitive data and a grounding or citation check before the response is released. A) Unvalidated output is how leakage and injection payloads escape. B) Logging records the problem but does not stop it. D) Sampling misses the responses that are not sampled.

Before launching a customer-facing assistant, a team hires specialists to attempt jailbreaks, injections and data extraction against it. This activity is BEST called:

1. Unit testing of components
2. Adversarial red-teaming
3. Benchmark evaluation runs
4. Canary deployment staging

Answer: B — Adversarial red-teaming

B) Correct - adversarial testing by people trying to elicit harmful or policy-violating behaviour before release is AI red-teaming. A) Unit tests verify deterministic component behaviour. C) Benchmarks measure capability on fixed tasks, not adversarial resilience. D) Canary release is a rollout strategy applied after launch begins.

Synthetic data is BEST justified when:

1. Real data is already plentiful, balanced, and free of bias
2. You want to inflate reported accuracy on a holdout test set
3. You intend to skip formal evaluation of the trained model
4. Real data is scarce, sensitive, or imbalanced and validated

Answer: D — Real data is scarce, sensitive, or imbalanced and validated

D) Correct — synthetic data earns its place when real records are scarce, too sensitive to use directly, or badly imbalanced, and only when the generated set is validated against the real distribution. A) Incorrect — abundant clean data removes the reason to synthesize at all. B) Incorrect — inflating a score is measurement fraud, not a justification. C) Incorrect — synthetic data raises the need for evaluation rather than removing it.

An internal model endpoint is currently reachable by any workload in the VPC with no credential. The MOST appropriate first improvement is:

1. Rotate the deployed model artefact every month
2. Require authenticated per-service identities
3. Compress requests to reduce network bandwidth
4. Increase the model's available context window

Answer: B — Require authenticated per-service identities

B) Correct - model access control begins with authenticating callers and authorising them per request so usage is attributable and constrained. A) Rotating artefacts addresses integrity, not who may call the endpoint. C) Compression is a performance measure. D) Context size is a capability setting with no access implication.

Differential privacy (DP) provides:

1. Encryption of stored training datasets while they sit at rest
2. An automatic improvement in accuracy on the downstream task
3. Faster training through reduced gradient computation cost
4. An epsilon-bounded limit on any single record's influence

Answer: D — An epsilon-bounded limit on any single record's influence

D) Correct — differential privacy adds calibrated noise so that the presence or absence of any one record changes the output distribution by no more than the epsilon budget, which is what blunts membership inference. A) Incorrect — encryption at rest protects storage but says nothing about what the model reveals. B) Incorrect — the added noise generally costs accuracy rather than improving it. C) Incorrect — the mechanism adds work rather than accelerating training.

Model Registry?

Versioned catalog of trained models with metadata, lineage, deployment status. MLflow, Vertex, SageMaker have these.

How is least privilege applied to an AI agent with tool access?

Give the agent its own identity, not a human's. Grant only the specific tools the task needs, read-only wherever possible, scoped to the minimum data set. Require human approval for irreversible or high-impact actions, issue short-lived credentials, and review the granted scopes as the agent's tasks change.

What is a prompt firewall?

An inspection layer that sits between the user or retrieved content and the model, scanning for injection patterns, policy-violating requests and sensitive data, and blocking, sanitising or flagging them before they reach the model. It is the AI-application analogue of a web application firewall.