NVIDIA Inference Stack — NIM, NeMo, Triton, TensorRT-LLM — NVIDIA NCA Gen AI LLMs practice questions

GPU memory utilization for serving LLMs is dominated by which factor for long-context, high-concurrency workloads?

1. Tokenizer vocabulary tables held alongside the model weights
2. KV cache, scaling with batch, sequence length, layers, and heads
3. Optimizer states such as Adam moments kept for each parameter
4. Static execution-graph metadata retained by the inference runtime

Answer: B — KV cache, scaling with batch, sequence length, layers, and heads

B) Correct — KV cache grows with batch size, sequence length, layers, heads, head dim, and precision, so it dominates long-context serving. A) Vocabulary tables are tiny. C) Optimizer states exist only during training. D) Graph metadata is negligible.

INT8 quantization reduces:

1. Model accuracy, driving it to zero across all benchmarks
2. Memory footprint and latency, at a small accuracy cost
3. Memory only, leaving inference latency unchanged
4. Latency only, leaving the memory footprint unchanged

Answer: B — Memory footprint and latency, at a small accuracy cost

A) Hyperbole; the impact is modest. B) Correct — INT8 cuts memory roughly 4× versus FP32 and speeds up GEMMs, with small accuracy loss under good calibration. C/D) It affects both memory and latency.

Which is the BEST description of paged attention (vLLM)?

1. A new positional encoding scheme for long context
2. A tokenizer optimized for serving throughput
3. A weight quantization method for KV tensors
4. Block-paged KV cache modeled on virtual memory

Answer: D — Block-paged KV cache modeled on virtual memory

D) Paged attention stores the KV cache in fixed-size blocks with an indirection table, like OS virtual memory, so requests stop overprovisioning contiguous cache and blocks can be shared. A), B) and C) name unrelated components of the stack.

DGX systems are most accurately described as:

1. Pre-integrated AI servers with NVLink-connected GPUs from NVIDIA
2. Reference designs for partners that are never sold as systems
3. Consumer-grade gaming machines rebranded for developer work
4. Network switches routing traffic between GPU compute nodes

Answer: A — Pre-integrated AI servers with NVLink-connected GPUs from NVIDIA

A) Correct — DGX systems ship as complete servers and workstations tuned for large-model training and inference. B) That conflates DGX with the HGX baseboard reference design. C) They are data center class, not gaming hardware. D) Switching is a different product line.

NVLink is best described as:

1. A high-bandwidth GPU-to-GPU interconnect for model parallelism
2. A full replacement for PCIe in servers, retiring that bus
3. A storage protocol for attaching NVMe arrays to GPU servers
4. A virtualization technology that partitions one GPU into many

Answer: A — A high-bandwidth GPU-to-GPU interconnect for model parallelism

A) Correct — NVLink carries fast tensor exchange between GPUs, which tensor and pipeline parallelism depend on. B) It coexists with PCIe rather than replacing it. C) It is not a storage protocol. D) That describes GPU virtualization, a separate feature.

Which is the BEST high-level architectural separation between NIM, TensorRT-LLM, and Triton?

1. NIM is the packaged microservice, Triton the server, TensorRT-LLM the compiler
2. All three are interchangeable and perform exactly the same function in a stack
3. NIM handles training, TensorRT-LLM handles serving, and Triton handles data prep
4. Triton performs vector search, TensorRT-LLM storage, and NIM business analytics

Answer: A — NIM is the packaged microservice, Triton the server, TensorRT-LLM the compiler

A) Correct — NIM productizes a microservice commonly built over Triton and TensorRT-LLM, which are the server and the LLM inference compiler respectively. B) The three sit at different layers. C) Scrambles the roles. D) None of those functions belong to these components.

Which is the BEST description of NeMo Retriever Embedding/Reranking NIMs?

1. Embedding and reranking models packaged as NIM microservices
2. Open-source vector databases distributed by NVIDIA for retrieval
3. Hardware appliances that accelerate retrieval in the data center
4. A new tokenizer family tuned for retrieval-augmented generation

Answer: A — Embedding and reranking models packaged as NIM microservices

A) Correct — NeMo Retriever ships production embedding and reranking models as microservices with optimized GPU inference. B) The vector store remains a separate component. C) They are containers, not appliances. D) They are models, not tokenizers.

Which NVIDIA service targets generative AI for visual content (images, 3D, video)?

1. BioNeMo
2. Riva
3. Picasso
4. Morpheus

Answer: C — Picasso

A) Biology. B) Speech. C) Correct — NVIDIA Picasso is the visual generative AI cloud service. D) Cybersecurity.

Which is the BEST description of TensorRT-LLM?

1. A general-purpose deep-learning training library for model authoring
2. A library compiling LLMs into optimized TensorRT inference engines
3. An integration layer connecting vector databases to LLM applications
4. A reinforcement-learning environment for training agent policies

Answer: B — A library compiling LLMs into optimized TensorRT inference engines

B) Correct — TensorRT-LLM builds optimized engines with kernel fusion, paged attention, in-flight batching, and quantization. A) It targets inference, not training. C) Retrieval integration is a separate concern. D) It is not an RL environment.

Tensor parallelism in Megatron-LM splits:

1. Whole layers placed on separate GPUs sequentially
2. Different micro-batches processed on separate GPUs
3. Individual GEMMs split across GPUs by feature dimension
4. Optimizer states sharded across data-parallel ranks

Answer: C — Individual GEMMs split across GPUs by feature dimension

C) Tensor parallelism shards the matrix multiplications inside a layer along hidden or head dimensions. A) Splitting whole layers is pipeline parallelism. B) Splitting batches is data parallelism. D) Sharding optimizer state is ZeRO.

What role does NeMo Retriever play in a RAG system?

NeMo Retriever provides the GPU-accelerated information-retrieval building blocks for RAG: document extraction and ingestion, embedding models that turn chunks and queries into vectors, and reranking models that reorder candidates by relevance. It supplies retrieval quality and throughput; it is not the generator and is not itself a vector database.

NIM (NVIDIA Inference Microservices)?

Containerized, optimized inference for foundation models. Pre-built endpoints for Llama, Mistral, etc.

Paged attention (vLLM)?

Memory technique for KV cache: split into pages so memory isn't reserved per request. Dramatically improves throughput.

NVIDIA NeMo Framework?

End-to-end framework for training, fine-tuning, customizing, deploying generative AI models. Includes Megatron-LM, NeMo Curator (data).