The NCA-GENL Playbook

1.0 Start here: ML for software engineers ~40 min

If you write code but have never done machine learning, start here. Nothing is assumed. Lessons 1.1 and 1.2 build on this page, so come back whenever they lose you.

The idea in plain English

In normal programming you write the rules: if (email.includes("FREE")) return "spam". That breaks down when the rules are too many or too fuzzy to write by hand, like spotting spam, understanding a sentence or recognising a face.

Machine learning flips it around. You collect lots of examples with the right answer attached, and an algorithm finds the rules for you. The result is an ML model: a function such as predict(email) → probability of spam. Its logic lives in a huge number of adjustable numbers called parameters (also called weights) instead of hand-written if-statements.

Training means automatically adjusting those numbers until the function's outputs match the examples. Inference means calling the trained function in production on new input. That's it. An LLM is the same idea with billions of weights, trained to predict the next word.

The figure below runs the spam filter both ways, first with hand-written rules and then with ML.

Step 1 · Examples: features and labels

Training data is shaped like a test suite. Each example is an input plus its expected answer:

// one training example (one row of the dataset)
{ features: { freeWords: 2, links: 3, knownSender: 0 },   label: 1 /* 1 = spam */ }

• The input features are the facts you measure about each example: one field or column each.
• The label is the expected output, like expected in assertEquals(expected, actual). A dataset of examples with labels is called labeled data.

Two jobs come up constantly. Classification picks a bucket (spam / not spam). Regression predicts a number (delivery time in minutes).

Step 2 · A neuron is three lines of code

The smallest building block of a neural network is a neuron. It does two things: add up its inputs with weights, then squash the total.

const sigmoid = z => 1 / (1 + Math.exp(-z));      // squash any number into 0..1

function neuron(inputs, weights, bias) {
  let z = bias;
  for (let i = 0; i < inputs.length; i++) z += inputs[i] * weights[i];   // weighted sum
  return sigmoid(z);                                                     // "activation"
}

Line by line: multiply each input by its weight, add them up, and add a bias (a constant starting offset). This weighted sum, called z, can be any number: −50, 0.3, 812. Then the sigmoid squashes z into the range 0 to 1.

The weights decide how much each input matters. A big positive weight on "FREE words" pushes the email towards spam. A big negative weight on "known sender" pulls it towards your inbox. Finding good weights is the entire job of training. In the next figure, you play the training algorithm.

Step 3 · Stack neurons into a network

One neuron can only draw one straight dividing line through the data. A neural network is many neurons arranged in layers, where each layer's outputs are the next layer's inputs:

• the input layer is just the features;
• one or more hidden layers in the middle combine them into more useful signals;
• the output layer gives the answer (for example, one neuron that outputs the spam probability).

"Deep learning" simply means networks with many hidden layers.

Step 4 · Training: the loop that finds the weights

A new network starts with random weights, so its predictions are garbage. Training repeats one loop, a bit like CI re-running until the tests pass:

1. Forward pass. Run the network on some examples and get its predictions.
2. Loss. A loss function turns "how wrong were we?" into a single number. Training tries to make it small. For yes/no answers the usual choice is cross-entropy, which punishes confident wrong answers hard.
3. Backpropagation. Work out, for every weight, which direction to nudge it (up or down) and how strongly, so the loss shrinks. That direction-and-strength is the weight's gradient. Backprop computes it using the chain rule from calculus, run backwards from the output to the input.
4. Optimizer step. The optimizer (SGD, Adam) nudges every weight a small step against its gradient: w -= lr * gradient.

The next figure walks one full turn of the loop through a tiny network.

Two more words you'll see everywhere. One full pass over all the training data is an epoch. The data is fed in small groups called batches, and each batch produces one weight update. Training usually runs for many epochs.

Step 5 · The learning rate: how big each nudge is

The lr in w -= lr * gradient is the learning rate, the step size. Picture walking downhill in thick fog: you can only feel the slope under your feet (the gradient), so you take a step downhill and check again. That's gradient descent.

• Steps too small: you get there, eventually. Training crawls.
• Steps too big: you overshoot the valley and land higher up the other side. The loss zig-zags, then explodes to infinity or NaN.

Getting the learning rate wrong is the most common reason training fails. Try each preset in the figure.

The learning rate is a hyperparameter: a setting you choose before training, like a config flag. Batch size and the number of layers are hyperparameters too. Contrast that with parameters (the weights), which training learns.

Here's the whole loop running for real on the simplest possible model, a straight line with two weights:

Step 6 · Why the squash? Activation functions

The sigmoid is one example of an activation function: a small non-linear function applied after each weighted sum. Why bother? Without it, every layer is just multiply-and-add, and multiply-and-add of a multiply-and-add is still one multiply-and-add. A hundred such layers collapse into one, and they can only learn straight lines. The activation adds a bend, so networks can learn curves.

• ReLU is Math.max(0, x): negatives become 0, positives pass through. It's the default for hidden layers.
• GELU and SwiGLU are smoother cousins used inside modern LLMs.
• Sigmoid is used for yes/no outputs. Softmax is used when the answer is one of many options (lesson 1.2).

The slope of the activation matters too. The gradient has to flow backwards through it during backprop. Where the curve is flat, the slope is about 0 and learning stalls. Lesson 1.2 calls this the vanishing-gradient problem.

Step 7 · The kinds of learning

Everything above assumed you have the right answers (labels). You don't always. The kinds of learning differ in what signal the model learns from:

Cheat sheet: the jargon, translated

Every word from this page in one lookup table, plus a few you'll meet in the next lessons.

Next: 1.1 adds overfitting, validation and cross-validation (with figures), and 1.2 adds loss functions, vanishing gradients and dropout.

1.1 Machine-learning foundations

Before this: 1.0 Start here (what a model, a label, the loss and training are).

The idea in plain English

Every ML project answers the same four questions. What signal do I have to learn from? Did the model learn the pattern, or just memorise the examples? How do I measure that honestly? Which model and which inputs should I use?

Think of the training data as a test suite. A model that passes every test because it hard-coded the expected outputs is useless in production. Most of this lesson is about catching that before you ship.

Five ways a model can learn

The paradigms differ in what signal the model learns from. 1.0 introduced them in Step 7; here they are with the algorithm names the exam uses and where each one appears in the LLM world.

• Supervised: every example has the right answer (a label). Learn to reproduce it.
• Unsupervised: no labels. Find structure, such as groups of similar items.
• Self-supervised: make the labels from the data itself, for example by hiding the next word and predicting it.
• Reinforcement: try things, get a reward or penalty, do more of what scored well.
• Transfer: start from a model someone already trained and adapt it to your task.

Learning without labels: clustering with k-means

With unsupervised learning there are no right answers to copy. The most common job is clustering: putting similar items into groups, such as support tickets by topic or log lines by error type.

The classic algorithm is k-means. You choose k, the number of groups. It drops k centre points at random, then repeats two steps until nothing changes:

1. Assign: each point joins its nearest centre.
2. Update: each centre moves to the average position of its points.

Watch it run below. Notice that nobody tells it which point belongs where.

Other unsupervised names in the table above: DBSCAN (density-based clustering), hierarchical clustering (a tree of nested groups), PCA (compress many columns into a few) and autoencoders (a network that learns to compress and rebuild its input). In LLM systems, the same idea is used to cluster embeddings and discover topics.

Did it learn or memorise? Underfitting and overfitting

A model is only useful on data it has never seen. So you hold some data back as a validation set and compare two numbers: the error on the training data and the error on the validation data.

• Underfitting (high bias): bad on train and validation. The model is too simple to capture the pattern. Fix with a bigger model, better features, or longer training.
• Overfitting (high variance): great on train, bad on validation. The model memorised the noise in the training data. Fix with more data, augmentation, dropout, weight decay (L2), L1, early stopping, or a simpler model.

In the figure, the slider controls how flexible the model is. Find the setting where validation error is lowest.

Learning curves show the same thing over training time instead of model size. Plot training loss and validation loss after each epoch. If validation loss turns upward while training loss keeps falling, stop there and keep that checkpoint. That's early stopping.

Measuring honestly: train, validation and test

Split your data three ways before you do anything else:

• Train: fits the weights. These are the practice problems.
• Validation: tunes hyperparameters and picks the best checkpoint. These are the mock exams.
• Test: the final, unbiased score. Touch it once, like the real exam. If you tune against it, the score stops meaning anything.

One train/validation split has a problem: the score depends on the luck of which rows landed in validation. k-fold cross-validation fixes that. Cut the non-test data into k equal folds, train k times (each time holding out a different fold for validation), and average the k scores.

Pick the variant that matches your data:

• Stratified k-fold keeps the class ratio the same in every fold. Use it for imbalanced data, such as 2% fraud.
• Group splits keep related rows (the same user, the same patient) together in one fold.
• Time-series splits always validate on data that comes later than the training data.

Classic models to recognise

Before deep learning these were the whole toolbox, and they still win on small and tabular datasets. The exam expects you to recognise the names and know each one's strength.

Feature engineering: preparing the inputs

Models only see numbers. Feature engineering turns raw columns into numbers a model can learn from. It's the ETL step of ML.

• Scaling: standardization (z-score: subtract the mean, divide by the standard deviation) or min-max (squeeze into 0 to 1). It matters for distance and gradient methods, not for trees.
• Encoding categories: one-hot (one 0/1 column per category) or ordinal (numbers 0, 1, 2 in a meaningful order). Use embeddings for high cardinality, for example 100,000 product IDs.
• Text: bag of words, n-grams and TF-IDF, then word embeddings, then contextual embeddings. All of these are in 1.3.
• Cleanup and extras: imputation (filling in missing values), log transforms for skewed numbers, and interaction features (combining two columns).
• Use an sklearn Pipeline + ColumnTransformer to avoid leakage. The pipeline fits every step on the training data only, so validation data can't leak in.

1.2 Neural networks & training

Before this: 1.0 Start here, Steps 2–6 (the neuron, the training loop and activation functions, all animated).

The idea in plain English

A neural network is the neuron from 1.0 copied many times and stacked in layers. Each neuron computes activation(w·x + b): a weighted sum of its inputs plus a bias, then a bend. Stacked layers learn hierarchical features: early layers spot simple patterns, and later layers combine them into complex ones (for images: edges, then shapes, then faces).

Training is the same loop as in 1.0: forward pass → loss → backpropagation (chain rule) → optimizer step. This lesson is about the choices you make inside that loop: which bend, which loss, which optimizer settings, and what to do when training goes wrong.

Activation functions: which bend, where

Where the activation sits decides which one you use.

• Hidden layers want something cheap whose slope doesn't fade away: ReLU, or its smooth cousins GELU and SwiGLU in LLMs.
• The output layer depends on the shape of the answer:
  • one yes/no answer, or several independent yes/no answers (an email can be both "urgent" and "billing", called multi-label) → sigmoid on each output;
  • exactly one answer out of many classes (which of 50,000 words comes next) → softmax.

function softmax(xs) {
  const e = xs.map(Math.exp);              // make every score positive
  const sum = e.reduce((a, b) => a + b);
  return e.map(v => v / sum);              // shares that add up to 1
}
softmax([2.0, 1.0, 0.1])   // → [0.66, 0.24, 0.10]

The exam-reference table below lists the ones you'll be asked about. To see their shapes and slopes, open the activation figure in 1.0 Step 6.

"Saturates" means the curve goes flat at the ends, so its slope (and the gradient flowing back through it) is close to 0. FFN = the feed-forward block inside each transformer layer.

Loss functions: how "wrong" is measured

The loss turns "how wrong was that prediction?" into one number for training to shrink. Which loss you use depends on the task:

Optimization knobs

These are the hyperparameters and tricks that decide whether training converges. Each one has a classic failure mode.

When the learning signal fades or explodes

Backpropagation walks from the output back to the input, multiplying by one factor per layer. Multiply many numbers below 1 and you get almost nothing: the sigmoid's slope is at most 0.25, and 0.25 ** 10 ≈ 0.000001. The early layers receive almost no gradient, so they never learn. That's the vanishing gradient problem, a classic for RNNs. Multiply many numbers above 1 instead and the gradient blows up: exploding gradients, and the loss turns into NaN.

• Fixes for vanishing: ReLU/GELU instead of sigmoid/tanh, residual connections (a shortcut that adds a layer's input to its output, x + f(x)), LayerNorm, careful initialisation. In RNNs, LSTM gates help.
• Fix for exploding: gradient clipping (and sane initialisation).

Dropout: practising with neurons switched off

Big networks overfit easily (see 1.1). Dropout is a regularization trick against that. On every training step it switches off a random fraction p of the neurons, so the network can't lean on any single one and has to spread what it learns. At inference every neuron is on.

Think of it as chaos engineering for a network: kill random nodes in staging so production doesn't depend on one of them.

Architectures to recognise

An architecture is a standard way of wiring layers for a kind of data. Know each name and what it's for:

The free NVIDIA Fundamentals of Deep Learning notebooks cover MNIST MLP → CNN → data augmentation → pretrained models / transfer learning → sequence models.

1.3 Classic NLP (the “surprise” topics)

Before this: 1.0 Start here (features, labels and self-supervised learning).

The idea in plain English

Models only eat numbers, and text is a string. Classic NLP (natural language processing) is the set of steps that turns text into numbers a model can learn from. It's an ETL job for text: split the string into words, clean them up, count them, and load the counts into a numeric table.

Before LLMs, every spam filter and sentiment classifier was built this way. The exam still asks about it, which surprises people who only studied LLMs. LLMs do the same job, but they learn most of the steps instead of having them hand-built.

The pipeline, stage by stage

Text flows left to right through five stages. Each one is a small, separate transformation:

• Tokenize means splitting text into pieces called tokens, usually words or parts of words.
• Stop words are very common words such as "the", "is" and "and". They carry little meaning, so classic pipelines often drop them.

The figure runs one sentence through every stage so you can see what each one changes.

Stemming vs lemmatization

Both map the variants of a word to one form, so "studies" and "studying" count as the same feature. They differ in how they do it:

Both shrink the vocabulary and the sparsity for BoW/TF-IDF models. You plug them into sklearn's CountVectorizer(tokenizer=…), as in the study-guide reading.

Words as counts: bag of words and TF-IDF

The simplest way to turn a document into numbers is to count words. One-hot / BoW (bag of words) gives each vocabulary word its own column and stores how often it appears. The result is sparse (mostly zeros), has no word order and no semantics: "dog bites man" and "man bites dog" look identical. n-grams count runs of n words ("not good") to keep a little order.

Raw counts over-reward words like "the". TF-IDF = tf × log(N/df) fixes that by rewarding distinctive words:

• tf (term frequency): how often the word appears in this document;
• idf (inverse document frequency): log(N/df), where N is the number of documents and df is how many contain the word. A word in every document gets idf = log(1) = 0.

Words as meaning: word embeddings

Counts know nothing about meaning: to TF-IDF, "printer" and "copier" are unrelated. An embedding fixes that by giving each word a dense list of numbers (a vector), learned so that words used in similar contexts end up close together.

word2vec (2013) learns these vectors from raw text with no labels, by turning each sentence into prediction tasks. It comes in two flavours:

• CBOW (continuous bag of words) predicts the word from its context. Fast, good for frequent words.
• Skip-gram predicts the context from the word. Better for rare words.

To keep training cheap it uses negative sampling: instead of scoring every word in the vocabulary, it compares the true neighbour with a few random "wrong" words.

Once trained, directions in the space carry meaning, so vector arithmetic works: king − man + woman ≈ queen.

Other names to know:

• GloVe: global co-occurrence matrix factorization (it learns from word-pair counts across the whole corpus).
• fastText: char n-grams (pieces of words), so it handles OOV (out-of-vocabulary) words.
• All of these are static embeddings: one vector per word, whatever the sentence. ELMo/BERT produce contextual embeddings: "bank" gets a different vector in "river bank" and "bank loan".
• WordNet ≠ embeddings. It is a hand-curated lexical database of synsets (synonym sets), hypernyms (is-a links: car → motor vehicle) and antonyms.

Subword tokenization: how LLMs split text

Word-level vocabularies break on new words: any word not seen in training is unknown. Single characters never break but make sequences very long. Subword tokenization sits in between: frequent words become one token, and rare words split into reusable pieces.

BPE (byte-pair encoding) builds that vocabulary greedily. Start from single characters, find the most frequent adjacent pair, merge it into a new token, and repeat. GPT models use byte-level BPE, BERT uses WordPiece, and T5 and Llama 2 use SentencePiece. Tokens ≠ words, and context windows, latency and API cost are all counted in tokens.

NLP tasks and tools

These are the standard jobs the exam names, with the tool you'd reach for. spaCy and NLTK are the two classic Python NLP libraries.

GPE = geopolitical entity (countries, cities). NER = named entity recognition. Extractive QA copies the answer span out of a given passage; generative QA writes a new answer. NLI (natural language inference) decides whether one sentence implies another, which lets a model classify text it was never trained on.

1.4 The Transformer, deeply most tested

Before this: 1.0 Start here (the neuron and training loop) and 1.3 Classic NLP (tokens and word embeddings).

The idea in plain English

An LLM such as GPT or Llama is a Transformer that does one thing: given some text, it predicts the next token (roughly, the next word piece). To write a whole answer it predicts one token, appends it to the input, and runs again. That loop is called autoregressive generation.

Inside, the Transformer is a pipeline. Think of the input as an array with one object per token. Each layer is like middleware: it reads the whole array, lets every token gather information from the other tokens, and writes an update back into each object. After many layers, the object for the last token holds enough context to guess what comes next.

The rest of this lesson walks through that pipeline in five steps. The exam reference details come after.

Step 1 · Text becomes tokens

A tokenizer splits the text into sub-word pieces and maps each piece to an integer ID from a fixed vocabulary. “Bangalore is the capital of” becomes six IDs, because “Bangalore” splits into “Bang” + “alore”. The model never sees letters, only these IDs. (Tokenizers such as BPE are covered in 1.3.)

Step 2 · Tokens become vectors (embeddings), plus position

Each ID is used as a row index into a learned embedding table, a lookup of shape [vocab_size, d_model]. Out comes a vector of d_model numbers per token (768 in GPT-2 small and BERT-base). Words with related meanings end up with similar vectors.

The next step, attention, treats its input as an unordered set, so “dog bites man” and “man bites dog” would look the same. To fix that, a positional encoding (a vector that depends on the token's position) is added to each embedding. The options are compared further down.

Step 3 · Attention: every token looks at every other token

Take “The animal didn't cross the street because it was too tired”. To understand “it”, the model must work out that it means “animal”, not “street”. Self-attention is the mechanism that lets the vector for “it” pull in information from the vector for “animal”.

It works like a search over the sentence. Each token produces three vectors, each by multiplying its embedding by a learned weight matrix:

• Query (Q): what this token is looking for (“it” looks for the noun it refers to).
• Key (K): what this token offers, used to match against queries.
• Value (V): the content this token hands over if it is picked.

A query is compared with every key. Good matches get high scores, the scores become weights that add up to 1, and the token's new vector is the weighted average of all the values.

The figure below uses that sentence with hand-made weights, so you can see what “it” attends to.

The attention formula, one piece at a time

1. Each token embedding x is projected: Q = xW_Q, K = xW_K, V = xW_V.
2. Scores = Q·Kᵀ: a dot product of every query with every key, i.e. how relevant each token is to each other token.
3. ÷ √d_k (d_k is the length of each key vector): keeps the variance of the scores near 1 so softmax doesn't saturate. A saturated softmax gives tiny gradients, so the layer stops learning.
4. Mask: the causal mask sets future positions to −∞ in decoders, so a token can't peek at words that come after it. The padding mask ignores pad tokens.
5. Softmax turns each row of scores into weights that sum to 1.
6. The weighted sum of V gives each token a context-aware representation.
7. Multi-head: h attention “heads” run in parallel, each with its own W_Q, W_K, W_V (e.g., 12 heads × 64 dims = 768). Their outputs are concatenated and projected by W_O. Each head can learn a different relation, such as “which noun does this pronoun mean” or “what was the previous word”.

The next figure runs exactly this formula on four tokens with tiny numbers, so you can check every step yourself.

Cost. Every token is compared with every other token, so attention costs O(n²·d) in sequence length n. That is why long contexts are expensive. Fixes: FlashAttention (IO-aware, still exact), sliding-window or sparse attention, and MQA/GQA (smaller KV cache, see below).

Step 4 · One layer = attention + feed-forward, wrapped in residuals

A Transformer block (one layer) has two sub-layers. Attention lets tokens exchange information. Then a feed-forward network processes each token on its own. Both are wrapped the same way, and the block is stacked N times (12 in GPT-2 small, 96 in GPT-3).

• Feed-forward network (FFN): a position-wise 2-layer MLP. It expands to ~4× the width, applies a nonlinearity (GELU or SwiGLU), and projects back (d → 4d → d), for each token independently. It holds most of the parameters and is thought to store much of the model's “knowledge”. In Mixture-of-Experts models it is replaced by many expert FFNs plus a router (1.5).
• Residual connection: each sub-layer computes x + Sublayer(LN(x)) (pre-LN). The input is passed through unchanged and the sub-layer only adds a delta, which gives gradients a highway through deep stacks.
• LayerNorm: rescales each token's features to a standard range. It normalizes per token (not across the batch), so it works with variable-length sequences and batch size 1. RMSNorm drops the mean-centering and is cheaper.

Step 5 · Output: scoring every possible next token

After the last block, the vector at the last position goes through a final norm and a linear layer called the LM head. It produces one raw score (a logit) for every token in the vocabulary: 50,257 scores for GPT-2. Softmax turns the scores into probabilities. A decoding rule (greedy, temperature, top-p; see 1.6) picks one token, which is appended to the input, and the whole pipeline runs again.

Putting it together

Here is one decoder block, GPT-style, read from the bottom up. The dashed box repeats N times.

The next figure animates this whole pipeline on a real-sized model, showing the shape of the data at each stage.

Positional encoding: how the model knows word order

Attention is permutation-invariant (shuffle the tokens and the outputs shuffle the same way), so order must be injected. Four schemes come up:

• Sinusoidal (the original 2017 paper): fixed, no parameters. PE(pos,2i) = sin(pos/10000^(2i/d)), and cos for odd dimensions. Because it is a formula, it can extrapolate to positions not seen in training.
• Learned absolute (BERT, GPT-2): a trained table with one vector per position, up to a maximum length.
• RoPE (rotary): rotates Q and K by a position-dependent angle, so attention scores depend on relative position (Llama, Mistral, Nemotron).
• ALiBi: adds a linear distance bias to the attention scores.

Three families of Transformer

The same building blocks are arranged in three ways. Knowing which family a model belongs to, and what it is good at, is a very common exam question.

• Encoder-only (BERT, RoBERTa, DeBERTa, embedding models): bidirectional, so every token sees the whole sentence. Pretrained with MLM (masked language modeling: fill in hidden words). Used for classification, NER, embeddings and extractive QA.
• Decoder-only (GPT, Llama, Mistral, Nemotron): causal LM, so each token only sees earlier tokens. Trained to predict the next token. Used for generation, chat and code.
• Encoder-decoder (original Transformer, T5, BART): seq2seq. The encoder reads the input, and the decoder writes the output while attending to it through cross-attention. Used for translation and summarization.

Generating fast: the KV cache

When generating, each new token needs the keys and values of every earlier token. Those never change (the causal mask means old tokens never see new ones), so instead of recomputing them on every step the model stores them. That store is the KV cache: keep past K/V and compute only the new token each step.

• The cache grows linearly with context length × batch size, and it often limits how many users one GPU can serve.
• MQA/GQA (multi-query / grouped-query attention) let several query heads share K/V heads, which shrinks the cache.
• A paged KV cache stores it in fixed-size blocks to avoid memory fragmentation (more in Inference optimization).

1.5 LLM families, landmarks & scaling

Before this: 1.4 The Transformer (decoder-only models and the feed-forward layer).

The idea in plain English

A large language model is a decoder-only Transformer trained on a huge pile of text to predict the next token. Nothing more is built in. Grammar, facts and some reasoning appear because they help predict the next token across trillions of examples.

Think of it like a generic library. You train it once, at great cost, on broad data. Then many teams reuse it for their own tasks by prompting it, adding retrieval, or fine-tuning it. This lesson covers what kinds of LLMs exist, why bigger usually means better, and the key milestones the exam expects you to recognise.

Foundation models, base models and instruct models

• Foundation model: large, pretrained on broad data, and adaptable to many tasks (NVIDIA blog on the reading list).
• Base vs instruct/chat model: a base model just continues text. Ask it a question and it may write three more questions. An instruct model went through SFT (supervised fine-tuning on example conversations) plus preference tuning (such as RLHF or DPO), and follows instructions using a chat template. See the customization ladder for how that tuning is done.

Scaling laws: bigger model, more data, lower loss

Scaling laws say that loss (the model's prediction error) falls smoothly, as a power law, as you increase parameters, data and compute. Doubling the model doesn't halve the error, but the improvement is predictable enough to plan a training run with.

For a fixed compute budget you must trade model size against training data. Chinchilla (2022) found the compute-optimal balance is about 20 training tokens per parameter. GPT-3 was trained on far less data than that. Modern small models are deliberately “over-trained” well beyond 20:1, because a smaller model is cheaper to serve at inference.

Mixture of Experts: a bigger model at the same cost per token

In a normal (dense) model, every parameter is used for every token. A Mixture of Experts (MoE) model replaces the feed-forward network in each block with several “expert” FFNs plus a small learned router. For each token the router picks the top few experts (often 2) and only those run.

The result is a model with many total parameters (capacity) but roughly the compute per token of a much smaller dense model. The catch: all experts must still sit in GPU memory, and training needs a load-balancing loss so the router doesn't send every token to the same few experts. Mixtral 8×7B is the standard example: about 47B parameters in total, about 13B active per token.

Emergent abilities and in-context learning

Emergent abilities are skills that show up only once models get large enough. The best-known is in-context learning: show the model a few examples in the prompt and it does the task, with no weight updates. This few-shot ability appeared with scale (GPT-3). It is the basis of few-shot prompting in 1.6.

Beyond text: multimodal and diffusion models

• Multimodal: vision-language models (an image encoder feeding an LLM), and CLIP contrastive image-text embeddings. Image generation uses diffusion.
• Diffusion (reading list): the forward process adds Gaussian noise over T steps. A network (U-Net or transformer) learns to predict the noise. Generation starts from noise and denoises iteratively. Text conditioning works via cross-attention, and latent diffusion runs in a VAE latent space (a compressed image representation) to save compute.

Landmarks timeline

You don't need exact dates. Do know what each landmark introduced, especially the bold ones and anything marked as on the reading list.

1.6 Prompt engineering & decoding

Before this: 1.4 The Transformer (the model outputs a probability for every possible next token).

The idea in plain English

You can't recompile an LLM, but you control its whole input. A prompt is that input: the spec, the reference data and the test cases for a function you can only call. Prompt engineering is writing that input so the model reliably does what you want. It does not change any weights.

The second half of this lesson is about the other knob you control: decoding. At every step the model gives a probability to each possible next token. Decoding settings decide which one actually gets picked: always the top one, or a weighted random draw.

Anatomy of a good prompt

1. Role / system: “You are a support agent for…”. Usually sent as the system prompt.
2. Instruction: clear, specific, positive (“do X”, not only “don't Y”).
3. Context: retrieved docs, data, constraints.
4. Examples: zero-, one- or few-shot.
5. Output format: JSON schema, bullet list, length.
6. Delimiters: separate instructions from untrusted data (like a parameterised SQL query instead of string concatenation).

Treat prompts like code. Test them on a set of cases, compare versions, and keep them under version control. Prompt templates (LangChain PromptTemplate) and model-specific chat templates help keep them consistent.

The figure lets you add these parts one at a time to a real task, triaging a support ticket.

Prompting techniques

Showing versus telling. Zero-shot means instruction only. Few-shot means you include k worked demonstrations. The model picks up the pattern from them; this is in-context learning.

Getting better reasoning. These help with math, logic and multi-step problems:

• Chain-of-thought (CoT): ask the model to “think step by step”, or give worked examples that show the steps. Zero-shot CoT is just adding that phrase.
• Self-consistency: sample several chains of thought and take the majority answer.
• Tree-of-Thoughts: branch into several reasoning paths, evaluate them, and keep the best.
• Generated knowledge (have the model list relevant facts first) and self-critique / reflection (have it review and fix its own answer).

Breaking work up and using tools.

• Prompt chaining: break a task into sequential calls, each feeding the next, like a Unix pipe.
• ReAct: a Thought → Action (call a tool) → Observation loop. This is the basis of agents (Building LLM apps).

How the model picks the next token: decoding

For each step the model outputs one raw score per vocabulary token, called a logit. Softmax turns the logits into probabilities. Then a decoding rule chooses:

• Greedy decoding: always take the most likely token (argmax). Deterministic, but it can get stuck in loops.
• Sampling: make a weighted random draw, so you get variety. Three knobs shape the draw:
  • Temperature T divides the logits before softmax. T<1 makes the distribution sharper (safer), T>1 flatter (more creative), and T→0 is the same as greedy.
  • Top-k keeps only the k most likely tokens.
  • Top-p (nucleus) keeps the smallest set of tokens whose probabilities add up to at least p.

None of these knobs add knowledge. They only change how the model picks from what it already predicts.

Beam search is the deterministic alternative to greedy. Instead of committing to the single best token at each step, it keeps the B best partial sequences (beams) and picks the sequence with the highest overall probability at the end. It suits translation and summarization, but the output tends to be bland for chat.

Decoding cheat sheet

Exam trap: p-tuning is not prompt engineering

1.7 Embeddings, vector search & RAG high yield

Before this: 1.6 Prompting (what goes into a prompt) and 1.3 Classic NLP (word embeddings).

The idea in plain English

An LLM only knows what was in its training data. It doesn't know your company's HR policy, last week's tickets, or anything private. Retraining it every time a document changes is far too slow and expensive.

RAG (retrieval-augmented generation) turns the problem into an open-book exam. When a question arrives, search your documents for the most relevant passages, paste them into the prompt, and have the model answer from them, with citations. It is a lookup before the LLM call: ctx = search(question); answer = llm(prompt(ctx, question)). The model's weights never change.

To make that search work on meaning rather than exact words, you need embeddings. The lesson builds up in four steps: embeddings, chunking, indexes, then the full pipeline.

Step 1 · Embeddings: meaning as a vector

An embedding model turns a piece of text into a dense vector, a list of a few hundred to a few thousand numbers. It is trained so that texts with similar meaning get vectors pointing in similar directions. “I was charged twice” and “duplicate payment on my card” end up close together even though they share no words.

• Cosine similarity = A·B/(‖A‖‖B‖): the cosine of the angle between two vectors (1 = same direction). On L2-normalized vectors (length 1), dot product = cosine and ranks the same as Euclidean distance.
• Choose the model by retrieval benchmarks (MTEB), domain, language, dimension (cost), max input length and license.
• Changing the embedding model means re-embedding the whole corpus: vectors from different models are not comparable.
• Asymmetric models use different query vs passage prefixes (e.g., input_type="query"/"passage" in NVIDIA embedding NIMs).

Dense search (comparing embeddings) is bad at one thing: rare exact strings such as product codes or error numbers carry little “meaning”. Classic keyword search (BM25, a smarter grep over an inverted index) is good at exactly those. Hybrid search runs both and merges the rankings.

Step 2 · Chunking: splitting documents into searchable pieces

You don't embed a 60-page handbook as one vector; it would mix every topic into a blur. Documents are split into chunks, typically 256–1,024 tokens with 10–20% overlap, and each chunk gets its own embedding. Keep metadata (file, page, section) with each chunk so you can cite it and filter by it.

• Too small: a fact gets cut off from the sentence that explains it.
• Too large: each embedding mixes many topics, so relevance gets diluted, and retrieved chunks eat the context window.
• Overlap: a sentence that straddles a boundary survives intact in at least one chunk.

Step 3 · Vector indexes: finding the nearest vectors fast

Comparing the query with millions of vectors one by one is too slow, so vector databases use ANN (approximate nearest neighbour) indexes. They trade a little recall for big speed gains.

• Flat: exact brute force; fine for small corpora.
• IVF: k-means clusters; search only the nearest nprobe lists.
• PQ (product quantization): compress vectors into sub-codebooks, so memory goes down.
• HNSW: multi-layer proximity graph; fast and high recall, but memory-hungry.
• GPU: NVIDIA cuVS (CAGRA, IVF-PQ) runs inside Milvus and FAISS.

Step 4 · The full RAG pipeline

A RAG system has two halves. The ingestion half runs offline, whenever documents change:

The query half runs for every question:

The figure steps through both halves for one HR question, and lets you compare against a plain LLM with no retrieval.

When to use RAG, fine-tuning or prompting

• RAG: fresh, private or changing knowledge; citations; no retraining; access control.
• Fine-tuning: behavior, style, format, domain jargon, narrow tasks. It doesn't keep facts fresh.
• Prompting: quickest; instructions and format.
• Combine them: a fine-tuned model + RAG.

When RAG goes wrong: failure → fix

• Facts split across chunks → bigger or overlapping, structure-aware chunks.
• Exact codes and names missed → hybrid BM25 + dense.
• Relevant chunk ranked 9th → reranker, larger k.
• Stale answers → re-index pipeline, timestamps.
• Answer ignores context → stronger instructions, fewer and better chunks, faithfulness rail.
• “Lost in the middle” (models use facts in the middle of a long context less reliably than facts at the start or end) → put the best chunks first or last; fewer chunks.
• Injection in documents → treat context as data, retrieval rails.

Patterns beyond basic RAG

• Better queries: query rewriting / HyDE (generate a hypothetical answer and search with it) · multi-query.
• Better context: parent-document (small-to-big) retrieval · GraphRAG · caching.
• Agentic RAG: the model decides when and what to retrieve.
• Conversational RAG: condense the chat history into a standalone question before retrieving.
• Chatbots add memory (a buffer or summary of history) within the context window. Summarizers on long docs use map-reduce, refine, or long-context models.

1.8 The Python toolbox (objectives 1.6 / 1.10)

The idea in plain English

The exam asks “which library would you use for X?”. Treat this lesson as a dependency map: one row per library, what it is for, and the one detail worth remembering. You don't need to know any API by heart.

Most of the stack sits in layers. NumPy arrays are at the bottom. pandas and scikit-learn build on them. Deep learning frameworks (PyTorch, TensorFlow, Keras) sit beside them. The LLM libraries (Hugging Face, LangChain, vector stores) sit on top. NVIDIA's RAPIDS libraries are drop-in GPU versions of the bottom layer.

Data and classic ML

Classic NLP

Deep learning and LLMs

Building LLM apps

Charts and experiment tracking

2.1 Classification metrics

Before this: 1.0 Start here (what a model and a prediction are) and 1.1 ML foundations (train/test split).

The idea in plain English

A classifier is a function that answers a yes/no question: is this email spam, is this card payment fraud? To grade it, you run it on a test set where you already know the true answers and tally the results, like a CI test report.

The catch is that there are two different ways to be wrong. Think of an alerting rule. It can page you when nothing is broken (a false alarm), or stay silent during a real outage (a miss). Which mistake hurts more depends on the product, and that choice decides which metric you should optimize.

The confusion matrix: four buckets

Every prediction lands in one of four cells. “Positive” means “the thing we are looking for” (fraud, spam, disease), not “good”.

This table is called the confusion matrix. Every metric below is just a different ratio of these four counts.

Five metrics from four counts

Read each formula as a question:

• Accuracy = (TP+TN) / all Of all predictions, how many were right?
• Precision = TP / (TP+FP) Of everything I flagged, how much was really positive? (Low precision = lots of false alarms.)
• Recall (TPR, sensitivity) = TP / (TP+FN) Of all the real positives, how many did I catch? (Low recall = lots of misses.) Also called the true positive rate.
• F1 = 2·P·R / (P+R) One number that combines precision and recall. It is their harmonic mean, so it is low if either one is low.
• Specificity = TN / (TN+FP); FPR = 1 − specificity Of all the real negatives, how many did I correctly leave alone? FPR is the false positive rate.

Why accuracy lies on imbalanced data

Suppose 1 in 100 payments is fraud. A “model” that always answers not fraud is 99% accurate, yet it catches zero fraud: its recall is 0. When one class is rare (class imbalance), accuracy is dominated by the easy majority class. Use precision, recall, F1, PR-AUC and per-class reports instead.

The threshold: trading precision for recall

Most classifiers don't output yes/no directly. They output a score between 0 and 1, and your code applies a classification threshold: flag = score >= 0.5. The threshold is a config value, not part of the model.

Lower the threshold and you flag more: you catch more real positives (recall ↑) but also raise more false alarms (precision ↓). Raise it and the opposite happens. Moving the threshold trades precision against recall. The figure lets you drag it on a fraud model.

Scoring across all thresholds: ROC and PR curves

Picking one threshold hides how good the model's ranking is overall. So we sweep every threshold and plot the result:

• The ROC curve plots the true positive rate (recall) against the false positive rate. ROC-AUC is the area under it: a threshold-independent measure of ranking quality. 0.5 is chance (random guessing), 1.0 is perfect.
• The precision-recall curve plots precision against recall; its area is PR-AUC. On imbalanced data it is the more honest of the two, which is why it appears in the advice above.

More than two classes: macro, micro, weighted

With several classes (say billing, bug and feature-request tickets), you get a precision and recall per class and need to merge them into one number:

• Macro average weights classes equally: average the per-class scores. A rare class counts as much as a common one.
• Micro average pools counts: add up all TPs, FPs and FNs first, then compute once. Big classes dominate.
• Weighted average averages per-class scores weighted by support (the number of true examples in each class).

Choosing the metric (exam scenarios)

2.2 Regression & language-model metrics

Before this: 2.1 Classification metrics. Helpful: 1.3 Classic NLP (tokens and n-grams).

The idea in plain English

The last lesson graded yes/no answers. Here the outputs are different, so the grading is too.

A regression model predicts a number: delivery time in minutes, tomorrow's GPU-hours. You grade it by how far off it was, like measuring how much an ETA estimate missed by.

A language model produces text, and there is rarely one right answer. So you either measure how well it predicts real text (perplexity), compare its output word by word with a human-written reference (BLEU, ROUGE), or ask a judge (a person or another LLM).

Regression: how far off were we?

For each example, the error is y − ŷ: the actual value minus the predicted one. The metrics differ only in how they combine those errors.

Mean absolute error: the average size of a miss, in the same units as the target (“off by 3 minutes on average”). Robust to outliers.

Mean squared error squares each miss before averaging, so it is in units² and big misses dominate: one 30-minute miss counts as much as 900 one-minute misses. RMSE takes the square root to get back to the target's units, but it stays outlier-sensitive. MSE/RMSE punish large errors more.

R² is the “proportion of explained variance” named in exam objective 2.2. 1 = perfect, 0 = no better than always predicting the mean, and it can be negative (worse than the mean, possible on test data).

The figure fits a line to delivery times. Drag one order into a traffic jam and compare how each metric reacts.

Perplexity: how surprised is the model by real text?

A language model predicts the next token by assigning a probability to every possible one. To score it, feed it real text and look at the probability p it gave to each token that actually came next.

• Surprise for one token is −ln p. A token it gave 0.9 costs almost nothing; one it gave 0.01 costs a lot.
• The average surprise per token is the cross-entropy, the same loss number the training loop prints.
• Perplexity = exp(mean cross-entropy). It turns that loss into a friendlier unit: a perplexity of 4 means the model was, on average, as unsure as if it were choosing between 4 equally likely words.

Lower is better (1 = always certain and right). Two catches: perplexity is only comparable between models with the same tokenizer evaluated on the same data, and it measures prediction, not whether answers are helpful, safe or true.

Comparing output to a reference: BLEU and ROUGE

When you have a human-written “correct” answer (a reference translation or summary), you can count how many n-grams (runs of n words) the model's output shares with it. Think of it as a fuzzy diff: cheap, deterministic, and blind to meaning.

• BLEU: n-gram precision (1–4-grams) × a brevity penalty so that very short outputs can't cheat. Classic for machine translation. The original paper (Papineni et al., 2002) is on the exam reading list.
• ROUGE-1/2/L: n-gram / longest-common-subsequence recall, i.e. how much of the reference the output covers. Classic for summarization. ROUGE-1 and ROUGE-2 count single words and word pairs; ROUGE-L uses the longest common subsequence.

Beyond word overlap

Because overlap metrics miss paraphrases and meaning, other metrics fill the gaps:

Benchmarks and evaluation workflows are covered in 3.4 Evaluation & benchmarking.

2.3 Visualization chooser

The idea in plain English

Choosing a chart is like choosing a data structure: the access pattern decides. A hash map isn't better than a list in general; it is better for lookups. Likewise, a chart isn't good or bad in general; it answers one kind of question well.

So before plotting anything, say the question in one sentence: “How are API latencies spread out?”, “Did loss go down over training?”, “Which model scores highest?”. The question names the chart.

The everyday charts

• Histogram: split a numeric range into bins and count values per bin. Shows the shape of one variable: where the peak is, whether it is skewed (a long tail on one side), and whether it has several peaks (modes). A KDE (kernel density estimate) is the smoothed-curve version.
• Box plot: a five-number summary per group: the box spans the middle 50% of values, the line inside is the median, the whiskers show the normal range, and dots beyond are outliers. Great for comparing many groups side by side. A violin plot adds the distribution's shape.
• Scatter plot: one dot per example, x against y. Shows whether two numeric variables move together (add a trend line to make it obvious).
• Line chart: a value over time or steps, such as training loss per epoch or p95 latency per day.
• Bar chart: one bar per category, for comparing counts or scores (class counts, benchmark scores per model).
• Heatmap: a grid coloured by value. Used for pairwise correlations between features and for a confusion matrix.

Question → chart

The same choices as a lookup table. This is the version to memorize.

The figure turns this table into pictures. For each question it draws the right chart from sample data, next to a tempting worse choice.

Two ML-specific cases

Embeddings. An embedding is a list of hundreds of numbers per item, and you can't plot 768 axes. You first squeeze it down to 2 numbers per item, then draw a scatter plot. PCA does this linearly and keeps the directions of most variance; t-SNE and UMAP are nonlinear and keep each point's neighbours close, so clusters show up well. How they work is explained in 2.4 EDA.

Classifier trade-offs. To show how a classifier behaves at every threshold, use a ROC curve or a precision-recall curve (see 2.1 Classification metrics).

Charts to be careful with

• A single average (one bar or one number) hides the shape of the data, including the slow tail users actually feel.
• Pie charts are hard to compare by eye. For part-of-a-whole, prefer a stacked bar; use a pie only with 2–4 slices.
• A line chart over categories implies an order or trend that isn't there; use bars.

2.4 EDA workflow & factors that distort results

Before this: 2.3 Visualization chooser. Helpful: 1.1 ML foundations (train/test split, data leakage).

The idea in plain English

EDA (exploratory data analysis) is reading the data before you trust it, the way you'd read an unfamiliar codebase before refactoring it, or skim the logs before writing an alert rule. You profile what's there, find the gaps and the weirdness, and write down what you learned.

The second half of this lesson is about how data can fool you even when the code is correct: a pattern that looks real but is caused by something you didn't measure, or a model that was right last month but wrong today.

The EDA checklist

1. Shape and types: rows, columns, dtypes. In pandas: df.info(), df.describe().
2. Missing values and duplicates: drop them, or fill them in (imputation) using statistics from the training set only, so test data doesn't leak into training.
3. Distributions and outliers: plot histograms. Flag outliers with the IQR rule (more than 1.5 × the interquartile range beyond the middle 50%) or a z-score (how many standard deviations from the mean).
4. Class balance: if one label is rare, stratify your splits (keep the same label ratio in train and test), reweight the classes, or resample (for example SMOTE, which creates synthetic examples of the rare class).
5. Correlations and leakage suspects: Pearson measures linear relationships; Spearman works on ranks, so it catches any steadily rising or falling relationship. A feature that predicts the label suspiciously well is often leakage.
6. Text data: length distribution in tokens, language, encoding issues, PII (personal data such as emails and phone numbers).
7. Document findings with charts.

Seeing many features at once: PCA, t-SNE and UMAP

A dataset with 50 features, or an embedding with 768 numbers per item, can't be drawn directly. Dimensionality reduction squeezes each item down to 2 or 3 numbers so you can scatter-plot it, look for clusters, or feed a smaller input to a model.

PCA (principal component analysis) is the classic way. It finds the direction in which the data is most spread out (the first principal component), then the next most spread-out direction at a right angle to it, and so on. Each component's share of the total spread is its explained variance. Keep the top 2 or 3 and drop the rest: a lossy compression that keeps as much of the spread as possible.

PCA is linear: it can only rotate and drop axes. t-SNE and UMAP are nonlinear: they try to keep each point's near neighbours close in 2D, so clusters separate well. That makes them the usual choice for visualizing embedding clusters, but the distances between far-apart clusters in their plots aren't meaningful. GPU versions of all three ship in RAPIDS cuML (2.5).

What can fool you (objective 2.5)

These are the classic ways a correct analysis reaches a wrong conclusion. Each one has a short example.

Correlation ≠ causation, and confounders

Two things moving together doesn't mean one causes the other. Ice-cream sales and drownings rise together, because hot weather drives both. That hidden third variable is a confounder. Also, a correlation of 0 only means no linear relationship: a strong curved one can still be there, so plot your data.

Simpson's paradox

A trend can reverse when groups are combined. In a famous kidney-stone study, treatment A had the higher success rate for small stones and for large stones, yet B looked better overall, because A was mostly given the hard (large-stone) cases. The group mix is a confounder. The same trap hits model evaluation: one model can win on every data slice and still lose on the pooled metric.

The other usual suspects

• Selection / sampling bias: your sample isn't representative (only users who opted in to telemetry). Survivorship bias is a special case: you only see what survived (only the startups still around, only the tickets that weren't auto-closed).
• Data leakage; train/test contamination: information from the test set or the future sneaks into training, so the score is too good to be true.
• Class imbalance hiding behind accuracy (see 2.1).
• Small samples, so there's no statistical significance: the difference you see could easily be noise.
• Outliers dominating MSE or means: one huge value drags the average (see 2.2).

Data drift and concept drift over time

A trained model is a snapshot of the world, like a cache that never invalidates. In production the world moves on:

• Data drift (covariate shift): the inputs change. Users start writing longer messages, or a new product line appears.
• Concept drift: the inputs look the same but the right answer changes. The same question about a refund policy now has a different correct reply.

Detect drift by comparing live data with the training baseline using statistical tests (PSI, KL divergence, the KS test) and by monitoring performance on freshly labelled samples. Respond by investigating, refreshing the data, retraining or fine-tuning, and re-evaluating.

2.5 GPU-accelerated data science (RAPIDS)

Before this: 1.8 The Python toolbox (pandas, scikit-learn). Helpful: 2.4 EDA (PCA, t-SNE, UMAP).

The idea in plain English

pandas and scikit-learn run on the CPU, which has a handful of fast cores. A GPU has thousands of simpler cores, which is ideal when you do the same operation to millions of rows: filter, group by, join, compute a distance.

RAPIDS is NVIDIA's open-source set of GPU libraries that copy the APIs data scientists already know. It is like swapping a database driver for a faster one without rewriting your queries: same calls, different engine underneath. Here's what each library replaces.

cuDF: pandas on the GPU

cuDF is a GPU DataFrame with a pandas-like API. It stores data in the Apache Arrow memory format (a standard column-oriented layout).

• %load_ext cudf.pandas (in a notebook) or python -m cudf.pandas script.py accelerates existing pandas code with zero code changes.
• Operations cuDF doesn't support fall back to the CPU automatically, so the code still runs.
• NVIDIA cites speedups of up to ~150×.
• cuDF also powers the Polars GPU engine.

cuML: scikit-learn on the GPU

cuML offers scikit-learn-style GPU machine learning: linear and logistic regression, random forest, k-means, DBSCAN, PCA, t-SNE, UMAP, k-NN and SVM.

• %load_ext cuml.accel gives zero-code-change acceleration for existing scikit-learn code.
• XGBoost (a popular gradient-boosted trees library) has native GPU support: set device="cuda".

cuGraph and Dask: graphs and scaling out

• cuGraph: GPU graph analytics such as PageRank, Louvain (community detection) and BFS. It is also a NetworkX backend (nx-cugraph), so NetworkX code can run on the GPU.
• Dask / Dask-cuDF: scales the same work across multiple GPUs and nodes when data doesn't fit on one GPU.

Exam lookup: need → tool

NVIDIA's “Fundamentals of Accelerated Data Science” course covers cuDF, cuML, cuGraph, Dask, XGBoost and deploying on Triton (FIL backend).

3.1 The customization ladder (NVIDIA's taxonomy) high yield

Before this: 1.0 Start here (weights and the training loop), 1.6 Prompting and 1.7 RAG.

The idea in plain English

A pretrained LLM is like a large third-party library your product depends on. When it doesn't behave the way you need, you have a menu of fixes, from cheap to expensive: change how you call it (prompting), bolt on a small plugin that you train (prompt learning, LoRA), fork the whole thing and edit it (full fine-tuning), or fork it and add a human review process that keeps steering it (alignment).

NVIDIA calls this menu the customization ladder. It comes from the “Mastering LLM Techniques: Customization” blog on the exam reading list. Each rung down needs more data, compute and expertise, and offers a bigger potential accuracy gain.

The five rungs

Knowledge problem or behaviour problem?

Before climbing, ask what is actually wrong. If the model doesn't know something (your internal wiki, last week's prices), that's a knowledge problem: use RAG, which looks the facts up at request time and can cite them. Fine-tuning facts in gives a frozen snapshot that goes stale. If the model knows enough but answers the wrong way (wrong tone, format, domain habits), that's a behaviour problem: climb the ladder.

The rule: start at rung 1, measure on your own test set, and only climb when the cheaper option falls short. The helper below applies that rule to a few yes/no questions about your project.

The same decisions as exam-style scenarios:

Close-up: how LoRA works

Inside the model, each layer holds big square grids of numbers (weight matrices). In a 7B model one such matrix is about 4096 × 4096, roughly 16.7 million numbers. Full fine-tuning edits all of them.

LoRA (low-rank adaptation) leaves the original matrix W frozen and learns a small patch instead: ΔW = B·A, where B is tall and thin (d × r) and A is short and wide (r × d). The rank r is tiny compared with d (r ≪ d, for example r = 8). With r = 8 the patch has 65,536 trainable numbers instead of 16.7 million. LoRA is typically applied to the attention projections.

At deploy time you have two options. Merge the patch into W (W' = W + ΔW): the result is an ordinary matrix of the same shape, so there is no inference latency. Or keep it separate and swap per-task adapters on one shared base model (multi-LoRA serving). NVIDIA cites ~10,000× fewer trainable parameters and ~3× less memory than full fine-tuning of GPT-3. QLoRA stores the frozen base in 4-bit NF4, so even the read-only part shrinks.

Close-up: teaching a model what people prefer

SFT learns from “here is the right answer” examples, like golden test fixtures. But qualities such as helpful, polite or harmless are hard to write down and easy to judge. So alignment methods learn from preferences: show people two or more answers and record which one is better, like code-review votes.

• RLHF (reinforcement learning from human feedback) has three stages: SFT → a reward model trained on human rankings (an automated reviewer that scores any answer) → PPO, a reinforcement-learning loop that updates the model to earn higher scores, with a KL penalty that keeps it close to the original (reference) model.
• DPO (direct preference optimization) optimizes directly on chosen/rejected pairs: no reward model, no RL loop. Simpler and more stable.
• RLAIF uses AI feedback: an AI judge produces the preference labels instead of humans.
• GRPO uses group-relative rewards: score a group of answers to the same prompt against each other. Popular for reasoning models with verifiable rewards (math, code).
• SteerLM (NVIDIA) is attribute-conditioned SFT: training data is labeled with attributes, and users dial helpfulness, humor, toxicity and so on at inference.

3.2 Training at scale

Before this: 1.0 Start here (the training loop) and 1.2 Neural networks & training (gradients, the Adam optimizer).

The idea in plain English

Training a large model is a distributed-systems problem. Two resources run out: memory on a single GPU (80 GB on an H100) and wall-clock time. The fixes are ones you already know from backend work: use smaller data types, paginate, trade cache for recompute, shard state across machines, replicate workers, and sync them over a fast network.

This lesson builds up in that order. First, what each number costs in bytes. Then, what has to sit in GPU memory during training. Then, how to squeeze more out of one GPU. Finally, how to spread the work over many GPUs and keep them in sync.

Step 1: how many bytes per number?

Every weight is a floating-point number. A float is stored like scientific notation (6.02 × 10²³): a sign bit, exponent bits that set the range (how big or small a value can be), and mantissa bits that set the precision (how many significant digits). Fewer bits means less memory and faster Tensor Core math, but less range or precision. It's the int64-vs-int16 column choice, for floats.

Loss scaling exists because FP16's range is narrow: very small gradients round down to zero (underflow) and are lost. The fix is to multiply the loss by a large factor before the backward pass, then divide the gradients back down before the update. BF16 keeps FP32's 8 exponent bits, so it has the same range and usually doesn't need this.

Mixed precision combines formats: compute in FP16/BF16/FP8 on Tensor Cores, but keep an FP32 master copy of the weights so small updates aren't rounded away. That gives about 2× speed and roughly half the activation memory.

Step 2: what has to fit in GPU memory

Now multiply bytes by parameter count. For inference, you mostly store the weights:

For full fine-tuning with Adam in mixed precision, every parameter drags along four more things, about 16 bytes in total:

• 16 = 2 (FP16 weights) + 2 (gradients) + 4 (FP32 master weights) + 4 + 4 (Adam's two running averages, m and v).
• Plus activations (grow with batch × sequence length × layers) and, at inference, the KV cache (grows with context length × batch).
• LoRA: gradients and optimizer state only for the adapters, so memory ≈ frozen weights + a little.
• QLoRA: 7B at 4-bit ≈ 3.5–4 GB of weights.

That 8× gap between serving and training is why a model you can run on one GPU may need a whole server to fine-tune. Try it below.

Step 3: squeeze more out of one GPU

Before buying more GPUs, there are cheaper tricks:

• Gradient accumulation: a big effective batch from small micro-batches. Run several small batches one after another, add up their gradients, and update the weights once. Like paginating a huge query and keeping a running total: same answer, no memory spike, but slower.
• Activation checkpointing: recompute instead of storing. Throw away most activations during the forward pass and recompute them when backward needs them. Classic cache-vs-recompute: much less memory, roughly 20–33% more compute.
• Mixed precision, PEFT (LoRA) and quantized bases (QLoRA) from earlier.

Step 4: spread the work over many GPUs

When one GPU isn't enough, you split the job. There are three basic ways, and they map onto patterns you know.

• Data parallelism (DP) = replication. Every GPU holds a full copy of the model, like stateless web servers behind a load balancer. Each gets a different slice of the batch. After the backward pass the GPUs average their gradients (an all-reduce) so every copy makes the same update. Use it when the model fits on one GPU and you want more throughput.
• ZeRO / FSDP = data parallelism with sharded state. Plain DP stores the same optimizer state on every GPU, which wastes memory. ZeRO shards it: stage 1 shards the optimizer state (Z1), stage 2 also the gradients (Z2), stage 3 also the parameters (Z3), fetching each piece just in time. PyTorch FSDP is the same idea.
• Tensor parallelism (TP) = sharding one big operation. Each layer's weight matrices are split across GPUs (Megatron-style), like one huge query split across database nodes. The partial results must be combined inside every layer, so it needs very fast links: keep it inside one NVLink node.
• Pipeline parallelism (PP) = a CI pipeline. Layers are split into consecutive stages on different GPUs or nodes; only activations pass between neighbours. Stages wait for each other (the idle time is called the pipeline bubble), so the batch is cut into micro-batches to keep every stage busy.

Two more splits appear in exam questions: sequence / context parallelism for very long inputs, and expert parallelism for Mixture-of-Experts models.

Real LLM training combines them (“3D parallelism” in Megatron-LM / NeMo). Total GPUs = DP × TP × PP (× CP): for example TP 8 inside a node × PP 4 × DP 16 = 512 GPUs.

Step 5: keeping GPUs in sync (all-reduce and NCCL)

In data parallelism, every GPU needs the average gradient after every step. Sending everything to one central server would make that server's network link the bottleneck, the classic “everyone calls one service” problem.

Ring all-reduce avoids the hot spot. N GPUs are arranged in a ring and each one only talks to its neighbour, with every link busy at once:

1. Reduce-scatter (N−1 steps): each GPU cuts its gradient vector into N slices and passes slices around the ring, adding as it goes. Each GPU ends with the full sum of one slice.
2. All-gather (N−1 steps): pass the finished slices around so everyone has every summed slice.

Each GPU sends about 2(N−1)/N × the data, which is nearly constant as N grows. That makes it bandwidth-optimal: adding GPUs barely adds traffic per link.

NCCL (NVIDIA Collective Communications Library) implements ring and tree all-reduce, all-gather, reduce-scatter and broadcast over NVLink/NVSwitch, PCIe and InfiniBand. These operations are called collectives: network calls that every GPU in the group makes together.

3.3 Preparing data for pretraining, fine-tuning & RLHF

Before this: 3.1 The customization ladder (SFT and RLHF) and 1.3 Classic NLP (tokenization).

The idea in plain English

Data preparation is an ETL pipeline. Extract text from messy sources, filter out junk, remove duplicates, scrub secrets, and write clean shards ready for training. The model learns whatever you feed it, including spam, repeated pages, people's phone numbers and the answers to benchmark tests. Garbage in, garbage out applies to model weights too.

The pipeline differs by stage. Pretraining needs trillions of tokens of clean general text. Fine-tuning needs thousands of high-quality example conversations. RLHF needs human judgements about which answer is better.

The pretraining pipeline, step by step

Three steps deserve a closer look:

• Deduplication comes in three strengths. Exact: hash each document and keep one copy per hash (like SELECT DISTINCT). Fuzzy: catch near-copies (same article, different footer) with MinHash + LSH, which compares short signatures instead of every pair of documents. Semantic: compare embeddings to catch paraphrases. Duplicates waste compute and make the model memorize.
• PII redaction masks personal data such as emails, phone numbers and IDs, like scrubbing secrets from logs.
• Decontamination removes benchmark test questions from the training data. Otherwise the model “passes” tests it has already seen the answers to.

NVIDIA NeMo Curator performs every one of those steps on GPUs at scale (RAPIDS-accelerated; Ray/Dask).

Fine-tuning data

Fine-tuning data is a set of instruction/response pairs, usually stored as JSONL (one JSON object per line) and formatted with the model's chat template (the exact markers that separate system, user and assistant turns). It should be diverse and high-quality, deduplicated, and decontaminated against your eval set.

Quality beats quantity: the LIMA paper fine-tuned a strong model with only 1k carefully chosen examples.

RLHF (preference) data

For RLHF you collect prompts + multiple responses, then humans rank or compare them, producing chosen/rejected pairs. To keep the labels trustworthy:

• Write clear annotator guidelines.
• Measure inter-annotator agreement: if two labellers often disagree, the task or guidelines are unclear.
• Get consent; pay and protect annotators.

Synthetic data and augmentation

Synthetic data: generate examples with a strong model, then filter them with a reward model or judge. NVIDIA's example is the Nemotron-4 340B Instruct + Reward pipeline; NeMo Data Designer is the tool. Check the generating model's license before training on its output.

Data augmentation creates variants of existing examples: back-translation (translate to another language and back), paraphrase, synonym swap.

Why tokenization matters

The tokenizer decides how text is cut into tokens, and that affects:

• Sequence length: more tokens per sentence means more cost, less fits in the context window, and more truncation.
• Vocabulary coverage of your domain and of other languages. A tokenizer trained on English splits Hindi or chemistry names into many tiny pieces.
• Perplexity comparability: perplexity is per token, so models with different tokenizers can't be compared on it directly.

Train or extend a tokenizer for new languages or domains (HF tokenizer.train_new_from_iterator).

3.4 Evaluation & benchmarking

Before this: 2.2 Regression & language-model metrics (perplexity, BLEU, pass@k) and 1.7 RAG.

The idea in plain English

A benchmark is a public test suite for models: a fixed set of questions with known answers and a scoring rule. Leaderboards are CI dashboards where everyone runs the same suite. They're useful for comparing models, but passing someone else's tests doesn't prove a model passes yours.

So evaluation has two layers. Use standard benchmarks to shortlist models, then build a small domain eval set from your real tasks (your support tickets, your documents) and measure on that.

The benchmark map

Group them by what they test: language understanding (GLUE, SQuAD), knowledge and reasoning (MMLU, HellaSwag, ARC, GSM8K), code (HumanEval), honesty (TruthfulQA), chat quality (MT-Bench, Arena), and specialist suites for embeddings (MTEB) and RAG (RAGAS).

Tools for running evaluations

• EleutherAI LM Evaluation Harness: runs the standard academic benchmarks with one command (on the reading list).
• NeMo Evaluator: NVIDIA's SDK/microservice for benchmarks + LLM-as-judge + RAG/agent metrics.
• RAGAS: the RAG metrics below.
• W&B / MLflow for tracking runs and results.

Pitfalls that make scores lie

• Contamination: test items in the pretraining data. The model has seen the answers, like a test fixture leaked into the code under test.
• Prompt-format sensitivity: few-shot count and template can move scores a lot.
• Judge biases: an LLM judge may prefer the first answer shown (position), the longer one (verbosity), or its own writing (self-preference).
• Benchmarks ≠ your task, so build a domain eval set.

Hallucination: two kinds

From the hallucination survey on the reading list:

• Factuality hallucination conflicts with world facts (“The Eiffel Tower is in Rome”).
• Faithfulness hallucination is inconsistent with the provided context or instructions (the document says 26 weeks, the summary says 12). Older terms: intrinsic contradicts the source, extrinsic adds unverifiable content.

Causes: data gaps, decoding randomness, exposure bias (in training the model always sees correct previous tokens, but at generation time it builds on its own, possibly wrong, output). Mitigate with RAG, citations, lower temperature, rails, fine-tuning on grounded data, and human review.

Evaluating a RAG system

A RAG pipeline has two halves that fail differently, so RAGAS tests each one separately, like unit tests for two services:

• Retriever: context precision (are the relevant chunks ranked at the top?) and context recall (did retrieval find everything the reference answer needs?).
• Generator: faithfulness (is every claim in the answer supported by the retrieved context?) and answer relevancy (does the answer address the question?).

3.5 Experiment design

Before this: 3.4 Evaluation & benchmarking (eval sets and metrics).

The idea in plain English

Experiment design is how you avoid fooling yourself. You already do a version of it: when you measure whether a code change made things faster, you fix the hardware, run it several times, and compare against the old build. When you ship a risky feature, you canary it to a slice of traffic and watch the dashboards.

ML needs the same discipline because results are noisy. A new prompt can look 2% better purely by luck. The rules below make sure a “win” is real: decide what “better” means before you look, change one thing at a time, and measure enough to tell signal from noise.

Offline experiments (on a fixed eval set)

1. Question → hypothesis → primary metric decided in advance.
2. Baseline (current model or prompt) and a fixed eval set.
3. Change one variable at a time (ablation: remove or swap one component and see how much the result moves).
4. Repeat with different random seeds; report variance and confidence intervals, not a single number.
5. Log everything: model, tokenizer, data version, prompt, hyperparameters, code commit, hardware. Same reason you pin dependency versions: otherwise nobody can reproduce the result.
6. Hyperparameter search: grid < random < Bayesian (Optuna).

Online A/B testing (with real users)

Offline benchmarks can't tell you whether real users are happier. An A/B test splits live traffic between the current version (A) and the new one (B) and compares outcomes.

1. Randomly assign users (not self-selected) to A or B.
2. Pick a success metric plus guardrail metrics (latency, cost, safety) that must not get worse.
3. Compute sample size and statistical power up front; don't “peek and stop”.
4. Run long enough to cover weekly cycles and novelty effects (users click on anything new at first).
5. Test significance (p-value / CI); then roll out or roll back.
6. Related: canary release, shadow deployment (B gets a copy of traffic but its answers aren't shown), interleaving (mix A's and B's results in one list and see which get clicked).

The simulator below lets you watch both effects: confidence intervals narrowing as users arrive, and how often peeking crowns a “winner” when there is no real difference.

Technology evaluation

Exam objective 3.4 also covers choosing between tools, models and platforms. Compare options on:

• Cost: GPU-hours, $/1M tokens.
• Portability: ONNX, containers, cloud/on-prem; NIM runs anywhere.
• Compatibility: frameworks, GPUs, APIs such as OpenAI-compatible endpoints.
• Usability: docs, support, team skills.
• Plus quality, latency and license.

A typical NVIDIA path: prototype on build.nvidia.com hosted endpoints, then self-host the same NIM.

4.1 Inference optimization high yield

Before this: 1.4 The Transformer (attention and the KV cache).

The idea in plain English

Serving an LLM is web-server performance work with a very expensive request handler. A user cares about two things: how soon the answer starts (like time to first byte) and how fast it streams after that. You, the operator, care about a third: how many users one GPU can serve at once (throughput), because GPUs are expensive.

Almost every trick in this lesson has a familiar web-engineering twin. Learn the twin and the NVIDIA name, and most exam questions answer themselves.

Two phases: prefill, then decode

Prefill happens once per request. The model reads your entire prompt in one parallel pass and fills the KV cache with the keys and values of every prompt token. That is a lot of math done all at once, so prefill is compute-bound. When it finishes, the first output token appears, so prefill sets TTFT (time to first token).

The decode phase comes next. The model produces one token per step, and every step re-reads all of the model's weights and the KV cache from GPU memory to produce that single token. Little math, lots of reading, so decode is memory-bandwidth-bound. It sets ITL (inter-token latency), also called TPOT (time per output token).

Try both phases below. Make the prompt long and watch TTFT grow; make the answer long and watch the decode stripe grow.

The numbers you will be asked about

• TTFT: time to first token. Prefill plus any time spent queueing.
• ITL / TPOT: time between output tokens during decode.
• End-to-end latency ≈ TTFT + (output tokens − 1) × ITL.
• Throughput = tokens per second (or requests per second) across all users on the server.
• Report p50/p95/p99 percentiles, not just averages. A good average can hide a terrible tail, exactly as with API latency.

Latency and throughput pull against each other. Packing more users onto one GPU raises total tokens per second, but each user waits a little longer per token. Every technique below either cuts the work per token or lets the GPU serve more users without that penalty.

Memory: the KV cache and why it fills up

The KV cache is memoization for attention. Without it, every new token would recompute keys and values for the whole conversation so far. With it, each decode step computes only the new token. The price is memory: the cache grows with every token of every active request (≈ 2 × layers × KV heads × head size × sequence length × batch × bytes per number).

That memory, not compute, usually caps how many users fit on a GPU. Three fixes:

• MQA/GQA (multi-query / grouped-query attention): attention heads share keys and values, so the cache is smaller.
• Paged KV cache (PagedAttention, from vLLM; TensorRT-LLM has its own): instead of reserving one big contiguous block per request for the longest possible answer, hand out small fixed-size blocks as the answer grows. Less fragmentation means more concurrent requests.
• KV cache reuse / prefix caching: when many requests start with the same system prompt, compute its KV cache once and share it. This mostly cuts TTFT.

The figure compares the two allocation strategies on the same 64 blocks of GPU memory.

Batching: serve many users with one pass

Because decode is limited by reading the weights, reading them once for 32 users costs about the same as reading them for one. So servers batch: they run several requests through each step together. This is request coalescing, and it is the biggest single throughput win.

The catch is how the batch is scheduled. Static batching starts a batch and waits for its slowest request before admitting new ones, like await Promise.all(batch). In-flight batching (TensorRT-LLM's name; vLLM and others say continuous batching) lets requests join and leave at every decode step, like a worker pool pulling from a queue. GPU utilization goes up and queueing goes down.

Chunked prefill is the companion trick: a very long prompt is split into chunks and mixed in with other users' decode steps, so one huge prompt doesn't stall everyone's streaming.

Speculative decoding: guess cheaply, verify in one pass

Decode produces one token per expensive pass of the big model. Speculative decoding lets a small, fast draft model guess the next few tokens, then the big target model checks all the guesses in a single pass. It keeps the guesses up to the first wrong one and supplies the correct token there. Checking k tokens costs about the same as generating one, because decode is memory-bound: the weights are read once either way.

The speed-up depends on the acceptance rate: how often the draft guesses right. Done properly, the output has the same distribution as the target model alone. It is lossless, only faster. Variants replace the separate draft model: Medusa (extra decoding heads), EAGLE (a light draft head that reads the target's features) and MTP (multi-token prediction heads).

Quantization: smaller numbers, faster decode

Quantization stores weights (and sometimes activations) in fewer bits: FP16 → FP8, INT8, INT4 or NVFP4. A 7B-parameter model drops from 14 GB at FP16 to about 3.5 GB at INT4. Fewer bytes to read per decode step means faster decode, and the freed memory holds more KV cache, so more users fit.

Each number is rounded onto a coarser grid, scaled by a scale factor chosen per tensor (one for the whole weight matrix) or per channel or group (finer, more accurate). A few extreme values (outliers) stretch the grid and hurt everyone else's precision; methods such as SmoothQuant (INT8) and AWQ/GPTQ (INT4) exist to tame them. FP8 is native on Hopper (H100) GPUs; NVFP4 on Blackwell.

Two ways to get there, and the exam likes to contrast them:

• PTQ (post-training quantization): quantize a finished model. Quick. Needs a small calibration dataset to pick the scale factors, and can lose some accuracy.
• QAT (quantization-aware training): simulate the rounding during fine-tuning so the model learns to tolerate it. Slower, but recovers accuracy.

The figure shows the rounding happening on one layer's weights.

The rest of the toolbox

• FlashAttention, kernel fusion and CUDA graphs: lower-level speed-ups. Fewer round trips to GPU memory and less overhead launching small GPU programs.
• Tensor and pipeline parallel inference: split a model that is too big for one GPU across several.
• Disaggregated serving (prefill and decode on separate GPU pools, shipping the KV cache between them): NVIDIA Dynamo. Each phase gets hardware sized for its own bottleneck.
• Distillation and pruning (NVIDIA's Minitron models): make the model itself smaller.

Cheat sheet: technique → what it fixes

4.2 The NVIDIA serving stack, in one picture

Before this: 4.1 Inference optimization (batching, KV cache, quantization).

The idea in plain English

The product names sound like soup, but the shape is one you already know: shipping a web service. You write the code, build an optimized binary, run it inside an app server, and ship the whole thing as a container with a standard HTTP API. NVIDIA has one product per step.

Serving is only one stage of the bigger lifecycle. The figure maps every NVIDIA tool the exam mentions onto that lifecycle and lets you quiz yourself.

Which one do I pick?

• “Deploy a model fast, anywhere, behind an OpenAI-compatible API” → NIM.
• “Serve many models from different frameworks with batching and metrics” → Triton Inference Server (now Dynamo-Triton).
• “Maximize LLM tokens per second with in-flight batching and a paged KV cache” → TensorRT-LLM.
• “Split prefill and decode across GPU pools for a large multi-node LLM fleet” → NVIDIA Dynamo.
• “Compile a non-LLM model (say a vision CNN) for the lowest latency on a specific GPU” → TensorRT, often via ONNX.
• “Quantize to FP8 or INT4 AWQ before deployment” → NVIDIA Model Optimizer.
• “Move a PyTorch model to another runtime” → ONNX.

TensorRT: the compiler

TensorRT is NVIDIA's inference SDK and compiler. It turns a trained network (often via ONNX) into an optimized engine built for a specific GPU. It gets there through layer and tensor fusion, precision calibration (FP16/INT8/FP8), kernel auto-tuning, dynamic tensor memory and multi-stream execution. Like a compiled binary, an engine generally has to be rebuilt for a different GPU type.

TensorRT-LLM: the LLM engine

TensorRT-LLM is the open-source library that applies the 4.1 techniques to LLMs. It has had a PyTorch-based Python LLM API since v1.0. Features to recognise:

• in-flight batching and a paged KV cache with reuse;
• FP8 / FP4 / INT4-AWQ / INT8 quantization;
• speculative decoding;
• tensor, pipeline and expert parallelism;
• multi-LoRA (many fine-tuned adapters on one base model) and guided decoding (forcing output to follow a format).

trtllm-serve gives an OpenAI-compatible server. TensorRT-LLM also acts as a backend for Triton, Dynamo and NIM.

Triton Inference Server (now “Dynamo-Triton”): the model server

Triton is a general-purpose, open-source model server: the gunicorn or Tomcat of models. It was renamed NVIDIA Dynamo-Triton in 2025; questions may use either name. What to know:

• Model repository: a folder per model, e.g. model/1/model.plan + config.pbtxt (version 1 of the model plus its config).
• Backends (the engine that runs each model): TensorRT, TensorRT-LLM, PyTorch, ONNX Runtime, OpenVINO, Python, vLLM, FIL (tree models such as XGBoost).
• Dynamic batching for stateless models (settings: preferred batch sizes, max queue delay) · sequence batching for stateful ones.
• Concurrent model execution: run several copies of a model at once via instance_group.
• Ensembles and BLS (Business Logic Scripting): chain pre-process → model → post-process inside the server.
• HTTP/REST and gRPC endpoints, Prometheus metrics, model versioning and live loading.
• Model Analyzer sweeps configurations to find the best one · Perf Analyzer and GenAI-Perf (now AIPerf) generate load to measure latency and throughput.

Dynamic batching is the setting most worth understanding. The server holds incoming requests briefly and sends them to the GPU as one batch, either when the batch is full or when the oldest request has waited max_queue_delay. It is the same trade-off as a log shipper's flush buffer: wait a little, batch a lot.

For LLMs, the TensorRT-LLM backend uses in-flight batching (from 4.1) instead of this request-level batching.

NVIDIA Dynamo: serving across many GPUs

NVIDIA Dynamo is an open-source distributed inference framework (announced at GTC 2025; 1.0 in March 2026). Think of it as the cluster scheduler and smart load balancer above the engines:

• Disaggregated serving: prefill and decode run on separate GPU pools; NIXL transfers the KV cache between them.
• A KV-aware router sends each request to the GPU where its prefix cache already lives (cache-affinity routing).
• The KV Block Manager offloads cache to CPU memory or SSD; an SLA-based planner autoscales.
• Engine-agnostic: TensorRT-LLM, vLLM or SGLang.

NVIDIA NIM: the packaged microservice

NIM (NVIDIA Inference Microservices) is a prebuilt container = model + optimized engine + server + OpenAI-compatible API (/v1/chat/completions, /v1/embeddings). Existing OpenAI client code works by changing the base URL.

• Detects the GPU at start-up and picks an optimized profile for it.
• Runs anywhere with NVIDIA GPUs: cloud, data center, workstation. Supports LoRA adapters, and Kubernetes via Helm or the NIM Operator.
• Try models at build.nvidia.com, pull containers from NGC, license production use via NVIDIA AI Enterprise.
• Types: LLM, embedding/reranking (NeMo Retriever), speech (Riva), vision, safety (Nemotron guard models).
• NIM for LLMs 2.x is built on vLLM.

ONNX: the portable format

ONNX (Open Neural Network Exchange) is an open graph format for portability and interoperability, a bit like a cross-platform bytecode for models. Export with torch.onnx.export, then run it in ONNX Runtime (CPU, CUDA or TensorRT execution providers) or build a TensorRT engine from it. The reading-list blog covers moving models to NVIDIA RTX PCs via ONNX.

4.3 Building LLM applications

Before this: 1.7 Embeddings, vector search & RAG and 4.2 The NVIDIA serving stack.

The idea in plain English

An LLM application is an ordinary web service with one unusual dependency: a remote API that is slow, costs money per call, and doesn't return the same thing twice. Everything you already do for flaky third-party APIs applies: timeouts, retries, validation, caching, monitoring and staged rollouts.

On top of that come a few LLM-specific parts: the prompts, a retrieval system that feeds the model your own documents, guardrails that check input and output, and an evaluation set that tells you whether a change made answers better or worse.

A typical RAG chatbot. Observability (logs, metrics, traces) wraps every box.

The components to identify (objective 4.4)

• Data: sources, connectors, ingestion jobs, vector DB, feature or document store.
• Models: the LLM (size, license, context length), an embedding model, a reranker, guard models.
• Hardware: GPU memory must hold the weights plus the KV cache; memory bandwidth sets decode speed; NVLink for multi-GPU; CPU and data loading can bottleneck.
• Software: inference server or NIM, orchestration (LangChain/LangGraph), API gateway, cache, guardrails, observability.

Engineering practices: treat the model like a flaky remote API

• Structured output: ask for JSON that follows a schema (or use function calling), then validate it and retry on failure. Never trust free text you have to parse.
• Streaming tokens to the user with Server-Sent Events (SSE). The total time is the same, but the answer starts appearing at TTFT, so it feels much faster.
• Timeouts, bounded retries with backoff, fallbacks (a smaller model or a cached answer), rate limits.
• Secrets in environment variables or a vault, never in code or prompts.
• Tests: unit-test parsers and tools, regression-test prompts on a golden set of known questions and good answers, load-test.
• Version models, prompts and indexes together; containerize (Docker); deploy on Kubernetes with autoscaling and health checks.
• Canary / blue-green rollouts with rollback.

Common app patterns: chatbots and summarizers

A chatbot = system prompt + conversation memory + optional RAG + guardrails. Memory comes in three flavours: a buffer (keep the whole conversation), a window (keep the last N turns) or a summary (keep a running summary of older turns).

A summarizer picks one of three strategies, depending on whether the document fits in the context window:

• Stuff: the whole text fits, so put it all in one prompt.
• Map-reduce: summarize each chunk separately, then combine the summaries.
• Refine: walk through the chunks in order, updating one running summary.

Evaluate either with ROUGE, human or LLM judges, and faithfulness (does every claim come from the source?).

Agents and tools

An LLM agent is a model running in a loop. The model plans, then emits a tool call: JSON naming a tool and its arguments. Your application executes that call and puts the result back into the context. Repeat until the model says it is done. This Reason → Act → Observe pattern is called ReAct.

The model never runs anything itself; it only writes text asking for a call. That makes the app the security boundary: validate arguments, apply least privilege, cap the number of iterations, and require human approval for risky actions.

Step through one full loop below and watch the context grow with each call.

Frameworks you may see: LangGraph (stateful graphs with cycles; used in NVIDIA's RAD course), LlamaIndex, CrewAI, and NVIDIA's NeMo Agent Toolkit (formerly AgentIQ; profiling and evaluation; MCP support). MCP (Model Context Protocol) standardizes how tools and data sources plug into LLM apps, so one connector works across many apps.

Monitoring in production (objective 4.5)

• System metrics: request rate, TTFT, ITL, p95/p99 latency, throughput, errors, GPU utilization and memory (DCGM, nvidia-smi), queue depth.
• Product metrics: token cost, user feedback, quality and safety scores, drift in inputs and embeddings, retrieval hit rate.
• Tooling: Prometheus + Grafana, OpenTelemetry tracing, W&B/MLflow for experiments, Nsight for profiling.

4.4 Hardware & platform quick facts

The idea in plain English

This is a short fact sheet, not a concept lesson. Two ideas tie it together. First, memory decides what fits and how fast it runs: capacity limits how big a model (plus KV cache) you can load, and bandwidth limits how fast decode can go (see 4.1). Second, the GPU is often waiting for something else: another GPU, the network, or the CPU feeding it data. Most rows below are about one of those two.

5.1 Trustworthy AI principles, privacy & transparency

Next: 5.2 Bias, security & guardrails covers the other half of this domain.

The idea in plain English

“Trustworthy AI” is to models what security, compliance and SRE practices are to ordinary software. It isn't one feature you bolt on. It's a set of principles you design in from the start, plus documents and tools that prove you did.

This domain is only about 10% of the exam, and the questions are mostly recall: name the principles, match a tool to a problem. This lesson covers the principles, transparency (documenting what a model is) and privacy (protecting the people in the data).

NVIDIA's four Trustworthy AI principles

1. Privacy: comply with regulations, safeguard data (consent, minimization).
2. Safety & Security: avoid unintended harm and malicious threats.
3. Transparency: make AI explainable; disclose how it is built and used.
4. Nondiscrimination: minimize bias.

Source: NVIDIA “Trustworthy AI for a Better World” and “What Is Trustworthy AI?” (both on the reading list).

The exam's domain description adds that AI should also be energy-conscious, reliable, fair and verifiable. Energy-conscious choices include right-sizing models, quantization, PEFT and accelerated computing (GPUs do more work per watt than CPUs for parallel workloads).

The NIST AI Risk Management Framework

The NIST AI RMF 1.0 (2023) is a framework from the US standards body for managing AI risk, much as a security framework structures how you manage software risk. It has four functions:

• GOVERN: culture, policies, accountability. It is cross-cutting: it applies to the other three.
• MAP: understand the context, the intended use, and the risks and impacts.
• MEASURE: assess, analyze and track risks with metrics.
• MANAGE: prioritize, respond, monitor and communicate.

Its trustworthy characteristics: valid & reliable · safe · secure & resilient · accountable & transparent · explainable & interpretable · privacy-enhanced · fair with harmful bias managed. The generative-AI companion is the GenAI Profile, NIST AI 600-1 (2024).

Transparency: model cards and explainability

A model card is the README + CHANGELOG + SECURITY.md of a model: what it is for, what it was trained on, how it was evaluated, and where it fails. It lets people judge whether a model fits their use before they integrate it.

NVIDIA's version, Model Card++, adds four subcards: Bias, Explainability, Privacy, and Safety & Security. These line up with the four principles above.

Transparency also covers explaining individual outputs and their origin:

• Explainability: SHAP and LIME estimate how much each input feature pushed a prediction; attention visualizations show what a transformer focused on; in RAG, citations show which source a claim came from.
• Provenance: C2PA content credentials and watermarking mark content as AI-generated and record where it came from.

Privacy and consent (objective 5.2)

Privacy starts before any model exists, with how you collect and handle data:

• Informed, specific, revocable consent, and purpose limitation (use data only for what people agreed to).
• Data minimization, retention limits and the right to deletion (GDPR/CCPA).
• PII detection and redaction (NeMo Curator, Presidio), anonymization and pseudonymization.
• Synthetic data (NeMo Safe Synthesizer) in place of real records.
• Access control on RAG documents; don't log secrets or PII.

The underlying trade-off: more data and more personalization versus individual rights.

Privacy-preserving training

Two techniques let you learn from sensitive data without exposing individuals.

Federated learning (NVIDIA's framework: NVIDIA FLARE) keeps the data where it is. Instead of copying every hospital's records to one server, you send the model to each hospital, train locally, and send back only the weight updates. The server averages them (FedAvg). Updates can still leak some information, so it's often combined with the next technique or with secure aggregation.

Differential privacy adds carefully calibrated random noise so that no answer, and no trained model, reveals whether any one person was in the data. The privacy budget ε (epsilon) sets the trade-off: lower ε means more noise, more privacy and less accuracy. DP-SGD applies this during training by clipping each example's gradient and adding noise.

5.2 Bias, security & guardrails

Before this: 5.1 Principles, privacy & transparency. Helpful: 1.7 RAG and 4.3 Building LLM apps.

The idea in plain English

Once a model is in production, two different things can go wrong. It can be unfair: it works well on average but fails one group of users, the way a healthy p50 latency can hide a terrible p99. Or it can be unsafe: someone tricks it, poisons it or gets it to say something harmful.

This lesson covers both. First where bias comes from and how to measure it, then the attacks on LLM apps, and finally guardrails: the middleware that sits around the model and checks what goes in and out.

Where bias comes from (objective 5.4)

Bias usually enters through the data or the process, not the code. Under-sampled groups cause representation bias; an unfair past baked into labels causes historical bias; noisy or proxy labels cause measurement bias. The full list:

Measuring and reducing bias

The core habit is subgroup evaluation: report your metrics per group (accent, region, language, age band), not just one overall number. Then compare groups with a fairness measure:

• Demographic parity: each group gets positive outcomes (say, loan approvals) at the same rate.
• Equal opportunity: the true-positive rate (recall) is equal across groups, so qualified people are approved equally often.
• Equalized odds: both the true-positive rate and the false-positive rate are equal across groups.

The figure shows why a single overall score isn't enough.

Mitigate across the whole lifecycle:

• Data: diverse, representative data; rebalancing and reweighting; augmentation; diverse annotators.
• Training: debiasing constraints or adversarial debiasing.
• After training: threshold adjustment per group.
• Process: subgroup evaluation, human review, ongoing monitoring, and document everything in model cards (5.1).

Threats to an LLM application

The attacks below map neatly onto web-security ideas you already know. The one to understand best is prompt injection: SQL injection for LLMs. The model can't reliably tell instructions from data, because both arrive as text in one prompt.

Guardrails: middleware around the model

NeMo Guardrails is NVIDIA's open-source toolkit for programmable rails, configured in Colang (a small language for conversation flows) plus YAML. Think of it as a middleware chain around the LLM call: each rail can pass a message, rewrite it, or stop it with a canned reply. It has five rail types:

• Input rails: block or mask user input (jailbreak detection, PII masking).
• Dialog rails: control conversation flows and topics (topical control).
• Retrieval rails: filter or mask RAG chunks before they reach the prompt.
• Execution rails: check tool and action inputs and outputs.
• Output rails: moderate the answer, fact-check it (hallucination rails).

Safety models plug in as rails. NVIDIA's are the Nemotron safety / NemoGuard NIMs: Content Safety, Topic Control and Jailbreak Detect. A non-NVIDIA alternative is Llama Guard.

Other NVIDIA tools to recognise (objective 5.3)

• garak: LLM vulnerability scanner for red-teaming (probing your own model for weaknesses before attackers do).
• NeMo Curator: PII redaction and toxicity filtering of training data.
• SteerLM: user-steerable attributes such as toxicity.
• NVIDIA FLARE: federated learning (see 5.1).
• Model Card++: NVIDIA model cards with bias, explainability, privacy and safety & security subcards (see 5.1).