{ "module": "FT11 — The Training Loop with TRL", "course": "3 — LLM Fine-Tuning Masterclass", "version": "1.0.0", "duration_minutes": 50, "total_questions": 15, "bloom_distribution": { "target": "20% recall / 40% application / 40% analysis-design", "actual": { "recall": 3, "application": 6, "analysis": 6 } }, "passing_score_percent": 70, "questions": [ { "id": "Q01", "bloom": "recall", "type": "multiple_choice", "prompt": "What is TRL, and what distinguishes its v1.0 release (March 31, 2026)?", "options": [ "A HuggingFace library for pretraining base models from scratch; v1.0 added multi-node support.", "The canonical post-training library for the HuggingFace stack — 75+ methods, ~3M downloads/month, ships a Stability Contract and production CLI. v1.0 is the pinned stable release.", "A replacement for transformers.Trainer that only does SFT; v1.0 dropped DPO and GRPO.", "A quantization library; v1.0 added GGUF export for fine-tuned models." ], "answer_index": 1, "rationale": "TRL (Transformers Reinforcement Learning) is the canonical post-training library: SFT, DPO, KTO, RLOO, GRPO, reward modeling, async RL. v1.0 (March 31, 2026) is the pinned stable release with a Stability Contract pinning the public API and a production CLI (trl sft, trl dpo, trl grpo). It does not pretrain, it is not limited to SFT, and it is not a quantization library." }, { "id": "Q02", "bloom": "recall", "type": "multiple_choice", "prompt": "Name the six stages of the SFTTrainer end-to-end pipeline, in order.", "options": [ "Load model → load dataset → train → eval → save → merge.", "Load dataset → load model → attach PEFT config → build SFTConfig → trainer.train() → save/merge/load.", "Preprocess data → build dataloader → define optimizer → train loop → eval → push to hub.", "Choose base → choose adapter → choose LR → train → quantize → serve." ], "answer_index": 1, "rationale": "The pipeline is: (1) dataset (chat-format, train/eval split), (2) model (bf16, FlashAttn), (3) PEFT config (LoraConfig or None), (4) SFTConfig (TrainingArguments levers), (5) trainer.train() (loss curve, eval every N steps), (6) save/merge/load. Config (stage 4) comes before train (stage 5); PEFT (stage 3) is attached before config." }, { "id": "Q03", "bloom": "recall", "type": "multiple_choice", "prompt": "What is the formula for effective batch size?", "options": [ "per_device_batch_size × num_train_epochs", "per_device_batch_size × gradient_accumulation_steps × world_size", "per_device_batch_size + gradient_accumulation_steps + world_size", "total_dataset_size ÷ per_device_batch_size" ], "answer_index": 1, "rationale": "effective_batch = per_device_batch_size × gradient_accumulation_steps × world_size. Gradient accumulation simulates a large batch by accumulating gradients across several forward/backward passes before stepping the optimizer — the optimizer 'sees' the product. This is how you get stable optimization when VRAM-limited." }, { "id": "Q04", "bloom": "application", "type": "multiple_choice", "prompt": "You are running a LoRA SFT job and set learning_rate=2e-5. The loss barely moves from step 1. What is the likely cause and the fix?", "options": [ "The dataset is too small; add more data.", "Wrong LR for the method. 2e-5 is the full-FT band; LoRA needs 1e-4 to 5e-4. Raise the LR to ~2e-4.", "Gradient checkpointing is on; turn it off.", "The model is too large; switch to a smaller base." ], "answer_index": 1, "rationale": "LoRA needs an order of magnitude higher LR than full FT because adapters are small and start random, needing faster learning against a frozen base. 2e-5 is the full-FT band (1e-5 to 5e-5); for LoRA use 1e-4 to 5e-4. The cardinal error is mismatching the method and the LR band. (Also worth checking the chat template/completion mask, but the LR mismatch is the first suspect.)" }, { "id": "Q05", "bloom": "application", "type": "multiple_choice", "prompt": "Your SFT run is stable for 200 steps, then the loss jumps to NaN. You are on FP16 on a Colab T4. What is the most likely cause and the immediate fix?", "options": [ "The LR is too low; raise it.", "FP16 overflow (narrow dynamic range causes gradients to overflow to inf during backward). The T4 lacks BF16. Fix: reduce per_device_batch_size to 1, raise gradient_accumulation_steps, ensure max_grad_norm=1.0 clipping. Real fix: BF16-capable hardware.", "The dataset is contaminated; clean it.", "FlashAttention is not installed; install it." ], "answer_index": 1, "rationale": "FP16 has a narrow dynamic range; gradients overflow to inf during backward, producing NaN (failure mode 2). BF16 (same memory, wider range) avoids this but the T4 doesn't support it. Stabilize by reducing batch size (less memory pressure), keeping the effective batch via accumulation, and ensuring gradient clipping. The real fix is Ampere+ hardware with BF16." }, { "id": "Q06", "bloom": "application", "type": "multiple_choice", "prompt": "You want to deploy your fine-tuned LoRA model as a single standalone model served by vLLM. Which operation do you perform?", "options": [ "trainer.save_model() and serve the adapter directory directly.", "model.merge_and_unload() to fold the adapter into the base, then save_pretrained() the merged model.", "PeftModel.from_pretrained() at inference time and serve the base + adapter separately.", "Quantize the adapter to GGUF and serve with Ollama." ], "answer_index": 1, "rationale": "For deployment as a single model, merge the adapter into the base with merge_and_unload(), producing a standalone transformers model with no adapter layer. save_pretrained() writes it as one checkpoint ready to quantize (FT19) or serve (FT20). Keeping the adapter separate (options A, C) is for experimentation/hot-swapping, not single-model deployment. Quantizing the adapter alone (D) isn't a thing." }, { "id": "Q07", "bloom": "application", "type": "multiple_choice", "prompt": "You set load_best_model_at_end=True with metric_for_best_model='eval_loss'. Why, and what would you risk without it?", "options": [ "To make training faster; without it the run is slower.", "So the model in memory at the end is the checkpoint with the LOWEST eval loss (the one to ship). Without it you get the LAST checkpoint, which may be past the eval-loss minimum (overfitting) — you'd ship a memorizing model.", "To reduce VRAM usage; without it you OOM.", "To enable packing; without it packing is disabled." ], "answer_index": 1, "rationale": "trainer.train() leaves you with the last checkpoint by default. If eval loss rose in the final steps (overfitting), the last checkpoint is worse than the best. load_best_model_at_end=True with metric_for_best_model='eval_loss' and greater_is_better=False ensures the model in memory is the one with the lowest eval loss — the one you ship." }, { "id": "Q08", "bloom": "application", "type": "multiple_choice", "prompt": "You are VRAM-limited on a 24GB GPU training a 3B model with LoRA. Which combination of settings maximizes your ability to fit the run?", "options": [ "per_device_batch_size=8, gradient_checkpointing=False, fp16=True.", "per_device_batch_size=2, gradient_accumulation_steps=8, gradient_checkpointing=True, bf16=True.", "per_device_batch_size=1, no gradient accumulation, gradient_checkpointing=False, fp32=True.", "per_device_batch_size=4, gradient_checkpointing=True, optim='adamw' (32-bit), bf16=False." ], "answer_index": 1, "rationale": "The VRAM-tight recipe: small per_device_batch (2), compensate with gradient accumulation (8 → effective batch 16), gradient_checkpointing=True (~60-70% activation memory cut for ~30% slower), and bf16=True (same memory as fp16, no overflow). Option A OOMs (large batch, no checkpointing). Option C forgoes accumulation (tiny effective batch, unstable). Option D uses 32-bit AdamW (more memory) and no BF16." }, { "id": "Q09", "bloom": "application", "type": "multiple_choice", "prompt": "You want to hot-swap between three different LoRA adapters on a single base model at inference (one base, many adapters). Which approach is correct?", "options": [ "Merge each adapter into a separate copy of the base and load three models.", "Use PeftModel.from_pretrained(base, adapter_path) and swap at runtime with model.load_adapter(other_path). The base is never touched.", "Concatenate the three adapters into one and merge.", "Fine-tune the base three times sequentially." ], "answer_index": 1, "rationale": "This is the FT00 swappability property in code: Layer 2 (adapter) detaches from Layer 1 (base) without affecting it. Load the base once, attach adapters with PeftModel.from_pretrained, and swap with load_adapter. Merging (A) defeats the purpose (three full-size models). Concatenation (C) and sequential fine-tuning (D) are not how adapters work." }, { "id": "Q10", "bloom": "analysis", "type": "multiple_choice", "prompt": "Your training loss descends steadily, but eval loss starts rising after step 300 while train loss keeps dropping. What is happening, and what should you do?", "options": [ "The model is converging; let it run to completion.", "The model is overfitting (memorizing the training set, not generalizing). Stop at the eval-loss minimum (around step 300) and ship that checkpoint. More epochs would worsen it.", "The learning rate is too low; raise it.", "The dataset is too large; reduce it." ], "answer_index": 1, "rationale": "Train dropping while eval rises is the textbook overfitting signal — the gap between the two curves is the overfitting measure. The model is memorizing training examples rather than steering general behavior. Stop at the eval-loss minimum and use that checkpoint (load_best_model_at_end=True would have done this automatically). Continuing past the minimum burns compute and ships a worse model." }, { "id": "Q11", "bloom": "analysis", "type": "multiple_choice", "prompt": "You see the grad norm jump 10x between two logging steps (from 0.8 to 8.0), but the loss hasn't NaN'd yet. What does this tell you, and what should you do?", "options": [ "Nothing — grad norm fluctuation is normal. Let it run.", "It's the early-warning system for instability. The run is about to explode (NaN). Kill it, lower the LR, ensure max_grad_norm=1.0 clipping, and retry.", "The model has converged; stop training.", "Switch from BF16 to FP16 for more precision." ], "answer_index": 1, "rationale": "Grad norm is the canary. A healthy run has a stable, slowly-decreasing grad norm. A sudden 10x spike means gradients are exploding — the loss NaN is imminent even if it hasn't happened yet. Kill the run, lower the LR, ensure gradient clipping (max_grad_norm default 1.0 is usually fine), and retry. Ignoring it (A) leads to NaN. Switching to FP16 (D) would make it worse (narrower range)." }, { "id": "Q12", "bloom": "analysis", "type": "multiple_choice", "prompt": "Why does TRL train only on the completion (masking the prompt tokens from the loss), and what goes wrong if this masking is absent or incorrect?", "options": [ "It speeds up training by skipping half the tokens. Without it, training is slower but correct.", "An instruct model should learn to RESPOND, not to parrot the user. Completion masking means gradients flow only from the assistant's response. Without it, the model learns to imitate the user too — producing a model that echoes prompts instead of answering. This is the FT07 template bug; the loss curve looks like 'LR too low' but the cause is the data.", "It reduces VRAM usage by not storing prompt gradients. Without it you OOM.", "It is required for BF16 to work correctly." ], "answer_index": 1, "rationale": "TRL applies the model's chat template and masks prompt tokens from the loss so the model learns to generate the assistant's response, not the user's turn. Without masking (or if the chat template is wrong), the model trains on the full sequence including the user's words — it learns to imitate the user and echo prompts. The loss curve deceptively looks like an LR problem, but the cause is data/template. This is the FT07 failure mode." }, { "id": "Q13", "bloom": "analysis", "type": "multiple_choice", "prompt": "A team runs two SFT jobs on the same data and model: one with LoRA (LR 2e-4) and one full FT (LR 2e-5). Both converge to similar eval loss. Why might they still choose LoRA for production, and what does this illustrate about the FT10 decision?", "options": [ "Full FT is always lower quality, so LoRA wins.", "LoRA produced a ~100MB adapter vs full FT's multi-GB checkpoint, trains faster, and is hot-swappable — the FT10 decision is about cost (VRAM, disk, swappability) as much as quality. When quality is comparable, the cheaper method wins.", "LoRA is always faster, so it wins regardless of quality.", "Full FT and LoRA are identical, so the choice is arbitrary." ], "answer_index": 1, "rationale": "The FT10 decision weighs quality against cost. When LoRA and full FT reach comparable eval loss (common for steering/format tasks per the FT00 thesis — steering is low-rank), LoRA wins on cost: a few hundred MB adapter vs multi-GB full model, faster training, lower VRAM, and hot-swappability (one base, many adapters). Full FT is chosen when LoRA can't match quality (some knowledge-heavy adaptation) or when you need to ship a single merged model and adapter overhead matters. It's not that LoRA is always better — it's that the decision is multi-factorial." }, { "id": "Q14", "bloom": "analysis", "type": "multiple_choice", "prompt": "A student argues: 'I don't need an eval split — I'll just train for 1 epoch and use the final checkpoint. The loss went down, so it worked.' Why is this wrong, and what specifically can't they detect without eval?", "options": [ "They're right — if loss went down, the model improved.", "Without eval they can't distinguish convergence from overfitting. A descending train loss with a rising eval loss means the model is memorizing, not steering. They also can't compare checkpoints to pick the best (lowest eval loss). They are flying blind — the 'no eval' anti-pattern.", "They need eval only to publish the model, not to train it.", "Eval is only needed for DPO, not SFT." ], "answer_index": 1, "rationale": "The 'no eval' anti-pattern. Train loss descending does NOT mean the model is generalizing — it could be memorizing the training set (train down, eval up = overfitting). Without a held-out eval split, you cannot: (1) detect overfitting, (2) identify the best checkpoint (lowest eval loss), (3) compare hyperparameter runs. You ship a memorizing model blind. Even 200 held-out examples is enough to catch the failure. Eval is not optional for any method." }, { "id": "Q15", "bloom": "analysis", "type": "multiple_choice", "prompt": "Why does the module say TRL's design principle is 'fine-tuning steers behavior,' and how does this connect to what TRL automates (chat templates, completion masking, PEFT, packing)?", "options": [ "TRL injects knowledge into models via chat templates; the steering is a side effect.", "Every TRL trainer (SFT, DPO, GRPO) is a steering tool — none inject knowledge. TRL automates the mechanics of steering (applying chat templates so the model learns to respond, completion masking so it learns responses not parroting, PEFT attachment so steering is cheap, packing so it's fast) so you spend time on data and hyperparameters, not glue code. This is the FT00 thesis realized in software.", "TRL's principle is to maximize throughput, not steering quality.", "TRL automates pretraining, which is why it steers." ], "answer_index": 1, "rationale": "TRL's trainers are all steering tools: SFTTrainer steers format/instruction-following, DPOTrainer steers preference, GRPOTrainer steers reasoning. None teach knowledge (the FT00 thesis). TRL's value is automating the steering mechanics — chat templates, completion masking, PEFT, packing — that you'd otherwise write as error-prone glue code. This frees you to focus on the actual work: the steering wheel (data, Pillar 1) and the levers (hyperparameters). Option D is wrong — TRL does not pretrain." } ] }