Course: Course 3 — LLM Fine-Tuning Masterclass
Module: FT11 — The Training Loop with TRL
Duration: ~50 minutes (spoken at ~140 wpm)
Format: Verbatim transcript with [SLIDE N] cues. Read aloud or use as speaker notes.
[SLIDE 1 — Title]
Welcome to module FT eleven, The Training Loop with TRL. This is the capstone of pillar two, PEFT. Everything in FT zero-eight through FT ten — the adapter mechanics, the QLoRA trick, the full-versus-PEFT decision — has been leading here. In this module you run a complete, instrumented SFT job, end to end, and you learn to read what it tells you.
If you have not done FT zero-eight and FT ten, go back. This module assumes you know what a LoRA adapter is and why you might choose PEFT over full fine-tuning. It does not re-teach those.
[SLIDE 2 — TRL: the canonical post-training library]
TRL — Transformers Reinforcement Learning — is the canonical post-training library for the HuggingFace stack. Version one-point-zero shipped March thirty-first, twenty twenty-six. It is the library you reach for when you want to steer a transformer model.
The numbers that make it the default. Seventy-five-plus training methods and trainer variants. Roughly three million downloads a month. A Stability Contract that pins the public API so your production pipelines don't break on a minor bump. And a production CLI — trl sft, trl dpo, trl grpo — that runs a complete, reproducible job from a YAML config with no Python glue.
The thesis from FT zero-zero — fine-tuning steers behavior — is TRL's design principle. Every trainer in the library is a steering tool. SFT trainer steers format. DPO trainer steers preference. GRPO trainer steers reasoning toward verifiable rewards. None of them inject knowledge. The library's job is to make the steering reliable, observable, and cheap.
[SLIDE 3 — Why TRL, not raw transformers]
You can write a training loop with the transformers Trainer and a custom data collator. People did, for years. They stopped because SFT is not just language modeling on a new dataset. Three things make it different, and TRL handles all three.
First, chat templates. A base model expects raw text. An instruct model expects its chat template applied. TRL applies the model's tokenizer template automatically and trains only on the completion — masking the prompt — so the model learns to respond, not to parrot the user. Get this wrong and you train the model to imitate the user. That is the classic FT zero-seven template bug.
Second, PEFT integration. Attach a LoraConfig, pass it to the trainer, and the model is wrapped, the base frozen, the adapter trained and saved — without a line of PEFT glue.
Third, packing. Short sequences waste compute. TRL packs multiple examples into one sequence with proper attention masking, often doubling throughput. Doing this by hand is error-prone.
The cost of TRL is learning its config surface. The benefit is you stop debugging glue code and start debugging your data and your hyperparameters — which is where the actual work is.
[SLIDE 4 — The SFTTrainer pipeline, end to end]
The SFTTrainer flow is six stages. Stage one, load the dataset — chat-format, with a held-out eval split. Never train on everything; that is flying blind. Stage two, load the model — BF sixteen, FlashAttention. Stage three, attach the PEFT config — a LoraConfig, or none for full fine-tuning. Stage four, build the SFTConfig — that is the TrainingArguments levers. Stage five, trainer dot train — the loop, with eval every N steps. Stage six, save, merge, or load.
You will write this loop, or a close variant of it, in every SFT job for the rest of the course. Memorize the shape. The lab walks you through each stage with runnable code.
[SLIDE 5 — Reading a training loss curve: the healthy case]
The loss curve is the EKG of your training run. Learn to read it before you touch a hyperparameter.
A healthy SFT loss curve descends from its initial value — often one-point-five to two-point-five for a base model on a new format. It drops steeply for the first few hundred steps, then flattens into a gentle decline. The eval loss tracks it, ideally staying below or near the training loss. That gap — train below eval, both descending — is the sign of a healthy steer. The model is generalizing, not memorizing.
[SLIDE 6 — The three failure modes]
Three failure modes account for almost every bad run you will ever see.
Failure one: loss not decreasing. Flat or barely dropping from step one. Three suspects, in order of likelihood. Learning rate too low — the most common cause. Data problem — the chat template isn't applied, or the completion masking is wrong so you're training on the prompts. Or a template bug — the tokenizer's chat template doesn't match the model. All three look identical on the loss curve, so inspect a tokenized batch before assuming it's the LR.
Failure two: loss exploding — NaN or a spike. Learning rate too high, the optimizer overshoots, weights blow up. Or numerical instability — the classic FP sixteen bug. FP sixteen has a narrow dynamic range; gradients overflow to infinity during backward. Use BF sixteen instead — same memory, wider range, no overflow on Ampere or later. Or an EOS or template bug producing degenerate targets — if the completion is empty, the loss target is undefined and you get NaN.
Failure three: loss plateaus. It descended, then goes flat. The eval loss tells you which. Eval flat and low — converged, stop. Eval flat but high, or eval rising while train drops — stuck or overfitting. Try a higher LR with warmup, a different schedule, or — most often — more and better data. A plateau on bad data is not a hyperparameter problem. It's a data problem.
[SLIDE 7 — The grad norm signal]
Beyond loss, watch the gradient norm. TRL logs it as grad-norm by default. A healthy run has a stable, slowly-decreasing grad norm. A sudden spike — ten-x between two logging steps — is the early-warning system for instability. Even if the loss hasn't NaN'd yet, that run is about to explode. Kill it, lower the LR, ensure gradient clipping — max-grad-norm, default one-point-zero in TRL, usually fine — and retry.
Grad norm is the canary. Loss is the canary's heartbeat. Watch both.
[SLIDE 8 — The TrainingArguments levers: LR and batch]
Now the knobs on the front panel. Learning rate is the single most important hyperparameter. For full fine-tuning, one-e-minus-five to five-e-minus-five — small steps to avoid catastrophic forgetting. For LoRA, one-e-minus-four to five-e-minus-four — an order of magnitude higher, because adapters are small and start random; they need faster learning to catch up to the frozen base.
The cardinal error: using a full-FT LR for LoRA — loss barely moves — or a LoRA LR for full FT — loss explodes. The method, from FT ten, sets the LR band. Mismatching them is the most common "why isn't my model learning" ticket.
Batch size plus gradient accumulation. Effective batch equals per-device batch times accumulation steps times world size. When VRAM-limited — and you usually are — you can't fit a large per-device batch. Gradient accumulation simulates a large batch by accumulating gradients across several forward-backward passes before stepping the optimizer. A batch of two with eight accumulation steps on one GPU is an effective batch of sixteen. This is how you get stable optimization on a twenty-four-gig card.
[SLIDE 9 — The levers: warmup, scheduler, checkpointing, attention, optimizer]
Warmup ratio, typically zero-point-zero-three to zero-point-one. Warmup ramps the LR from zero to the target over the first fraction of steps, stabilizing early training. Skip it and you risk an early spike, especially with a high LR. For LoRA with its higher LRs, warmup is near-mandatory.
LR scheduler. Cosine is the common default — smooth decay to near-zero, gentle and reliable. Linear is slightly more aggressive at the end. Constant-with-warmup is flat after warmup, used when you train a fixed budget and will pick the best checkpoint. Cosine is the safe default. Reach for the others when you have a reason.
Gradient checkpointing. Trade compute for memory: recompute activations during backward instead of storing them. Cuts activation memory roughly sixty to seventy percent for about a thirty-percent slowdown. This is the FT zero-one VRAM lever — turn it on before you reduce batch size.
FlashAttention two or three. Effectively mandatory for speed and memory. It computes exact attention in tiles that fit in SRAM, avoiding the full attention matrix. For long sequences, a two-to-four-x speedup and a large memory saving. Set attn-implementation to flash-attention-two when loading the model.
Optimizer. AdamW is the default — robust, well-understood, most memory. AdamW eight-bit, from bitsandbytes, quantizes the optimizer states, saves memory at a small accuracy cost. Adafactor is low-memory but finicky. PagedAdamW uses CUDA paging to offload states to CPU — the QLoRA default, pairs with four-bit quantized bases.
[SLIDE 10 — Saving, loading, merging]
Three operations, three purposes. Confuse them and you ship the wrong artifact.
Save. Trainer dot save-model saves what was trained. With a PEFT config, that's the adapter — a few hundred megabytes, not the multi-gigabyte base. With no PEFT config, it's the entire model.
Merge for deployment. Adapters are great for experimentation but inference with a separate adapter has overhead. For deployment, merge the adapter into the base with merge-and-unload. The result is a plain transformers model — no adapter layer — ready to quantize or serve. This is the standard path from "I trained a LoRA" to "I have a deployable model."
Load an adapter. For hot-swapping at inference — one base, many adapters — don't merge. Load the adapter on top of the base with PeftModel dot from-pretrained, and swap with load-adapter at runtime. This is the swappability property from FT zero-zero, in code. Layer two detaches from layer one without touching it.
[SLIDE 11 — Logging: what to watch]
If it isn't logged, it didn't happen. If you can't see the loss curve, you are not training — you are hoping.
Backends. Weights and Biases is the long-standing default — rich dashboards, experiment comparison. Trackio is HuggingFace's lighter, Spaces-backed tracker. TensorBoard or JSONL has zero external dependencies; the logs live next to your checkpoints.
What to log. At minimum, every step: train loss, learning rate so you can see the schedule, grad norm — the early-warning system. Every eval step: eval loss. Periodically: throughput, tokens per second, so you can spot a slowdown.
The single most useful plot is train loss and eval loss on the same chart. The gap between them is your overfitting signal. If train keeps dropping while eval rises, you are memorizing, not steering. Stop at the eval-loss minimum and use that checkpoint. Set load-best-model-at-end to true and let the trainer do this for you.
[SLIDE 12 — Anti-patterns]
Five anti-patterns to leave with.
First, no eval — flying blind. You cannot tell convergence from overfitting, you cannot compare checkpoints, and you will ship a memorizing model. The eval split is not optional. Even two hundred examples is enough to catch the failure.
Second, wrong LR for the method. Full-FT LR on a LoRA run — loss barely moves. LoRA LR on a full-FT run — loss explodes. The method sets the LR band.
Third, no logging. A run that NaN'd at step fifty and a run that converged at step fifty look identical from the final checkpoint. Log loss, LR, grad norm, eval loss — every step.
Fourth, ignoring grad norm spikes. The loss looks fine, but grad norm jumped ten-x. You ignore it. Fifty steps later, NaN. Grad norm is the canary; watch it.
Fifth, FP sixteen instead of BF sixteen. On Ampere-or-later hardware that supports BF sixteen, there is no reason to use FP sixteen for training. FP sixteen overflows; BF sixteen doesn't. Same memory, strictly better. If your GPU is pre-Ampere, that's a hardware constraint — plan to upgrade.
[SLIDE 13 — The lab: the full SFT loop]
The lab is the capstone of pillar two. You run a complete SFT job with TRL's SFTTrainer on Qwen two-point-five three-B or MiniCPM three four-B — your choice. You log to W and B or Trackio. You evaluate on a held-out split every N steps. You produce the loss curve, the eval curve, and a merged, deployable model.
You will see the steering effect with your own eyes — generating samples before and after fine-tuning on the same prompts. The base moves. That is the thesis, felt.
And if your run NaNs — good. The failure is the lesson. You diagnose it using the loss curve and the grad norm, you fix the lever, and you re-run. That diagnosis is the skill this module is actually teaching.
[SLIDE 14 — What you can now do]
You can now run a complete SFT job with TRL end to end. You can read a loss curve and diagnose the three failure modes. You can choose the TrainingArguments levers for a given hardware budget and method. You can save, merge, and load a fine-tuned model. And you can wire up logging and know what to watch.
That is the training loop. Next, module FT twelve: SFT, the baseline. Now that you can run the loop, we zoom into the method that is the starting point for almost every real fine-tuning project — supervised fine-tuning, done right.
End of module FT eleven. Duration: approximately fifty minutes at one-hundred-forty words per minute.
# Teaching Script — Module FT11: The Training Loop with TRL **Course**: Course 3 — LLM Fine-Tuning Masterclass **Module**: FT11 — The Training Loop with TRL **Duration**: ~50 minutes (spoken at ~140 wpm) **Format**: Verbatim transcript with `[SLIDE N]` cues. Read aloud or use as speaker notes. --- [SLIDE 1 — Title] Welcome to module FT eleven, The Training Loop with TRL. This is the capstone of pillar two, PEFT. Everything in FT zero-eight through FT ten — the adapter mechanics, the QLoRA trick, the full-versus-PEFT decision — has been leading here. In this module you run a complete, instrumented SFT job, end to end, and you learn to read what it tells you. If you have not done FT zero-eight and FT ten, go back. This module assumes you know what a LoRA adapter is and why you might choose PEFT over full fine-tuning. It does not re-teach those. [SLIDE 2 — TRL: the canonical post-training library] TRL — Transformers Reinforcement Learning — is the canonical post-training library for the HuggingFace stack. Version one-point-zero shipped March thirty-first, twenty twenty-six. It is the library you reach for when you want to steer a transformer model. The numbers that make it the default. Seventy-five-plus training methods and trainer variants. Roughly three million downloads a month. A Stability Contract that pins the public API so your production pipelines don't break on a minor bump. And a production CLI — `trl sft`, `trl dpo`, `trl grpo` — that runs a complete, reproducible job from a YAML config with no Python glue. The thesis from FT zero-zero — fine-tuning steers behavior — is TRL's design principle. Every trainer in the library is a steering tool. SFT trainer steers format. DPO trainer steers preference. GRPO trainer steers reasoning toward verifiable rewards. None of them inject knowledge. The library's job is to make the steering reliable, observable, and cheap. [SLIDE 3 — Why TRL, not raw transformers] You can write a training loop with the transformers Trainer and a custom data collator. People did, for years. They stopped because SFT is not just language modeling on a new dataset. Three things make it different, and TRL handles all three. First, chat templates. A base model expects raw text. An instruct model expects its chat template applied. TRL applies the model's tokenizer template automatically and trains only on the completion — masking the prompt — so the model learns to respond, not to parrot the user. Get this wrong and you train the model to imitate the user. That is the classic FT zero-seven template bug. Second, PEFT integration. Attach a LoraConfig, pass it to the trainer, and the model is wrapped, the base frozen, the adapter trained and saved — without a line of PEFT glue. Third, packing. Short sequences waste compute. TRL packs multiple examples into one sequence with proper attention masking, often doubling throughput. Doing this by hand is error-prone. The cost of TRL is learning its config surface. The benefit is you stop debugging glue code and start debugging your data and your hyperparameters — which is where the actual work is. [SLIDE 4 — The SFTTrainer pipeline, end to end] The SFTTrainer flow is six stages. Stage one, load the dataset — chat-format, with a held-out eval split. Never train on everything; that is flying blind. Stage two, load the model — BF sixteen, FlashAttention. Stage three, attach the PEFT config — a LoraConfig, or none for full fine-tuning. Stage four, build the SFTConfig — that is the TrainingArguments levers. Stage five, trainer dot train — the loop, with eval every N steps. Stage six, save, merge, or load. You will write this loop, or a close variant of it, in every SFT job for the rest of the course. Memorize the shape. The lab walks you through each stage with runnable code. [SLIDE 5 — Reading a training loss curve: the healthy case] The loss curve is the EKG of your training run. Learn to read it before you touch a hyperparameter. A healthy SFT loss curve descends from its initial value — often one-point-five to two-point-five for a base model on a new format. It drops steeply for the first few hundred steps, then flattens into a gentle decline. The eval loss tracks it, ideally staying below or near the training loss. That gap — train below eval, both descending — is the sign of a healthy steer. The model is generalizing, not memorizing. [SLIDE 6 — The three failure modes] Three failure modes account for almost every bad run you will ever see. Failure one: loss not decreasing. Flat or barely dropping from step one. Three suspects, in order of likelihood. Learning rate too low — the most common cause. Data problem — the chat template isn't applied, or the completion masking is wrong so you're training on the prompts. Or a template bug — the tokenizer's chat template doesn't match the model. All three look identical on the loss curve, so inspect a tokenized batch before assuming it's the LR. Failure two: loss exploding — NaN or a spike. Learning rate too high, the optimizer overshoots, weights blow up. Or numerical instability — the classic FP sixteen bug. FP sixteen has a narrow dynamic range; gradients overflow to infinity during backward. Use BF sixteen instead — same memory, wider range, no overflow on Ampere or later. Or an EOS or template bug producing degenerate targets — if the completion is empty, the loss target is undefined and you get NaN. Failure three: loss plateaus. It descended, then goes flat. The eval loss tells you which. Eval flat and low — converged, stop. Eval flat but high, or eval rising while train drops — stuck or overfitting. Try a higher LR with warmup, a different schedule, or — most often — more and better data. A plateau on bad data is not a hyperparameter problem. It's a data problem. [SLIDE 7 — The grad norm signal] Beyond loss, watch the gradient norm. TRL logs it as grad-norm by default. A healthy run has a stable, slowly-decreasing grad norm. A sudden spike — ten-x between two logging steps — is the early-warning system for instability. Even if the loss hasn't NaN'd yet, that run is about to explode. Kill it, lower the LR, ensure gradient clipping — max-grad-norm, default one-point-zero in TRL, usually fine — and retry. Grad norm is the canary. Loss is the canary's heartbeat. Watch both. [SLIDE 8 — The TrainingArguments levers: LR and batch] Now the knobs on the front panel. Learning rate is the single most important hyperparameter. For full fine-tuning, one-e-minus-five to five-e-minus-five — small steps to avoid catastrophic forgetting. For LoRA, one-e-minus-four to five-e-minus-four — an order of magnitude higher, because adapters are small and start random; they need faster learning to catch up to the frozen base. The cardinal error: using a full-FT LR for LoRA — loss barely moves — or a LoRA LR for full FT — loss explodes. The method, from FT ten, sets the LR band. Mismatching them is the most common "why isn't my model learning" ticket. Batch size plus gradient accumulation. Effective batch equals per-device batch times accumulation steps times world size. When VRAM-limited — and you usually are — you can't fit a large per-device batch. Gradient accumulation simulates a large batch by accumulating gradients across several forward-backward passes before stepping the optimizer. A batch of two with eight accumulation steps on one GPU is an effective batch of sixteen. This is how you get stable optimization on a twenty-four-gig card. [SLIDE 9 — The levers: warmup, scheduler, checkpointing, attention, optimizer] Warmup ratio, typically zero-point-zero-three to zero-point-one. Warmup ramps the LR from zero to the target over the first fraction of steps, stabilizing early training. Skip it and you risk an early spike, especially with a high LR. For LoRA with its higher LRs, warmup is near-mandatory. LR scheduler. Cosine is the common default — smooth decay to near-zero, gentle and reliable. Linear is slightly more aggressive at the end. Constant-with-warmup is flat after warmup, used when you train a fixed budget and will pick the best checkpoint. Cosine is the safe default. Reach for the others when you have a reason. Gradient checkpointing. Trade compute for memory: recompute activations during backward instead of storing them. Cuts activation memory roughly sixty to seventy percent for about a thirty-percent slowdown. This is the FT zero-one VRAM lever — turn it on before you reduce batch size. FlashAttention two or three. Effectively mandatory for speed and memory. It computes exact attention in tiles that fit in SRAM, avoiding the full attention matrix. For long sequences, a two-to-four-x speedup and a large memory saving. Set attn-implementation to flash-attention-two when loading the model. Optimizer. AdamW is the default — robust, well-understood, most memory. AdamW eight-bit, from bitsandbytes, quantizes the optimizer states, saves memory at a small accuracy cost. Adafactor is low-memory but finicky. PagedAdamW uses CUDA paging to offload states to CPU — the QLoRA default, pairs with four-bit quantized bases. [SLIDE 10 — Saving, loading, merging] Three operations, three purposes. Confuse them and you ship the wrong artifact. Save. Trainer dot save-model saves what was trained. With a PEFT config, that's the adapter — a few hundred megabytes, not the multi-gigabyte base. With no PEFT config, it's the entire model. Merge for deployment. Adapters are great for experimentation but inference with a separate adapter has overhead. For deployment, merge the adapter into the base with merge-and-unload. The result is a plain transformers model — no adapter layer — ready to quantize or serve. This is the standard path from "I trained a LoRA" to "I have a deployable model." Load an adapter. For hot-swapping at inference — one base, many adapters — don't merge. Load the adapter on top of the base with PeftModel dot from-pretrained, and swap with load-adapter at runtime. This is the swappability property from FT zero-zero, in code. Layer two detaches from layer one without touching it. [SLIDE 11 — Logging: what to watch] If it isn't logged, it didn't happen. If you can't see the loss curve, you are not training — you are hoping. Backends. Weights and Biases is the long-standing default — rich dashboards, experiment comparison. Trackio is HuggingFace's lighter, Spaces-backed tracker. TensorBoard or JSONL has zero external dependencies; the logs live next to your checkpoints. What to log. At minimum, every step: train loss, learning rate so you can see the schedule, grad norm — the early-warning system. Every eval step: eval loss. Periodically: throughput, tokens per second, so you can spot a slowdown. The single most useful plot is train loss and eval loss on the same chart. The gap between them is your overfitting signal. If train keeps dropping while eval rises, you are memorizing, not steering. Stop at the eval-loss minimum and use that checkpoint. Set load-best-model-at-end to true and let the trainer do this for you. [SLIDE 12 — Anti-patterns] Five anti-patterns to leave with. First, no eval — flying blind. You cannot tell convergence from overfitting, you cannot compare checkpoints, and you will ship a memorizing model. The eval split is not optional. Even two hundred examples is enough to catch the failure. Second, wrong LR for the method. Full-FT LR on a LoRA run — loss barely moves. LoRA LR on a full-FT run — loss explodes. The method sets the LR band. Third, no logging. A run that NaN'd at step fifty and a run that converged at step fifty look identical from the final checkpoint. Log loss, LR, grad norm, eval loss — every step. Fourth, ignoring grad norm spikes. The loss looks fine, but grad norm jumped ten-x. You ignore it. Fifty steps later, NaN. Grad norm is the canary; watch it. Fifth, FP sixteen instead of BF sixteen. On Ampere-or-later hardware that supports BF sixteen, there is no reason to use FP sixteen for training. FP sixteen overflows; BF sixteen doesn't. Same memory, strictly better. If your GPU is pre-Ampere, that's a hardware constraint — plan to upgrade. [SLIDE 13 — The lab: the full SFT loop] The lab is the capstone of pillar two. You run a complete SFT job with TRL's SFTTrainer on Qwen two-point-five three-B or MiniCPM three four-B — your choice. You log to W and B or Trackio. You evaluate on a held-out split every N steps. You produce the loss curve, the eval curve, and a merged, deployable model. You will see the steering effect with your own eyes — generating samples before and after fine-tuning on the same prompts. The base moves. That is the thesis, felt. And if your run NaNs — good. The failure is the lesson. You diagnose it using the loss curve and the grad norm, you fix the lever, and you re-run. That diagnosis is the skill this module is actually teaching. [SLIDE 14 — What you can now do] You can now run a complete SFT job with TRL end to end. You can read a loss curve and diagnose the three failure modes. You can choose the TrainingArguments levers for a given hardware budget and method. You can save, merge, and load a fine-tuned model. And you can wire up logging and know what to watch. That is the training loop. Next, module FT twelve: SFT, the baseline. Now that you can run the loop, we zoom into the method that is the starting point for almost every real fine-tuning project — supervised fine-tuning, done right. --- *End of module FT eleven. Duration: approximately fifty minutes at one-hundred-forty words per minute.*