Fine-tuning
What is Fine-tuning?
Pre-training and Fine-tuning
After pre-training, the model may perform well on next-token prediction but struggle with QA tasks, which are more common in everyday interactions. So we need to adapt the LLM into a chat model.
Training a separate model from scratch for every task requires substantial labeled data. Pre-training provides a shared starting point for fine-tuning on specific downstream tasks.
Instruction Fine-tuning
Training on QA Interactions
Starting from the pre-trained parameters, we minimize the next-token prediction loss on instruction–response pairs using a small learning rate.
What is the task? Learn to follow instructions in QA interactions.
- Reminder: Instruction fine-tuning continues training with a next-token prediction loss, but uses instruction–response pairs.
Aligning with practical tasks
Misalignment can arise from noisy training data and the fact that good next-token prediction does not guarantee good sequence-level performance.
To align the model with practical tasks, one approach is reinforcement fine-tuning: The model generates an answer sequence. A human evaluator scores the whole sequence. The model uses the score to update its parameters.
Reinforcement Fine-tuning
Framework
Given a prompt and a response , suppose we have a human reward score , where higher is better:
To update , we need , but we cannot backpropagate directly through discrete sampling. We therefore use the following algorithms.
REINFORCE algorithm
REINFORCE
For simplicity, fix a prompt and omit it from the notation. We maximize the expected reward:
The log-derivative trick gives:
Treating the reward function as fixed, we have:
We only differentiate the log-probability, so no gradient through the reward or discrete sampling is needed.
It is difficult to compute the expectation exactly, so we estimate it by sampling. At iteration , sample responses and compute their rewards. Estimate the gradient by:
This is an unbiased Monte Carlo estimate of . Update the parameters by gradient ascent with learning rate :
Intuition: Positive rewards encourage the model to generate the sampled responses; negative rewards discourage it.
In implementation, minimize the following loss, treating the sampled responses and rewards as constants:
The minus sign converts gradient ascent into gradient descent. Sample new responses from the updated model for the next iteration.
REINFORCE with a baseline
In the original REINFORCE algorithm, both small and large positive rewards give an upward contribution. We compare rewards with a fixed baseline , independent of the sampled response :
Positive gives an upward contribution; negative gives a downward contribution.
Notice that the baseline contributes zero to the expected gradient:
Replace with in the gradient estimate and loss above.
REINFORCE with a baseline + reuse samples
Still GUNMU
PPO algorithm
Still GUNMU


