✗ AIP1 Isambard-AI Phase 1 unsupported
✓ AIP2 Isambard-AI Phase 2 supported
✗ I3 Isambard 3 unsupported
✗ BC5 BlueCrystal 5 unsupported

LLM Fine-tuning

Abstract

This tutorial introduces fine-tuning a large language model (LLM) on Isambard-AI. It provides a reference for supervised fine-tuning with Quantised Low-Rank Adaptation (QLoRA) using the HuggingFace Transformers Reinforcement Learning (TRL) library. By the end of this tutorial, you will have produced your own fine-tuned Llama 3.1 8B, exported it into the HuggingFace format and served it with the vLLM inference engine.

Prerequisites

We welcome people from all domain backgrounds that have experience:

Training AI models with PyTorch.
Fine-tuning AI models.
Using Apptainer containers.
Using HuggingFace transformers and datasets libraries.

High Performance Computing (Slurm) knowledge is not required.

You will need a HuggingFace account.

We recommend accepting the Llama 3.1 license first, as approval may take some time.

Learning Objectives

The learning objectives of this tutorial are as follows:

Fine-tune a Meta Llama 3.1 8B LLM using QLoRA with the HuggingFace TRL library.
Understand the role that fine-tuning plays in the training process of an LLM.
Understand how and why low-rank adaptation enables efficient fine-tuning.
Evaluate the chat capabilities of a foundation model against a supervised fine-tuned model.

Tutorial Contents¶

Introduction¶

What is fine-tuning?¶

A pre-trained foundation model, such as the one produced in our LLM pre-training tutorial, learns broad language ability and world knowledge from trillions of tokens of general text. However, it has only ever been trained to predict the next token in a stream of text; it has no built-in notion of "following an instruction" or "being helpful". If you prompt a purely pre-trained model with a question, it is just as likely to continue with more questions, or drift into unrelated text, as it is to answer it.

Fine-tuning adapts a pre-trained model towards a narrower goal by continuing training on a smaller, more targeted dataset. Supervised fine-tuning (SFT), the specific technique covered in this tutorial, trains the model on a dataset of prompt-response pairs so that it learns to produce a helpful response to a given instruction, rather than simply continuing the text. This is the third stage in the LLM training pipeline described in our pre-training tutorial's section on how LLMs are trained, sitting between pre-training/mid-training and preference tuning/alignment.

Fine-tuning requires far fewer tokens, and far less compute, than pre-training. Hundreds to low millions of examples, rather than trillions of tokens, are typically enough to noticeably change a model's behaviour, since fine-tuning refines capabilities the model already has, rather than teaching them from scratch.

Parameter-efficient fine-tuning with LoRA¶

Even though fine-tuning uses far less data than pre-training, naively updating every parameter of an LLM, known as full fine-tuning, is still expensive. An optimiser such as AdamW must track two extra values per parameter (its first and second moment estimates), alongside a full-precision copy of the weights, so that small gradient updates are not lost to rounding error. For an 8 billion parameter model such as Llama 3.1 8B, trained in mixed precision, this works out at roughly 16 bytes of memory per parameter: 2 bytes for a bfloat16 copy of the weights, 2 bytes for a bfloat16 gradient, 4 bytes for a full-precision (fp32) master copy of the weights, and 8 bytes for the two fp32 Adam moments. That is around 119 GiB just for the weights and optimiser state, before accounting for activations; more than the 96 GiB of memory available on a single GH200 GPU.

Low-Rank Adaptation (LoRA) avoids this cost by freezing the pre-trained weight matrices entirely and learning a small update alongside them, instead of updating the weights directly. For a frozen weight matrix W of shape d x k, LoRA introduces two much smaller trainable matrices, A (shape r x k) and B (shape d x r), where the rank r (typically between 8 and 64) is chosen to be far smaller than d and k. During the forward pass, the model computes Wx + BAx, scaled by a factor of alpha / r, instead of just Wx. Only A and B are updated during training; W never changes. Because r is small, the number of trainable parameters, and therefore the memory needed for their gradients and optimiser state, shrinks from billions to a tiny fraction of that; typically well under 1% of the base model's parameter count.

Diagram comparing full fine-tuning, LoRA and QLoRA

QLoRA: fine-tuning on a single GPU¶

LoRA still requires the full, frozen base model to be resident in GPU memory. For Llama 3.1 8B in bfloat16, that is around 15 GiB, plus a comparatively small overhead for the LoRA adapters and their optimiser state. QLoRA reduces this further by additionally quantising the frozen base weights to 4-bit precision, using the NormalFloat4 (NF4) data type implemented by the bitsandbytes library. Each weight is dequantised back to bfloat16 on the fly for every matrix multiplication during the forward and backward pass; the LoRA adapters themselves remain in bfloat16 throughout, so only the much larger set of frozen weights benefits from the reduced footprint.

Memory required to fine-tune Llama 3.1 8B

Method	Approx. memory for weights + optimiser state
Full fine-tuning (bfloat16 weights, fp32 optimiser state)	~119 GiB
LoRA (bfloat16 frozen weights, trainable adapters)	~16 GiB
QLoRA (4-bit NF4 frozen weights, trainable adapters)	~5 GiB

Figures exclude activation memory. Each GH200 in Isambard-AI provides 96 GiB of GPU memory. Full fine-tuning of an 8 billion parameter model does not fit on a single GPU without additional sharding techniques, which are outside the scope of this tutorial, whereas both LoRA and QLoRA comfortably do. This is why, unlike the multi-node pre-training tutorial, this tutorial only needs a single GPU.

Meta Llama 3.1 8B¶

We use the same Llama 3.1 8B model as our pre-training tutorial; see that tutorial for background on the model and its Transformer architecture. Unlike the pre-training tutorial, where you trained an under-trained checkpoint from scratch for a limited number of iterations, here we start from Meta's fully pre-trained release of the model, downloaded directly from HuggingFace. This lets us observe a meaningful change in behaviour after fine-tuning, since the base (non-instruction-tuned) Llama 3.1 8B model has broad language ability, but has not been trained to follow instructions.

Tools and frameworks¶

HuggingFace TRL, PEFT and bitsandbytes¶

Our pre-training tutorial used NVIDIA's Megatron-LM and Megatron Bridge frameworks, designed for maximum throughput when training large models across many GPUs and nodes. Fine-tuning with LoRA or QLoRA has very different requirements: the workload fits on a single GPU, and ease of experimentation matters more than multi-node throughput. For this tutorial, we instead use a set of lighter-weight libraries from the HuggingFace ecosystem, illustrating that there is no single correct toolchain for training LLMs; the right choice depends on the scale and nature of the job.

Transformer Reinforcement Learning (TRL) provides ready-made trainer classes for the post-training stages of the LLM pipeline, including the SFTTrainer we use in this tutorial for supervised fine-tuning.
Parameter-Efficient Fine-Tuning (PEFT) implements LoRA, and other parameter-efficient fine-tuning methods, as a thin wrapper around HuggingFace transformers models.
bitsandbytes provides the 4-bit and 8-bit quantisation kernels, including the NF4 data type, that make QLoRA possible.

These libraries are designed to interoperate: SFTTrainer accepts a peft_config argument to automatically wrap a model with LoRA adapters, and a quantization_config to load the base model in 4-bit precision via bitsandbytes.

The Stanford Alpaca dataset¶

We fine-tune on Stanford Alpaca, a dataset of 52,000 instruction-following examples. Each example contains an instruction, an optional input providing further context, and an expected response, originally generated by prompting OpenAI's text-davinci-003 model and then filtering the results. Alpaca's HuggingFace release additionally pre-formats every example into a single block of text, combining the instruction, input, and response into the prompt template Stanford used to collect the data; we use this pre-formatted text column directly, rather than writing our own formatting function.

What does an Alpaca example look like?

Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
Give three tips for staying healthy.

### Response:
1. Eat a balanced diet and make sure to include ...

Walkthrough¶

Now that we have covered the required prior knowledge, we can begin setting up our LLM fine-tuning experiment.

Authorise with HuggingFace¶

This step is necessary to complete this tutorial.

The Llama family of models are gated on HuggingFace. Users need to agree to the terms of the licensing agreement to access the model weights, tokenizer and other configuration files.

Go to the Meta Llama 3.1 8B model card and request access.

You will need to wait for your request to be approved. This may take some time.

Create a HuggingFace access token with the permission:

Read access to contents of all public gated repos you can access

Install uv and run:
```
uvx hf auth login
```

Set up working environment¶

We set the WORK_PATH environment variable, referenced throughout the scripts used in this tutorial, to define our working directory in the /projects directory. We also create this directory and move into it.

user@login43:~> export WORK_PATH=${PROJECTDIR}/${USER}/llm-fine-tuning
user@login43:~> mkdir -p ${WORK_PATH} && cd ${WORK_PATH}
user@login43:/projects/PROJECT/USER/llm-fine-tuning>

Set up the Python environment¶

Unlike the container-based approach in the pre-training tutorial, here we install our Python packages directly with uv, into two separate virtual environments: one for training, and one for serving with vLLM. We use two environments because vllm and trl can require different, and sometimes conflicting, versions of transformers and torch; keeping them isolated avoids dependency conflicts.

Since these packages need to detect a GPU correctly while installing, we build both environments as a Slurm job rather than on the login node.

build_environments.sh

#!/bin/bash
#SBATCH --job-name=build-environments
#SBATCH --nodes=1
#SBATCH --gpus=1
#SBATCH --time=00:30:00
#SBATCH --output=out/%x.%j.out

export WORK_PATH=${WORK_PATH:-${PROJECTDIR}/${USER}/llm-fine-tuning}

uv venv --seed --python 3.12 ${WORK_PATH}/.venv-train
source ${WORK_PATH}/.venv-train/bin/activate
uv pip install \
  torch==2.11.0 \
  transformers==5.15.1 \
  trl==1.10.0 \
  peft==0.20.0 \
  bitsandbytes==0.50.1 \
  accelerate==1.14.0 \
  datasets==5.0.1 \
  sympy==1.14.0 \
  --index-url https://download.pytorch.org/whl/cu128 \
  --extra-index-url https://pypi.org/simple \
  --index-strategy unsafe-best-match
deactivate

uv venv --seed --python 3.12 ${WORK_PATH}/.venv-serve
source ${WORK_PATH}/.venv-serve/bin/activate
uv pip install vllm==0.24.0 --extra-index-url https://wheels.vllm.ai/0.24.0/cu129 --extra-index-url https://download.pytorch.org/whl/cu129 --index-strategy unsafe-best-match
deactivate

All of these packages provide aarch64 wheels, so no special build flags are required. We pin exact versions of everything here, since fast-moving libraries like transformers and trl regularly make breaking changes to their APIs between releases; the versions above are the ones this tutorial was tested against. See our Machine Learning packages page if you run into any issues, or if you would prefer to track the latest releases instead of these pinned versions.

Run the following commands to download the job script, then launch it:

$ curl -fsSL -o ${WORK_PATH}/build_environments.sh https://docs.isambard.ac.uk/user-documentation/tutorials/llms/fine-tuning/build_environments.sh
$ sbatch ${WORK_PATH}/build_environments.sh
Submitted batch job JOBID

You can monitor progress by viewing the logs at ${WORK_PATH}/out/build-environments.JOBID.out. Once complete, you should have two virtual environments, ${WORK_PATH}/.venv-train and ${WORK_PATH}/.venv-serve, ready to use in the rest of this tutorial.

Run QLoRA fine-tuning - 15 minutes¶

We are almost ready to begin fine-tuning Llama 3.1 8B with QLoRA.

Training script¶

Our training script, train.py, loads Llama 3.1 8B in 4-bit precision, wraps it with a LoRA adapter, and fine-tunes it on Alpaca using TRL's SFTTrainer.

Training script

train.py

import os

from datasets import load_dataset
from peft import LoraConfig
from transformers import AutoTokenizer, BitsAndBytesConfig
from trl import SFTConfig, SFTTrainer


WORK_PATH = os.environ["WORK_PATH"]
MODEL_ID = "meta-llama/Llama-3.1-8B"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
tokenizer.pad_token = tokenizer.eos_token

dataset = load_dataset("tatsu-lab/alpaca", split="train")

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype="bfloat16",
    bnb_4bit_use_double_quant=True,
)

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules="all-linear",
)

training_args = SFTConfig(
    output_dir=f"{WORK_PATH}/sft-output",
    dataset_text_field="text",
    max_length=1024,
    packing=False,
    max_steps=200,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    gradient_checkpointing=True,
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_steps=10,
    logging_steps=10,
    save_steps=100,
    bf16=True,
    report_to="none",
    model_init_kwargs={
        "quantization_config": quantization_config,
        "dtype": "bfloat16",
    },
)

trainer = SFTTrainer(
    model=MODEL_ID,
    args=training_args,
    train_dataset=dataset,
    processing_class=tokenizer,
    peft_config=lora_config,
)

print(trainer.model.print_trainable_parameters())

trainer.train()
trainer.save_model(f"{WORK_PATH}/sft-output/final")

Run the following command to download the script:

$ curl -fsSL -o ${WORK_PATH}/train.py https://docs.isambard.ac.uk/user-documentation/tutorials/llms/fine-tuning/train.py

The BitsAndBytesConfig object tells transformers to load the base model's weights in 4-bit NF4 precision, as described in QLoRA above. The LoraConfig object then configures the LoRA adapters themselves: r=16 sets the rank of the low-rank update, target_modules="all-linear" applies an adapter to every linear layer in the model, and lora_alpha=32 sets the scaling factor applied to the adapter's output.

Llama 3.1's tokenizer has no dedicated padding token by default, so we reuse the end-of-sequence token for padding; this is a common convention when fine-tuning causal language models in batches.

We leave packing disabled. Packing concatenates multiple short examples into a single sequence to reduce wasted compute on padding, but doing so safely requires an attention implementation that masks each packed example separately, such as flash_attention_2. Without one, attention can leak across the boundary between two unrelated examples packed into the same sequence, which we would rather avoid than build Flash Attention from source for this tutorial; see our Machine Learning packages page if you want to pursue that yourself.

We limit training to max_steps=200, an effective batch size of 4 x 4 = 16 sequences, so that the tutorial completes in a reasonable time. This covers only a small fraction of Alpaca's 52,000 examples; see Reflections below for more on this trade-off.

Bring your own dataset

SFTTrainer supports a few different dataset shapes, and which one suits your own data best depends on how it's structured.

prompt and completion columns are the best fit if you are fine-tuning a base model like we are in this tutorial. SFTTrainer detects these columns automatically and, unlike Alpaca's pre-formatted text column, masks the loss so training only happens on the completion tokens rather than the whole prompt:
```
dataset = load_dataset("json", data_files="my_data.jsonl", split="train")
# each line: {"prompt": "...", "completion": "..."}
```
Drop dataset_text_field="text" from SFTConfig when using this format; it is not used.
A messages column, in the same role/content format used by chat APIs, lets SFTTrainer apply the tokenizer's chat template automatically. This only works if the tokenizer actually has a chat template, which base checkpoints such as the meta-llama/Llama-3.1-8B model used in this tutorial do not ship with; it suits fine-tuning an Instruct variant instead.
A custom formatting_func, for any other column layout, such as Alpaca's original instruction/input/output columns before they were pre-formatted into a text column. This is a function passed to SFTTrainer itself, not SFTConfig, that turns one example into a training string:
```
def formatting_func(example):
    return f"### Instruction:\n{example['instruction']}\n\n### Response:\n{example['output']}"

trainer = SFTTrainer(..., formatting_func=formatting_func)
```

Whichever shape you use, datasets can load it from a local file (load_dataset("json", data_files=...), also "csv", "parquet", or "text"), from Python objects already in memory (Dataset.from_list(...), Dataset.from_pandas(...)), or from your own dataset on the HuggingFace Hub, the same way we loaded tatsu-lab/alpaca above.

Slurm job script¶

Unlike our pre-training tutorial, this job runs on a single GPU and does not need Apptainer, NCCL tuning, or multi-node coordination, since QLoRA's memory savings mean the whole fine-tuning run fits comfortably within one GH200.

sft.sh

#!/bin/bash
#SBATCH --job-name=llama-qlora-sft
#SBATCH --nodes=1
#SBATCH --gpus=1
#SBATCH --time=00:30:00
#SBATCH --exclusive
#SBATCH --output=out/%x.%j.out

export WORK_PATH=${WORK_PATH:-${PROJECTDIR}/${USER}/llm-fine-tuning}
export HF_DATASETS_CACHE=${HF_DATASETS_CACHE:-${WORK_PATH}/.cache/huggingface/datasets}
export HF_HUB_CACHE=${HF_HUB_CACHE:-${WORK_PATH}/.cache/huggingface/hub}

source ${WORK_PATH}/.venv-train/bin/activate

srun python ${WORK_PATH}/train.py

Run the following command to download the job script:

$ curl -fsSL -o ${WORK_PATH}/sft.sh https://docs.isambard.ac.uk/user-documentation/tutorials/llms/fine-tuning/sft.sh

Launching our job¶

Finally, we can launch our QLoRA fine-tuning job with Slurm:

$ sbatch ${WORK_PATH}/sft.sh
Submitted batch job JOBID

You can view the logs in the ${WORK_PATH}/out/llama-qlora-sft.JOBID.out file. Shortly after the job starts, PEFT will print a summary of how many parameters are trainable, something like:

trainable params: 41,943,040 || all params: 8,072,204,288 || trainable%: 0.5196

This confirms that only around half a percent of the model's 8 billion parameters are actually being updated; everything else stays frozen and quantised, exactly the memory saving described in QLoRA above. As training proceeds, SFTTrainer will log the training loss every 10 steps; you should see it decrease over the course of the run, showing that the LoRA adapter is learning to follow Alpaca's instruction-response format.

Once training completes, our LoRA adapter is saved to ${WORK_PATH}/sft-output/final:

$ ls ${WORK_PATH}/sft-output/final/
adapter_config.json  adapter_model.safetensors  tokenizer.json  tokenizer_config.json  ...

Note that this directory contains only the small LoRA adapter weights, not a full copy of Llama 3.1 8B; adapter_model.safetensors is a few hundred megabytes, rather than the ~16 GiB of the full model.

Merge the LoRA adapter and export - 5 minutes¶

A LoRA adapter is only useful alongside the base model it was trained against. vLLM can serve LoRA adapters directly, but to keep our comparison step simple and to produce a model we can serve like any other HuggingFace checkpoint, we merge the adapter's weights back into the base model, producing a single, self-contained set of .safetensors files.

merge_and_export.py

import os

import torch

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer


WORK_PATH = os.environ["WORK_PATH"]
MODEL_ID = "meta-llama/Llama-3.1-8B"
ADAPTER_PATH = f"{WORK_PATH}/sft-output/final"
MERGED_PATH = f"{WORK_PATH}/sft-output/merged"

base_model = AutoModelForCausalLM.from_pretrained(MODEL_ID, dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

model = PeftModel.from_pretrained(base_model, ADAPTER_PATH)
model = model.merge_and_unload()

model.save_pretrained(MERGED_PATH)
tokenizer.save_pretrained(MERGED_PATH)

Run the following command to download the script:

$ curl -fsSL -o ${WORK_PATH}/merge_and_export.py https://docs.isambard.ac.uk/user-documentation/tutorials/llms/fine-tuning/merge_and_export.py

Note that we reload the base model in bfloat16 here, rather than in 4-bit. PEFT's merge_and_unload combines the adapter directly into the base weights, so we need a full-precision copy of those weights to merge into; this step briefly needs more memory than training did, but Llama 3.1 8B in bfloat16 still comfortably fits on a single GH200.

merge.sh

#!/bin/bash
#SBATCH --job-name=llama-merge-lora
#SBATCH --nodes=1
#SBATCH --gpus=1
#SBATCH --time=00:10:00
#SBATCH --exclusive
#SBATCH --output=out/%x.%j.out

export WORK_PATH=${WORK_PATH:-${PROJECTDIR}/${USER}/llm-fine-tuning}
export HF_DATASETS_CACHE=${HF_DATASETS_CACHE:-${WORK_PATH}/.cache/huggingface/datasets}
export HF_HUB_CACHE=${HF_HUB_CACHE:-${WORK_PATH}/.cache/huggingface/hub}

source ${WORK_PATH}/.venv-train/bin/activate

srun python ${WORK_PATH}/merge_and_export.py

Run the following command to download the job script, then launch it:

$ curl -fsSL -o ${WORK_PATH}/merge.sh https://docs.isambard.ac.uk/user-documentation/tutorials/llms/fine-tuning/merge.sh
$ sbatch ${WORK_PATH}/merge.sh
Submitted batch job JOBID

You can monitor progress by viewing the logs at ${WORK_PATH}/out/llama-merge-lora.JOBID.out. Once complete, you should see a merged, HuggingFace-format model at ${WORK_PATH}/sft-output/merged.

$ ls ${WORK_PATH}/sft-output/merged/
config.json  generation_config.json  model.safetensors  tokenizer_config.json  tokenizer.json

Compare the base and fine-tuned models - 15 minutes¶

The final step in this tutorial is to use vLLM to compare chat completions from the base Llama 3.1 8B model against our fine-tuned version.

vLLM serving script

serve_vllm.sh

#!/bin/bash
#SBATCH --job-name=llama-serve-vllm
#SBATCH --nodes=1
#SBATCH --gpus=1
#SBATCH --time=00:15:00
#SBATCH --exclusive
#SBATCH --output=out/%x.%j.out

module load cudatoolkit

export WORK_PATH=${WORK_PATH:-${PROJECTDIR}/${USER}/llm-fine-tuning}
export HF_DATASETS_CACHE=${HF_DATASETS_CACHE:-${WORK_PATH}/.cache/huggingface/datasets}
export HF_HUB_CACHE=${HF_HUB_CACHE:-${WORK_PATH}/.cache/huggingface/hub}
export MODEL_PATH=${MODEL_PATH:?Set MODEL_PATH when submitting this job, e.g. sbatch --export=ALL,MODEL_PATH=... serve_vllm.sh}
export SERVED_NAME=${SERVED_NAME:-llama-3.1-8b}

export CC=/usr/bin/gcc-13
export CXX=/usr/bin/g++-13

source ${WORK_PATH}/.venv-serve/bin/activate

srun vllm serve ${MODEL_PATH} \
     --served-model-name ${SERVED_NAME} \
     --max-model-len 1024 \
     --gpu-memory-utilization 0.9

wait

Run the following command to download the job script:

$ curl -fsSL -o ${WORK_PATH}/serve_vllm.sh https://docs.isambard.ac.uk/user-documentation/tutorials/llms/fine-tuning/serve_vllm.sh

This script reads a MODEL_PATH and SERVED_NAME from its environment, so we can reuse it for both models. First, launch it to serve the unmodified base model, straight from HuggingFace:

$ MODEL_PATH=meta-llama/Llama-3.1-8B SERVED_NAME=llama-3.1-8b-base sbatch ${WORK_PATH}/serve_vllm.sh
Submitted batch job JOBID

This will take a few minutes. You can monitor progress by viewing the logs at ${WORK_PATH}/out/llama-serve-vllm.JOBID.out; the server is ready once you see Application startup complete.

Now, we can enter an interactive prompt to get a completion from the base model. Replace JOBID with the ID of your serve_vllm.sh job in the command below.

$ source .venv-serve/bin/activate
$ srun --overlap --jobid JOBID --gpus=1 --pty vllm complete --model llama-3.1-8b-base --max-tokens 512
Please enter prompt to complete:
> Below is an instruction that describes a task. Write a response that appropriately completes the request. ### Instruction: Give three tips for staying healthy. ### Response:
1. Eating healthy food is the first step to a healthy body. 2. Exercising regularly is the second step to a healthy body. 3. Getting enough sleep is the third step to a healthy body. ### Instruction: Give three tips for staying healthy. ### Response: 1. Exercise regularly. 2. Eat healthy foods. 3. Get enough sleep.
Below is an instruction that describes a task. Write a response that appropriately completes the request. ### Instruction: Give three tips for staying healthy. ### Response: 1. Eating healthy food is the first step to a healthy body. 2. Exercising regularly is the second step to a healthy body. 3. Getting enough sleep is the third step to a healthy body. ### Instruction: Give three tips for staying healthy. ### Response: 1. Exercise regularly. 2. Eat healthy foods. 3. Get enough sleep.
Below is an instruction that describes a task. Write a response that appropriately completes the request. ### Instruction: Give three tips for staying healthy. ### Response: 1. Eating healthy food is the first step to a healthy body. 2. Exercising regularly is the second step to a healthy body. 3. Getting enough sleep is the third step to a healthy body. ### Instruction: Give three tips for staying healthy. ### Response: 1. Exercise regularly. 2. Eat healthy foods. 3. Get enough sleep.
Below is an instruction that describes a task. Write a response that appropriately completes the request. ### Instruction: Give three tips for staying healthy. ### Response: 1. Eating healthy food is the first step to a healthy body. 2. Exercising regularly is the second step to a healthy body. 3. Getting enough sleep is the third step to a healthy body. ### Instruction: Give three tips for staying healthy. ### Response: 1. Exercise regularly. 2. Eat healthy foods. 3. Get enough sleep.

Because the base model has never been trained to follow this prompt format, its completion can be inconsistent; it may drift off-topic, trail off, or start generating further instructions rather than answering the one it was given, since it has seen similar-looking text during pre-training without ever being taught to treat it as a task to complete.

Type CTRL+C to exit the interactive CLI, then cancel this job and launch a second one serving our fine-tuned model:

$ scancel JOBID
$ MODEL_PATH=${WORK_PATH}/sft-output/merged SERVED_NAME=llama-3.1-8b-sft sbatch ${WORK_PATH}/serve_vllm.sh
Submitted batch job JOBID

Once this job's server is ready, enter the same prompt again, replacing JOBID with this new job's ID:

$ srun --overlap --jobid JOBID --gpus=1 --pty vllm complete --model llama-3.1-8b-sft --max-tokens 512
Please enter prompt to complete:
> Below is an instruction that describes a task. Write a response that appropriately completes the request. ### Instruction: Give three tips for staying healthy. ### Response:
1. Eat a balanced diet. 2. Get regular exercise. 3. Get enough sleep.

This time, the model should reliably produce a direct, structured response to the instruction, such as a short numbered list of health tips, rather than drifting or continuing with further instructions. This difference in behaviour, despite both models sharing the same pre-trained weights for all but a small fraction of their parameters, is the practical effect of supervised fine-tuning.

Type CTRL+C to exit, then cancel the job with scancel JOBID.

Reflections¶

Our fine-tuning run trained on 200 x 16 = 3,200 example sequences, drawn from Alpaca's 52,000 examples; at this batch size, a single full pass (epoch) over the dataset would take around 3,250 steps, so our run covered under 6% of one epoch. Despite this, the change in the model's behaviour towards following instructions was clear in the previous step. This illustrates a key difference from pre-training: because the base model already has strong general language ability, fine-tuning only needs to shift its behaviour, not teach it new knowledge from scratch, so comparatively little data goes a long way.

We also saw that only around 0.5% of the model's parameters were ever updated, thanks to LoRA, and that those parameters were the only ones requiring full-precision gradients and optimiser state; the remaining 99.5% of the model stayed frozen and quantised to 4 bits throughout. This is what allowed the entire fine-tuning run to fit on a single GH200, in contrast to the two nodes and eight GPUs used by our pre-training tutorial for a similarly sized model.

Going further

In production, supervised fine-tuning is typically followed by a preference tuning or alignment stage, such as Direct Preference Optimisation (DPO), to further refine a model's behaviour towards human preferences; see the pre-training tutorial's discussion of the LLM training pipeline for where this fits in. TRL provides trainer classes for several of these methods, following the same pattern as the SFTTrainer used in this tutorial. We will not cover alignment here.

Storage cleanup

The merged model exported in this tutorial is a full copy of Llama 3.1 8B, so it uses close to the same amount of storage as the original model (~16 GiB). You may wish to clean up your working directory after you have finished with it.

Conclusion¶

In this tutorial, you've learned how to fine-tune Meta's Llama 3.1 8B large language model using QLoRA on Isambard-AI. We covered the role that supervised fine-tuning plays in the LLM training pipeline, and how LoRA and QLoRA make it practical to fine-tune an 8 billion parameter model on a single GPU by freezing and quantising almost all of the model's weights.

We then saw how the HuggingFace TRL, PEFT and bitsandbytes libraries work together to implement this in practice, a deliberately different toolchain from the NVIDIA Megatron frameworks used in our pre-training tutorial, reflecting how the right tools for training an LLM depend on the scale of the job at hand. After fine-tuning on the Stanford Alpaca dataset, you merged the resulting LoRA adapter back into the base model and used vLLM to compare chat completions from the base and fine-tuned models directly, observing the practical effect that supervised fine-tuning has on a model's behaviour.

Resources¶

Datasets and models¶

Publications¶

Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). QLoRA: Efficient finetuning of quantized LLMs. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023). https://doi.org/10.48550/arXiv.2305.14314