如何使用SFT和QLoRA为AI代理定制LLM

✍️ zhirenhun 📅 2026/8/9 👁 199 阅读 ⏱ 51 分钟

在本教程中，我将向你展示如何使用QLoRA进行监督微调，为大语言模型在AI代理中的应用进行定制。这使我们能够自定义预训练模型，使其按照我们的需求行事。我们将使用轻量级训练流程，仅更新模型的一小部分。

我们将使用Unsloth和Hugging Face生态系统下载Qwen 1.5B基础模型，应用基于QLoRA的监督微调，并将生成的LoRA适配器权重保存在本地以供推理。整个过程在本地运行，因此无需支付模型API费用。

背景

训练语言模型意味着向它展示许多示例并更新其内部权重（称为参数），以便它更好地预测期望的输出。现代LLM可能拥有数百万或数十亿个参数，这也是训练它们成本高昂的原因之一。模型参数越多，训练所需的存储和计算通常就越多。

像Claude和ChatGPT这样的基础大语言模型在训练时也是面向通用目的的。这意味着它们的回复可能显得宽泛、不一致，或者与特定应用不够贴合。即使提示有助于改善，但在某些情况下，你希望模型直接从示例中学习更一致的模式。

这正是微调的用武之地。微调是使预训练模型更贴近你的任务行为的一般过程。其中一种常见形式是监督微调，即模型在带标签的输入/输出示例上进行训练，这些示例展示了你想要的行为类型。

本教程适用于macOS、Windows和Linux。我使用的是配备32 GB RAM且无外部GPU的MacBook Pro，但该工作流也可以通过使用较小的预训练模型在更有限的硬件上运行。

什么是监督微调？

监督微调（SFT）是指对预训练模型进行进一步训练，使用示例输入/输出对。并非从头开始训练模型，而是从一个已经较好理解语言的模型开始，教它以更符合你任务的方式做出响应。例如，你可能希望它以某种语气回答、遵循特定格式，或在狭窄任务上表现更一致。SFT通过向模型展示许多你想要的行为示例，推动模型朝这个方向前进。

你所需的数据量取决于任务。对于语气或格式等简单变更，几百个好的示例就已经足够。对于更复杂的行为或领域适配，你通常需要更多精挑细选的示例。

在本教程中，我们将使用五个示例来保持训练快速且简单，但同样的代码可以在实际生产工作流中用于更大的数据集。

什么是LoRA？

完整微调可能成本高昂，因为大语言模型拥有海量参数。更新所有这些参数需要大量GPU内存、计算时间和存储空间。

LoRA（低秩适配）是一种更轻量的模型微调方式。它是最常见的参数高效微调（PEFT）方法之一，这意味着它在不更新所有原始权重的情况下适配预训练模型。基础模型大部分保持冻结，而LoRA在其之上添加一组更小的可训练适配器权重。

在本教程中，我们将使用QLoRA，它将量化与LoRA结合，以低精度（通常为4位）加载基础模型，然后训练那些LoRA适配器。这进一步减少了内存使用，使微调在有限硬件上更加实用。

我们还将使用一个名为Unsloth的开源库，该库旨在使大语言模型微调更快、更节省内存。它从Hugging Face下载模型权重、分词器和配置，常用于诸如使用LoRA进行监督微调等工作流，尤其是在硬件有限时。

动机与架构

一旦你构建了一个AI代理，你可能会发现现成的模型需要冗长的提示、重复的指令和额外的上下文，才能为你的用例产生所需的输出。这可能会增加令牌使用量、延迟和成本，同时仍然给出不一致的结果。在这种情况下，自然的下一步就是训练模型以更贴合任务的方式做出响应。

架构是：加载量化基础模型，格式化标记的聊天示例，添加LoRA适配器，仅使用监督微调训练这些适配器，并保存生成的适配器权重，以便以后在AI代理中加载到基础模型之上进行推理。代码将在以下部分解释。

第1步：安装Python依赖

创建虚拟环境并安装所需包：

python3 -m venv venv
source venv/bin/activate

pip install unsloth datasets transformers trl torch peft accelerate bitsandbytes

第2步：训练代码

我们将使用Qwen2.5-1.5B-Instruct，因为它是一个小型指令微调模型，适合在有限硬件上进行QLoRA微调。

在较高层面上，代码以4位模式加载预训练的unsloth/Qwen2.5-1.5B-Instruct模型，这正是使其成为QLoRA风格工作流并保持GPU内存占用较低的原因。

然后，它获取一个包含五个聊天式客户支持示例的小型监督微调数据集（在文件顶部定义），使用Qwen的聊天模板对它们进行格式化，并将LoRA适配器附加到选定的注意力和MLP层。

在训练期间，SFTTrainer仅更新那些小的LoRA适配器权重，而不更新完整的基础模型，并使用内存效率高的设置，如梯度累积、8位AdamW和Unsloth梯度检查点。

将其保存为train.py。我在代码中加入了注释，以帮助您理解其工作原理：

from unsloth import FastLanguageModel
from datasets import Dataset
from transformers import TrainingArguments
from trl import SFTConfig, SFTTrainer

max_seq_length = 2048

dataset = Dataset.from_list([
    {
        "messages": [
            {"role": "system",
             "content": "You are a helpful ecommerce customer support assistant. Be polite, concise, and do not invent order details."},
            {"role": "user", "content": "Where is my order?"},
            {"role": "assistant", "content": "I can help with that. Please share your order number so I can check the latest shipping status."}
        ]
    },
    {
        "messages": [
            {"role": "system",
             "content": "You are a helpful ecommerce customer support assistant. Be polite, concise, and do not invent order details."},
            {"role": "user", "content": "My package arrived damaged. What should I do?"},
            {"role": "assistant", "content": "I’m sorry about that. Please upload a photo of the damaged item and packaging, and I’ll guide you through a replacement or refund request."}
        ]
    },
    {
        "messages": [
            {"role": "system",
             "content": "You are a helpful ecommerce customer support assistant. Be polite, concise, and do not invent order details."},
            {"role": "user", "content": "Can I return an item after 45 days?"},
            {"role": "assistant", "content": "Our standard return window is 30 days. If your item is defective or there was a shipping issue, please share the details and I can help review possible exceptions."}
        ]
    },
    {
        "messages": [
            {"role": "system",
             "content": "You are a helpful ecommerce customer support assistant. Be polite, concise, and do not invent order details."},
            {"role": "user", "content": "I entered the wrong shipping address."},
            {"role": "assistant", "content": "If the order has not shipped yet, we may be able to update the address. Please send the order number and the correct shipping address as soon as possible."}
        ]
    },
    {
        "messages": [
            {"role": "system",
             "content": "You are a helpful ecommerce customer support assistant. Be polite, concise, and do not invent order details."},
            {"role": "user", "content": "I was charged twice for my order."},
            {"role": "assistant", "content": "I’m sorry for the trouble. Please share your order number and the dates of the charges so we can review the duplicate payment and help resolve it quickly."}
        ]
    },
])

# Load a pretrained Qwen instruction model.
# Unsloth automatically configures efficient loading.
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Qwen2.5-1.5B-Instruct",  # Pretrained model to load from Hugging Face / Unsloth

    max_seq_length=max_seq_length,               # Maximum sequence length the model should be prepared for
                                                 # Longer context = more memory usage

    load_in_4bit=True,                           # Load model weights in 4-bit quantized form
                                                 # Greatly reduces VRAM usage for training/inference
                                                 # Common for LoRA / QLoRA workflows

    dtype=None,                                  # Let Unsloth / Torch auto-pick the numeric precision
                                                 # Often chooses something suitable like float16/bfloat16
)


def format_example(example):
    text = tokenizer.apply_chat_template(
        example["messages"],          # Read the conversation from the "messages" field
        tokenize=False,               # Return a formatted string, not token IDs yet
        add_generation_prompt=False,  # Do not append an empty assistant prompt
                                      # because this example already includes the assistant response
    )
    return {"text": text}            # Return a new dataset field containing the formatted chat text


formatted_dataset = dataset.map(format_example)

# Instead of training billions of parameters,
# LoRA inserts small trainable matrices into attention layers.

model = FastLanguageModel.get_peft_model(
    model,  # Base pretrained model; LoRA adapters will be attached here

    r=16,   # LoRA rank:
            # size of the low-rank adapter matrices
            # higher = more capacity + more trainable params
            # lower = lighter/faster but less expressive

    target_modules=[
        "q_proj",    # Query projection in attention
        "k_proj",    # Key projection in attention
        "v_proj",    # Value projection in attention
        "o_proj",    # Output projection in attention
        "gate_proj", # Gating projection in MLP block
        "up_proj",   # Up projection in MLP block
        "down_proj", # Down projection in MLP block
    ],  # LoRA adapters are inserted only into these layers

    lora_alpha=16,  # LoRA scaling factor
                    # controls how strongly adapter updates affect the base weights
                    # often set equal to r

    lora_dropout=0, # Dropout on LoRA path during training
                    # 0 is common in Unsloth examples

    bias="none",    # Do not train bias parameters
                    # only LoRA adapter weights will be trainable

    use_gradient_checkpointing="unsloth",  # Use Unsloth's memory-saving checkpointing
                                           # lowers VRAM usage by recomputing activations during backprop

    max_seq_length=max_seq_length,  # Maximum token sequence length expected during training
)


trainer = SFTTrainer(
    model=model,                      # The model to fine-tune (base model + LoRA adapters)
    tokenizer=tokenizer,              # Converts text into token IDs the model can understand
    train_dataset=formatted_dataset,            # Your training data
    dataset_text_field="text",        # Column in the dataset that contains the training text
    max_seq_length=max_seq_length,    # Maximum number of tokens per example

    args=SFTConfig(
        output_dir="../outputs",         # Folder where checkpoints/logs/results will be saved

        per_device_train_batch_size=2, # Number of examples processed at once on each GPU
        gradient_accumulation_steps=4, # Accumulate gradients for 4 mini-batches before updating weights
                                       # Effective batch size ~= 2 * 4 = 8 on 1 GPU

        max_steps=30,                 # Stop training after 10 optimizer update steps
        logging_steps=1,              # Print/log training metrics every 1 step

        warmup_steps=5,               # Gradually increase learning rate for first 5 steps
        learning_rate=2e-4,           # Main learning rate for training

        optim="adamw_8bit",           # Memory-efficient AdamW optimizer (good for low VRAM setups)
        weight_decay=0.01,            # Small regularization to help prevent overfitting
        lr_scheduler_type="linear",   # After warmup, reduce learning rate linearly over time

        seed=3407,                    # Random seed for more reproducible training
        report_to="none",             # Disable external logging tools like WandB
    ),
)

trainer.train()


# Saves only the LoRA adapter weights, not the full base model.
model.save_pretrained("qwen2_0_5b_lora")

# Save the tokenizer so inference uses the same vocabulary.
tokenizer.save_pretrained("qwen2_0_5b_lora")

第3步：推理代码

从高层次来看，推理代码包含 generate_reply() 函数，该函数使用 Unsloth 加载模型（可以选择从基础模型名称或本地保存的 LoRA 适配器目录加载），启用推理优化，将聊天消息格式化为 Qwen 期望的提示结构，对该提示进行分词，将其移动到可用设备，然后通过 model.generate() 生成回复。

将其保存为 inference.py：

from unsloth import FastLanguageModel
import torch

messages = [
    {
        "role": "system",
        "content": "You are a helpful ecommerce customer support assistant. Be polite, concise, and do not invent order details."
    },
    {
        "role": "user",
        "content": "I want to cancel my order."
    }
]


def generate_reply(model_name, messages):
    # Load the base model and automatically attach the saved LoRA adapter.
    # "qwen2_0_5b_lora" is the directory created by model.save_pretrained().
    model, tokenizer = FastLanguageModel.from_pretrained(
        model_name=model_name,  # Path or model name for your fine-tuned LoRA model/adapters
        max_seq_length=2048,  # Maximum context length the model should support. Longer context uses more memory
        load_in_4bit=True,  # Load weights in 4-bit quantized form. Reduces VRAM usage during inference
    )

    # Enable inference optimizations (faster generation, lower memory usage).
    FastLanguageModel.for_inference(model)


    # Convert the chat messages into the format expected by Qwen.
    inputs = tokenizer.apply_chat_template(
        messages,                       # List of chat messages: system / user / assistant turns
        tokenize=True,                  # Convert the formatted chat prompt into token IDs
        add_generation_prompt=True,     # Add the assistant prompt so the model knows to generate a reply
        return_tensors="pt",            # Return PyTorch tensors
    )

    # Move the input tensor to the same device as the model
    device = "cuda" if torch.cuda.is_available() else "cpu"
    inputs = inputs.to(device)

    # Generate the assistant's response.
    outputs = model.generate(
        input_ids=inputs,               # Tokenized prompt passed into the model
        max_new_tokens=80,              # Generate up to 80 new tokens in the response
        temperature=0.2,                # Low temperature = more deterministic / focused output
                                        # High temperature = more random / creative output
    )

    # Remove the prompt so that only the newly generated response remains.
    generated_tokens = outputs[0][inputs.shape[-1]:]
    # Convert token IDs back into readable text.
    response = tokenizer.decode(generated_tokens, skip_special_tokens=True)
    return response


before = generate_reply("unsloth/Qwen2-0.5B-Instruct-bnb-4bit", messages)
after = generate_reply("./qwen2_0_5b_lora", messages)

print("=== BEFORE SFT ===")
print(before)
print()
print("=== AFTER SFT ===")
print(after)

示例输出

训练运行具有以下输出：

$ python train.py
...
Unsloth: LoRA applied — 18,464,768 trainable params (4.04% of 456,701,440 total)
...
Unsloth: Training for 30 steps, BS=2, grad_accum=4, seq_len=2048
Unsloth: Features: CCE, GC, LR=linear, opt=adamw
  Step 1/30 | Loss: 3.9350 | Grad: 4.8440 | LR: 0.00e+00 | Tok/s: 352 | Peak: 2.35 GB
  Step 2/30 | Loss: 4.0082 | Grad: 4.9456 | LR: 4.00e-05 | Tok/s: 388 | Peak: 2.50 GB
...
  Step 30/30 | Loss: 0.0646 | Grad: 0.6915 | LR: 8.00e-06 | Tok/s: 379 | Peak: 2.57 GB

Unsloth: Training complete! Avg loss: 1.2078 | Total time: 35.6s | Steps: 30 | Tokens: 14480
Unsloth: LoRA adapters saved to outputs
Unsloth: Saved final adapters to outputs

输出显示LoRA已成功应用，且仅训练了约4%的模型参数，这使得微调过程保持轻量。

在30个训练步骤中，Unsloth记录了有用的指标，如损失、学习率、每秒处理的令牌数以及峰值内存使用量。损失从约3.9降至0.06，这表明模型正在从小数据集中学习，并且整个运行约在35秒内完成，仅使用约2.6 GB的内存。

最后，Unsloth将训练好的LoRA适配器权重保存到outputs目录以供后续推理使用。你将看到一个名为qwen2_0_5b_lora的新目录，其中包含以下文件：

adapter_config.json
adapters.safetensors
chat_template.jinja
tokenizer.json
tokenizer_config.json

保存的LoRA输出包含几个虽小但重要的文件。adapter_config.json存储LoRA适配器设置，例如哪些层被适配以及适配器如何配置。adapters.safetensors包含实际训练得到的LoRA权重。chat_template.jinja定义系统、用户和助手消息如何格式化为模型期望的提示结构。tokenizer.json存储分词器词表和分词规则，而tokenizer_config.json存储分词器设置，如特殊词元和格式化行为。

这些文件共同足以重新加载微调后的适配器，并使用与训练时相同的提示和分词设置运行推理。

现在运行推理脚本以测试微调后的模型。它将加载保存在qwen2_0_5b_lora目录中的LoRA适配器权重，并在推理期间将其应用于基础模型之上。

用户提示是“我想取消我的订单”。以下输出展示了同一模型在SFT前和SFT后的回答。

$ python inference.py

=== BEFORE SFT ===
Sure, I'd be happy to help you with that! To cancel your order, please follow these steps:

1. **Review Your Order**: Before you can cancel your order, you need to review it. Make sure you have all the necessary information, such as your order number, order date, and any other relevant details.

2. **Review Your Order Details**: Once you've reviewed your order

=== AFTER SFT ===
If your order has not yet shipped, we may be able to cancel it for you. Please share your order number and the reason for cancellation so I can help process the refund or credit.

在SFT之前，基础模型给出了一个泛泛的、有些冗长的回答，听起来很有帮助，但没有遵循清晰的电商支持工作流程。

在SFT之后，模型生成了更简洁、更具操作性的回复，正确地询问订单号，并围绕发货状态来考虑取消问题。这表明，即使使用相对较小的领域特定数据集，SFT也能改善角色对齐和回复风格。