Fine-Tuned Open-Source Models

QLoRA Basics

QLoRA (Quantized Low-Rank Adaptation) makes fine-tuning large models accessible by combining 4-bit quantization with low-rank adapters.

pip install bitsandbytes peft trl datasets

Loading a Quantized Model

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model

quant_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-chat-hf",
    quantization_config=quant_config
)

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05
)

model = get_peft_model(model, lora_config)

Dataset Preparation

Optimize token length, format as chat templates, and split train/validation.

def format_example(example):
    return {
        "text": f"<|user|>\n{example['question']}\n<|assistant|>\n{example['answer']}"
    }

dataset = dataset.map(format_example)

Hyperparameter Configuration

Use Weights & Biases (W&B) for experiment tracking and HuggingFace TRL for the trainer.

from trl import SFTTrainer

trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    args=TrainingArguments(
        output_dir="./lora-llama",
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        learning_rate=2e-4,
        logging_steps=10,
        save_steps=100,
        report_to="wandb"  # Track with W&B
    )
)

Monitoring Training

Watch for loss convergence, learning rate scheduling, and overfitting via validation loss.

Inference & Evaluation

Compare fine-tuned model outputs against the base model and frontier models on test prompts.