Gemma Fine-Tuning Guide: LoRA, QLoRA & Deployment

When to Fine-Tune (and When Not To)

Fine-tuning makes sense when you need the model to consistently produce a specific output format, adopt domain vocabulary, or perform a narrow task that prompting alone cannot reliably achieve. The most common wins include enforcing JSON schema compliance, adapting tone for customer support, and teaching industry-specific classification.

However, fine-tuning is the wrong tool for several common scenarios. If the task requires knowledge that changes frequently, retrieval-augmented generation (RAG) is a better fit because you can update the knowledge base without retraining. If you have fewer than 50 quality examples, few-shot prompting will outperform a fine-tune that overfits on sparse data. And if the base model already handles your query well with the right prompt, the engineering cost of maintaining a fine-tuned model is not justified.

LoRA vs QLoRA Explained

Both LoRA (Low-Rank Adaptation) and QLoRA (Quantized Low-Rank Adaptation) avoid updating the full model weights. Instead, they freeze the base model and inject small trainable matrices into the attention layers. The difference is how they handle the frozen weights.

LoRA keeps the base model in its original precision (typically FP16 or BF16). It trains adapter matrices that represent only 0.1% to 1% of the total parameter count. The adapters themselves are small and fast to train, but the full-precision base model still occupies significant VRAM.

QLoRA adds 4-bit NormalFloat quantization to the base weights before attaching the same LoRA adapters. This cuts VRAM usage by roughly 60% compared to standard LoRA while introducing less than 1% quality degradation on most benchmarks. The adapters still train in 16-bit, so gradient computation remains stable.

For the vast majority of use cases, QLoRA is the right choice. The quality difference is negligible for task-specific fine-tuning, and the hardware savings are substantial. This guide defaults to QLoRA in all code examples. If you have access to multi-GPU servers and need every fraction of a percent of quality, switch to standard LoRA by removing the quantization config.

Environment Setup with Unsloth

Unsloth is the most popular fine-tuning framework for Gemma in 2026. It delivers roughly 2x faster training and 70% less memory usage through custom CUDA kernels, and it natively supports all current Gemma model sizes. The setup is a single pip install.

# Create a virtual environment (recommended)
python -m venv gemma-ft
source gemma-ft/bin/activate    # Linux/macOS
gemma-ft\Scripts\activate       # Windows

# Install Unsloth (pulls transformers, peft, trl, etc.)
pip install unsloth

# Log in to Hugging Face for gated model access
huggingface-cli login

Unsloth patches the standard Hugging Face training pipeline at the CUDA kernel level. You write standard transformers code and Unsloth automatically intercepts the compute-heavy operations. No API changes are needed beyond the initial model load call.

If you prefer the standard Hugging Face stack without Unsloth, install transformers, peft, trl, bitsandbytes, and datasets individually. The code examples in this guide will work with minor import changes.

Step 2: Attach LoRA Adapters

LoRA adapters inject trainable low-rank matrices into the model's linear layers. The key parameters are rank (how many dimensions the adapter matrices have), alpha (a scaling factor, typically 2x rank), and target modules (which layers get adapters).

model = FastModel.get_peft_model(
    model,
    r=16,                     # LoRA rank: 16-64
    lora_alpha=32,             # Alpha = 2x rank
    lora_dropout=0.05,
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
    bias="none",
    use_gradient_checkpointing="unsloth",
)

Targeting all linear layers (attention projections plus MLP gates) gives the adapter maximum flexibility to learn your task. If VRAM is tight, you can drop gate_proj, up_proj, and down_proj to reduce memory at the cost of some adaptation quality.

Gradient checkpointing ("unsloth" mode) trades a small amount of compute time for significant VRAM savings by recomputing activations during the backward pass instead of storing them. Keep this enabled unless you have VRAM to spare.

Step 4: Train

Training uses the standard Hugging Face SFTTrainer from the TRL library. Unsloth intercepts the underlying operations to apply its CUDA kernel optimizations. The configuration below works for most single-GPU QLoRA runs.

from trl import SFTTrainer
from transformers import TrainingArguments

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset["train"],
    args=TrainingArguments(
        output_dir="./output",
        per_device_train_batch_size=4,
        gradient_accumulation_steps=4,
        num_train_epochs=3,
        learning_rate=2e-4,
        warmup_ratio=0.05,
        weight_decay=0.01,
        lr_scheduler_type="cosine",
        logging_steps=10,
        save_strategy="epoch",
        fp16=True,
        optim="adamw_8bit",
        seed=42,
    ),
    max_seq_length=2048,
)

trainer.train()

The adamw_8bit optimizer reduces the memory overhead of optimizer states. Combined with gradient accumulation (effective batch size = 4 x 4 = 16), this configuration maximizes throughput on a single GPU.

Monitor the training loss. It should decrease steadily for the first epoch and then flatten. If the loss drops to near zero, you are likely overfitting. Reduce epochs or increase the dataset size. If it plateaus above 1.0 and never decreases, increase the learning rate or check your data formatting.

Hyperparameter Tuning Guide

The default values in the training code above work well for most cases. When you need to squeeze out more quality or troubleshoot training instability, these are the knobs to turn.

FAQ

Can I fine-tune Gemma on Apple Silicon (M1/M2/M3)?

Unsloth requires NVIDIA CUDA, so it does not run on Apple Silicon directly. However, you can use the standard Hugging Face stack with MPS (Metal Performance Shaders) for small models like E2B. Training will be significantly slower than CUDA. For anything larger than E4B, use a cloud GPU (Colab, RunPod, Lambda) instead.

How long does fine-tuning take?

Highly variable. A typical QLoRA run with 1,000 examples on Gemma E4B using an RTX 4090 finishes in 15 to 30 minutes with Unsloth. The same run on an RTX 3070 takes roughly 45 to 90 minutes. Larger models and datasets scale linearly.

Can I combine fine-tuning with RAG?

Yes, and this is often the ideal production setup. Fine-tune for consistent formatting, tone, and task behavior. Use RAG to inject up-to-date facts at inference time. The fine-tuned model learns how to use retrieved context effectively, while RAG prevents hallucination on factual queries.

Do I need to fine-tune again when a new Gemma version releases?

LoRA adapters are tied to the specific model architecture they were trained on. If the new version changes the layer dimensions or architecture, you need to retrain. If it is the same architecture with updated base weights, you can sometimes reuse the adapter, but retraining typically yields better results since the new base weights shift the loss landscape.