
DeepSeek-R1-Distill models offer a compelling balance of reasoning capabilities and efficiency. Fine-tuning these models for specific domains or tasks often presents VRAM and training throughput challenges, particularly with larger variants. This guide details a production-grade methodology for fine-tuning DeepSeek-R1-Distill using Unsloth and LoRA/QLoRA, leveraging custom Triton kernels for significant VRAM reduction and accelerated training. We will cover data preparation, model training, and deployment considerations, including GGUF and vLLM export.
Compare token costs, prompt caching discounts & context margins
Model monthly token economics across Gemini 2.5 Flash, DeepSeek R1/V3, Claude 3.5 Sonnet, and GPT-4o with prompt caching discounts, token counter, and markdown export.
DeepSeek-R1-Distill is a series of open-source reasoning models, distilled from larger, more capable DeepSeek-R1 models. They are designed to excel in complex reasoning tasks, often employing a <think> token to delineate an explicit chain-of-thought (CoT) reasoning process before producing a final answer. This explicit CoT structure is crucial for interpretability and debugging, and its preservation during fine-tuning is critical. The models are typically based on a transformer architecture, utilizing multi-head attention and feed-forward networks.
Unsloth is a library designed to accelerate and optimize LoRA/QLoRA fine-tuning for large language models. It achieves substantial VRAM savings and speedups through several key innovations:
These optimizations make it feasible to fine-tune models like DeepSeek-R1-Distill on consumer-grade GPUs or with larger batch sizes on professional hardware.
<think> CoTThe DeepSeek-R1-Distill models are trained to use a specific <think> token to structure their reasoning. When preparing fine-tuning data, it is critical to maintain this format. Synthetic data generation or careful annotation is often required.
A typical DeepSeek-R1-Distill conversational turn with CoT looks like this:
User: <prompt>
Assistant: <think>Thought process leading to the answer.</think><answer>Final answer.</answer>
For fine-tuning, we need to format our data into conversational turns, ensuring the <think> and <answer> tags are correctly placed.
Consider a dataset of mathematical reasoning problems. Each entry should be structured as a list of dictionaries, representing a conversation.
[
{
"messages": [
{
"role": "user",
"content": "What is the sum of the first 10 prime numbers?"
},
{
"role": "assistant",
"content": "<think>The first 10 prime numbers are 2, 3, 5, 7, 11, 13, 17, 19, 23, 29. Summing them: 2+3+5+7+11+13+17+19+23+29 = 129.</think><answer>129</answer>"
}
]
},
{
"messages": [
{
"role": "user",
"content": "If a car travels at 60 mph for 2 hours, how far does it travel?"
},
{
"role": "assistant",
"content": "<think>Distance = Speed × Time. Speed = 60 mph, Time = 2 hours. Distance = 60 * 2 = 120 miles.</think><answer>120 miles</answer>"
}
]
}
]
This JSON structure is compatible with datasets library and Unsloth's data loaders.
This section provides a complete, runnable example for fine-tuning deepseek-ai/deepseek-r1-3b-base using Unsloth. The principles apply to larger DeepSeek-R1-Distill models as well.
First, install Unsloth and other necessary libraries.
pip install "unsloth[cu121] @ git+https://github.com/unslothai/unsloth.git"
pip install transformers peft accelerate bitsandbytes trl datasets torch
import torch
from unsloth import FastLanguageModel
from trl import SFTTrainer
from transformers import TrainingArguments, AutoTokenizer
from datasets import load_dataset
import os
# 1. Configuration
max_seq_length = 2048 # Max sequence length for DeepSeek-R1-Distill
model_name = "deepseek-ai/deepseek-r1-3b-base" # Or deepseek-ai/deepseek-r1-7b-base
dataset_path = "your_synthetic_reasoning_data.json" # Path to your JSON dataset
# 2. Load Model and Tokenizer with Unsloth
# Unsloth automatically handles 4-bit quantization (QLoRA)
# and loads the model with optimized kernels.
model, tokenizer = FastLanguageModel.from_pretrained(
model_name = model_name,
max_seq_length = max_seq_length,
dtype = None, # None for auto detection (bfloat16 if supported, else float16)
load_in_4bit = True, # Enable QLoRA
)
# 3. Configure LoRA Adapters
# Target all linear layers for optimal performance.
model = FastLanguageModel.get_peft_model(
model,
r = 16, # LoRA rank
target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"],
lora_alpha = 16,
lora_dropout = 0.05,
bias = "none",
use_gradient_checkpointing = "unsloth", # Use Unsloth's optimized gradient checkpointing
random_state = 3407,
max_seq_length = max_seq_length,
)
# 4. Load and Format Dataset
# The dataset should be a JSON file with the structure described above.
# We use `apply_chat_template` to format messages into the model's expected input format.
# DeepSeek models typically use a specific chat template.
# Ensure the tokenizer has a chat template or define one.
# For DeepSeek, it's often similar to:
# {% for message in messages %}{% if message['role'] == 'user' %}{{ '<|User|>
# ' + message['content'] + '<|EOT|>
# ' }}{% elif message['role'] == 'assistant' %}{{ '<|Bot|>
# ' + message['content'] + '<|EOT|>
# ' }}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ '<|Bot|>' }}{% endif %}
# If your tokenizer doesn't have a default, you might need to set it:
# tokenizer.chat_template = "{% for message in messages %}{% if message['role'] == 'user' %}{{ '<|User|>\n' + message['content'] + '<|EOT|>\n' }}{% elif message['role'] == 'assistant' %}{{ '<|Bot|>\n' + message['content'] + '<|EOT|>\n' }}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ '<|Bot|>' }}{% endif %}"
def formatting_prompts_func(examples):
"""
Formats the dataset examples into the model's expected chat template.
Ensures the <think> and <answer> tokens are preserved within the assistant's response.
"""
texts = []
for i in range(len(examples["messages"])):
# Apply chat template to each conversation
# `add_generation_prompt=False` is crucial for fine-tuning to ensure
# the assistant's response is fully included, not just the prompt for generation.
formatted_text = tokenizer.apply_chat_template(
examples["messages"][i],
tokenize=False,
add_generation_prompt=False
)
texts.append(formatted_text)
return { "text" : texts }
dataset = load_dataset("json", data_files=dataset_path, split="train")
dataset = dataset.map(
formatting_prompts_func,
batched = True,
)
# 5. Configure Training Arguments
trainer = SFTTrainer(
model = model,
tokenizer = tokenizer,
train_dataset = dataset,
dataset_text_field = "text", # Field containing the formatted text
max_seq_length = max_seq_length,
dataset_num_proc = os.cpu_count(), # Use all CPU cores for data processing
packing = False, # Set to True for more efficient packing of short sequences
args = TrainingArguments(
per_device_train_batch_size = 2, # Adjust based on VRAM
gradient_accumulation_steps = 4, # Accumulate gradients over 4 steps
warmup_steps = 5,
num_train_epochs = 3,
learning_rate = 2e-4,
fp16 = not torch.cuda.is_bf16_supported(), # Use fp16 if bfloat16 not supported
bf16 = torch.cuda.is_bf16_supported(), # Use bf16 if supported
logging_steps = 1,
optim = "adamw_8bit", # Use 8-bit AdamW optimizer
weight_decay = 0.01,
lr_scheduler_type = "linear",
seed = 3407,
output_dir = "outputs",
report_to = "none", # Disable reporting to W&B etc. for simplicity
),
)
# 6. Train the Model
trainer.train()
# 7. Save the LoRA Adapters
model.save_pretrained("deepseek_r1_lora_adapters")
tokenizer.save_pretrained("deepseek_r1_lora_adapters")
print("Fine-tuning complete. LoRA adapters saved to deepseek_r1_lora_adapters.")
max_seq_length: Crucial for memory. DeepSeek-R1-Distill models are often trained with larger contexts. Adjust based on your data and VRAM.load_in_4bit = True: Activates QLoRA, loading the base model in 4-bit precision. This is the primary VRAM saving mechanism.target_modules: Specifies which linear layers in the transformer architecture will have LoRA adapters applied. Targeting q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj is standard for comprehensive fine-tuning.use_gradient_checkpointing = "unsloth": Leverages Unsloth's optimized gradient checkpointing.formatting_prompts_func: This function is vital. It takes your structured data and converts it into the exact string format the model expects, including the <think> and <answer> tags. The tokenizer.apply_chat_template method is the recommended way to do this.per_device_train_batch_size and gradient_accumulation_steps: Adjust these to fit your GPU's VRAM. A smaller per_device_train_batch_size combined with gradient_accumulation_steps allows for larger effective batch sizes without increasing peak VRAM.optim = "adamw_8bit": Uses an 8-bit AdamW optimizer, further reducing optimizer state memory.After fine-tuning, you'll have LoRA adapters. For deployment, you typically want to merge these adapters back into the base model weights and then convert them to a suitable format.
import torch
from unsloth import FastLanguageModel
from transformers import AutoTokenizer
model_name = "deepseek-ai/deepseek-r1-3b-base"
lora_adapters_path = "deepseek_r1_lora_adapters"
output_merged_path = "deepseek_r1_merged_model"
# Load the base model and tokenizer
model, tokenizer = FastLanguageModel.from_pretrained(
model_name = model_name,
max_seq_length = 2048, # Must match training max_seq_length
dtype = None, # Use the same dtype as during training or float16/bfloat16
load_in_4bit = False, # Load in full precision for merging
)
# Load the LoRA adapters
model = FastLanguageModel.get_peft_model(
model,
r = 16, # Must match training r
target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"],
lora_alpha = 16,
lora_dropout = 0.05,
bias = "none",
use_gradient_checkpointing = False, # Not needed for inference/merging
random_state = 3407,
max_seq_length = 2048, # Must match training max_seq_length
)
model.load_adapter(lora_adapters_path)
# Merge LoRA adapters and save the full model
model.save_pretrained_merged(output_merged_path, tokenizer, save_method = "merged_16bit")
print(f"Merged model saved to {output_merged_path}")
GGUF is a format optimized for CPU inference with llama.cpp and its bindings. It supports quantization (e.g., Q4_K_M, Q5_K_M) for further memory reduction.
import os
from transformers import AutoTokenizer, AutoModelForCausalLM
from huggingface_hub import HfApi, create_repo
from pathlib import Path
merged_model_path = "deepseek_r1_merged_model"
output_gguf_path = "deepseek_r1_merged_model_gguf"
quantization_type = "q4_k_m" # Example: q4_k_m, q5_k_m, q8_0
# Ensure llama.cpp is installed and converted.py is available
# You might need to clone llama.cpp and build it:
# git clone https://github.com/ggerganov/llama.cpp.git
# cd llama.cpp && make
# Load the merged model
model = AutoModelForCausalLM.from_pretrained(
merged_model_path,
torch_dtype=torch.float16, # Or bfloat16
device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(merged_model_path)
# Save in a format compatible with llama.cpp's convert.py
# This typically means saving as a standard Hugging Face model
model.save_pretrained(output_gguf_path)
tokenizer.save_pretrained(output_gguf_path)
# Convert to GGUF using llama.cpp's convert.py
# Make sure you have llama.cpp cloned and built, and its `convert.py` is accessible.
# Adjust `llama_cpp_dir` to your llama.cpp installation path.
llama_cpp_dir = "/path/to/llama.cpp" # IMPORTANT: Set this path
convert_script = os.path.join(llama_cpp_dir, "convert.py")
if not os.path.exists(convert_script):
raise FileNotFoundError(f"llama.cpp convert.py not found at {convert_script}. "
"Please clone and build llama.cpp.")
# Step 1: Convert PyTorch weights to ggml format (intermediate step for convert.py)
# This step is often implicitly handled by newer convert.py versions,
# but historically involved a separate script or specific arguments.
# For DeepSeek, ensure convert.py supports its architecture.
print(f"Converting {output_gguf_path} to GGUF...")
os.system(f"python {convert_script} {output_gguf_path} --outfile {output_gguf_path}/model.gguf")
# Step 2: Quantize the GGUF model
quantize_script = os.path.join(llama_cpp_dir, "quantize")
if not os.path.exists(quantize_script):
# For older llama.cpp versions, quantize might be part of convert.py or a separate script
# For newer versions, `quantize` is a compiled binary.
# If not found, try `python {convert_script} --quantize {quantization_type} ...`
raise FileNotFoundError(f"llama.cpp quantize binary not found at {quantize_script}. "
"Please build llama.cpp or check its documentation for quantization.")
os.system(f"{quantize_script} {output_gguf_path}/model.gguf {output_gguf_path}/model-{quantization_type}.gguf {quantization_type}")
print(f"GGUF model ({quantization_type}) saved to {output_gguf_path}/model-{quantization_type}.gguf")
# Optional: Upload to Hugging Face Hub
# api = HfApi()
# repo_id = "your_hf_username/deepseek-r1-3b-reasoning-gguf"
# create_repo(repo_id, repo_type="model", exist_ok=True)
# api.upload_file(
# path_or_fileobj=f"{output_gguf_path}/model-{quantization_type}.gguf",
# path_in_repo=f"deepseek-r1-3b-reasoning-{quantization_type}.gguf",
# repo_id=repo_id,
# )
# print(f"GGUF model uploaded to Hugging Face Hub: {repo_id}")
vLLM is an optimized inference engine for LLMs, offering continuous batching and PagedAttention for high throughput. It directly uses Hugging Face transformers models.
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
merged_model_path = "deepseek_r1_merged_model"
output_vllm_path = "deepseek_r1_vllm_model"
# Load the merged model and tokenizer
model = AutoModelForCausalLM.from_pretrained(
merged_model_path,
torch_dtype=torch.bfloat16, # Use bfloat16 for vLLM if supported, else float16
device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(merged_model_path)
# Save the model in a format vLLM can directly load
# vLLM expects a standard Hugging Face model directory.
model.save_pretrained(output_vllm_path)
tokenizer.save_pretrained(output_vllm_path)
print(f"vLLM-compatible model saved to {output_vllm_path}")
# Example vLLM inference (requires vLLM to be installed)
# from vllm import LLM, SamplingParams
#
# llm = LLM(model=output_vllm_path, dtype=torch.bfloat16)
#
# prompts = [
# "<|User|>\nWhat is the capital of France?<|EOT|>\n<|Bot|>",
# "<|User|>\nIf x = 5 and y = 3, what is x + y?<|EOT|>\n<|Bot|>"
# ]
#
# sampling_params = SamplingParams(temperature=0.7, top_p=0.95, max_tokens=128)
#
# outputs = llm.generate(prompts, sampling_params)
#
# for output in outputs:
# prompt = output.prompt
# generated_text = output.outputs[0].text
# print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
| Feature | Unsloth LoRA/QLoRA | Full Fine-tuning | GGUF (llama.cpp) | vLLM |
|---|---|---|---|---|
| VRAM Usage (Training) | Very Low (4-bit) | Very High (16/32-bit) | N/A | N/A |
| Training Speed | Fast (Triton kernels) | Standard | N/A | N/A |
| Model Size (Disk) | Adapters only (MBs) | Full model (GBs) | Quantized (GBs) | Full model (GBs) |
| Inference Latency | Moderate (adapter merge) | Low | Low (CPU) | Very Low (GPU) |
| Inference Throughput | Moderate | High | Moderate (CPU) | Very High (GPU) |
| Hardware | Consumer GPUs | High-end GPUs | CPU, Edge devices | High-end GPUs |
| Complexity | Low (Unsloth abstraction) | Moderate | Moderate (conversion) | Low (HF compatible) |
| Use Case | Rapid iteration, resource-constrained training | Max performance, large datasets | Local/offline, CPU-only | High-scale API, GPU-only |
CUDA out of memory error.per_device_train_batch_size.gradient_accumulation_steps to compensate for smaller batch size.max_seq_length.load_in_4bit = True and optim = "adamw_8bit" are set.use_gradient_checkpointing = "unsloth".packing=True, ensure your dataset contains sufficiently long sequences or disable packing if sequences are very short, as packing can sometimes increase peak memory.<think>/<answer> Formatting:
<think> tags, or produces malformed responses.formatting_prompts_func and the raw JSON data. Ensure the <think> and <answer> tags are exactly as the base model expects and are correctly enclosed within the assistant's response in the training data. The tokenizer.apply_chat_template is crucial here.unsloth[cu121]).torch.cuda.is_available() returns True.nvidia-smi. Low utilization might indicate a data loading bottleneck (increase dataset_num_proc).packing=True if your sequences are short, as it can improve GPU utilization.convert.py fails or produces an unusable GGUF file.llama.cpp is cloned and built correctly.convert.py script and quantize binary paths are correct.llama.cpp's GitHub for the latest conversion instructions for DeepSeek models, as support for new architectures evolves.convert.py versions or flags.torch_dtype used for loading in vLLM matches the merged model's dtype (e.g., bfloat16).A data scientist is fine-tuning a DeepSeek-R1-Distill model for a specialized medical diagnostic task. They observe that despite achieving high accuracy on the training set, the model's reasoning process (the content within the `<think>` tags) is often nonsensical or deviates significantly from expert medical reasoning, even when the final `<answer>` is correct. Which of the following is the most likely root cause of this issue in a production fine-tuning scenario?
per_device_train_batch_size (e.g., 1 or 2) and compensate with gradient_accumulation_steps.<think> token after fine-tuning?
The most critical step is to ensure your fine-tuning dataset explicitly includes the <think> and <answer> tokens in the assistant's responses, formatted exactly as the base model expects. The tokenizer.apply_chat_template function, when used correctly with add_generation_prompt=False, will help enforce this. During inference, you should prompt the model with the same chat template and expect it to generate the <think> block.transformers training, which would require significantly more VRAM.max_seq_length?
If your reasoning chains frequently exceed max_seq_length, you have a few options:
max_seq_length if VRAM allows.max_seq_length is the most direct approach.This guide provides a robust framework for fine-tuning DeepSeek-R1-Distill models efficiently. By leveraging Unsloth's optimizations and carefully managing data formatting, you can deploy powerful, specialized reasoning models even with limited hardware resources.
Clean AI CLI logs, build cron expressions, decode JWTs, and calculate chmod permissions offline.

Deep dive into DeepSeek V3 architecture: Multi-Head Latent Attention (MLA) KV compression, DeepSeekMoE routing, and FP8 mixed-precision training.
Read more
Deploying DeepSeek-R1 distilled models locally with vLLM: reasoning chain formatting, token generation latency, and quantization benchmarks.
Read more
How vLLM PagedAttention solves GPU memory waste: virtual memory paging for KV caches, prefix sharing, and chunked prefill latency benchmarks.
Read more