Fine-Tuning a Llama 3 Model for Automated Clinical Documentation and EHR Integration
Why fine-tune instead of prompting a general model
Base instruction-tuned Llama 3 models can draft clinical notes with prompting alone, but they drift on institution-specific formatting (your health system's exact SOAP template, abbreviation conventions, section ordering) and on domain vocabulary density. Fine-tuning on a corpus of (transcript, note) pairs teaches the model your house style with far fewer output-formatting errors than prompting achieves, without the cost of full-parameter training. This tutorial uses QLoRA (4-bit quantized base weights + trainable low-rank adapters) on Llama 3 8B, which fits on a single 24GB GPU, and finishes with a FHIR DocumentReference integration so the model's output actually lands in the EHR.
All training data referenced here must be de-identified per HIPAA Safe Harbor (§164.514(b)) or run inside a BAA-covered compute environment before any real patient data touches it.
1. Technical architecture
Training path (offline)
┌───────────────────┐ ┌────────────────────┐ ┌───────────────────────┐
│ De-identified note │──▶│ Dataset builder │──▶│ QLoRA fine-tuning │
│ pairs (JSONL) │ │ (prompt/response fmt) │ │ job (single A10G/A100) │
└───────────────────┘ └────────────────────┘ └───────────┬───────────┘
▼
┌──────────────────────┐
│ Merged adapter weights │
│ pushed to model registry│
└──────────┬───────────┘
Inference path (online) │
┌───────────────────┐ ┌────────────────────┐ ┌────────────▼───────────┐
│ Clinician dictation │──▶│ vLLM inference server │◀──│ Load fine-tuned adapter │
│ or transcript │ │ (OpenAI-compatible API)│ └────────────────────────┘
└───────────────────┘ └─────────┬──────────┘
▼
┌────────────────────┐
│ Structured note │
│ parser (JSON schema) │
└─────────┬──────────┘
▼
┌────────────────────┐
│ FHIR client → EHR │
│ DocumentReference │
│ (draft status) │
└────────────────────┘
2. Complete code implementation
2.1 Dataset preparation
# data/build_dataset.py
import json
from pathlib import Path
from datasets import Dataset
SYSTEM_PROMPT = (
"You are a clinical scribe. Convert the encounter transcript into a SOAP note "
"using this exact section order: Subjective, Objective, Assessment, Plan. "
"Use only information stated in the transcript."
)
def build_prompt(transcript: str) -> str:
return (
f"<|start_header_id|>system<|end_header_id|>\n{SYSTEM_PROMPT}<|eot_id|>"
f"<|start_header_id|>user<|end_header_id|>\n{transcript}<|eot_id|>"
f"<|start_header_id|>assistant<|end_header_id|>\n"
)
def load_pairs(jsonl_path: str) -> Dataset:
"""jsonl_path rows look like: {"transcript": "...", "note": "..."}"""
rows = []
with open(jsonl_path, "r", encoding="utf-8") as fh:
for line in fh:
if not line.strip():
continue
record = json.loads(line)
if not record.get("transcript") or not record.get("note"):
continue # skip incomplete rows rather than crash the whole build
rows.append({
"text": build_prompt(record["transcript"]) + record["note"] + "<|eot_id|>"
})
if not rows:
raise ValueError(f"No valid transcript/note pairs found in {jsonl_path}")
return Dataset.from_list(rows)
if __name__ == "__main__":
ds = load_pairs("data/train_pairs.jsonl")
ds.save_to_disk("data/train_dataset")
print(f"Built dataset with {len(ds)} examples")
2.2 QLoRA fine-tuning script
# train/finetune_qlora.py
import torch
from datasets import load_from_disk
from transformers import (
AutoModelForCausalLM,
AutoTokenizer,
BitsAndBytesConfig,
TrainingArguments,
)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer
BASE_MODEL = "meta-llama/Meta-Llama-3-8B-Instruct"
OUTPUT_DIR = "checkpoints/llama3-clinical-scribe"
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True,
)
tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
tokenizer.pad_token = tokenizer.eos_token
model = AutoModelForCausalLM.from_pretrained(
BASE_MODEL,
quantization_config=bnb_config,
device_map="auto",
)
model = prepare_model_for_kbit_training(model)
lora_config = LoraConfig(
r=16,
lora_alpha=32,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters() # sanity check: expect ~0.5-1% of total params
train_dataset = load_from_disk("data/train_dataset")
training_args = TrainingArguments(
output_dir=OUTPUT_DIR,
per_device_train_batch_size=2,
gradient_accumulation_steps=8, # effective batch size 16
num_train_epochs=3,
learning_rate=2e-4,
bf16=True,
logging_steps=10,
save_strategy="epoch",
save_total_limit=2,
warmup_ratio=0.03,
lr_scheduler_type="cosine",
report_to="none",
)
trainer = SFTTrainer(
model=model,
args=training_args,
train_dataset=train_dataset,
dataset_text_field="text",
max_seq_length=4096,
)
if __name__ == "__main__":
trainer.train()
trainer.model.save_pretrained(f"{OUTPUT_DIR}/final_adapter")
tokenizer.save_pretrained(f"{OUTPUT_DIR}/final_adapter")
print("Training complete. Adapter saved to", f"{OUTPUT_DIR}/final_adapter")
2.3 Merge adapter and serve with vLLM
# train/merge_adapter.py
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
BASE_MODEL = "meta-llama/Meta-Llama-3-8B-Instruct"
ADAPTER_DIR = "checkpoints/llama3-clinical-scribe/final_adapter"
MERGED_DIR = "checkpoints/llama3-clinical-scribe/merged"
base = AutoModelForCausalLM.from_pretrained(BASE_MODEL, torch_dtype="bfloat16")
model = PeftModel.from_pretrained(base, ADAPTER_DIR)
model = model.merge_and_unload() # bakes LoRA deltas into the base weights
model.save_pretrained(MERGED_DIR, safe_serialization=True)
tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
tokenizer.save_pretrained(MERGED_DIR)
print("Merged model written to", MERGED_DIR)
# Serve the merged model with an OpenAI-compatible API
pip install vllm --break-system-packages
python -m vllm.entrypoints.openai.api_server \
--model checkpoints/llama3-clinical-scribe/merged \
--served-model-name clinical-scribe-llama3 \
--port 8000 \
--max-model-len 4096 \
--gpu-memory-utilization 0.90
2.4 Structured parsing and FHIR integration
# integration/fhir_push.py
import json
import base64
import requests
from requests.exceptions import Timeout, HTTPError
from openai import OpenAI # vLLM's OpenAI-compatible client
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
def generate_note(transcript: str) -> str:
resp = client.chat.completions.create(
model="clinical-scribe-llama3",
messages=[
{"role": "system", "content": "You are a clinical scribe producing a SOAP note."},
{"role": "user", "content": transcript},
],
temperature=0.2,
max_tokens=800,
)
return resp.choices[0].message.content
def push_to_ehr(fhir_base_url: str, access_token: str, patient_id: str, encounter_id: str,
note_text: str, timeout_seconds: int = 10) -> dict:
"""Creates a draft FHIR DocumentReference. Status is deliberately 'preliminary'
so it requires clinician sign-off inside the EHR before becoming final."""
encoded_note = base64.b64encode(note_text.encode("utf-8")).decode("ascii")
resource = {
"resourceType": "DocumentReference",
"status": "preliminary",
"type": {"coding": [{"system": "http://loinc.org", "code": "11506-3", "display": "Progress note"}]},
"subject": {"reference": f"Patient/{patient_id}"},
"context": {"encounter": [{"reference": f"Encounter/{encounter_id}"}]},
"content": [{
"attachment": {"contentType": "text/plain", "data": encoded_note}
}],
}
try:
resp = requests.post(
f"{fhir_base_url}/DocumentReference",
headers={
"Authorization": f"Bearer {access_token}",
"Content-Type": "application/fhir+json",
},
data=json.dumps(resource),
timeout=timeout_seconds,
)
resp.raise_for_status()
except Timeout:
raise RuntimeError("FHIR server timed out — encounter note NOT saved. Retry or queue for later.")
except HTTPError as exc:
status = exc.response.status_code
if status == 401:
raise RuntimeError("EHR access token expired — refresh OAuth2 token and retry.") from exc
if status == 422:
raise RuntimeError(f"FHIR server rejected resource as invalid: {exc.response.text}") from exc
raise
return resp.json()
3. Step-by-step configuration guide
- Provision GPU compute. A single NVIDIA A10G (24GB) or better handles QLoRA fine-tuning of the 8B model; use an A100 40GB+ if you plan longer context windows or larger batch sizes.
- Install dependencies:
pip install torch transformers peft trl bitsandbytes accelerate datasets --break-system-packages huggingface-cli login # required to pull the gated Llama 3 weights - Prepare training data as
data/train_pairs.jsonl, one{"transcript": ..., "note": ...}object per line, then runpython data/build_dataset.py. - Run training:
python train/finetune_qlora.py— monitorlogging_stepsoutput; loss should trend down steadily over the 3 epochs. - Merge and export:
python train/merge_adapter.py. - Start the inference server: run the
vllm.entrypoints.openai.api_servercommand from §2.3. - Configure EHR OAuth2 credentials. Register a SMART on FHIR backend service app with your EHR vendor (Epic, Cerner/Oracle Health, etc.), store the client ID/private key in a secrets manager, and implement the JWT-bearer token exchange to obtain
access_tokenbefore callingpush_to_ehr. - Smoke test: call
generate_note()with a sample transcript, inspect the output, then callpush_to_ehr()against your EHR sandbox environment — never point this at a production EHR until the note quality has been clinically validated.
4. Error handling and edge cases
- Gated model download failures — Llama 3 weights require an accepted license on Hugging Face; a 403 on
from_pretrainedmeans thehuggingface-cli logintoken doesn't have accepted access. Fail with a clear message rather than retrying blindly. - CUDA out-of-memory during training — reduce
per_device_train_batch_sizeto 1 and raisegradient_accumulation_stepsproportionally to keep the effective batch size constant; also confirmbnb_4bit_use_double_quant=Trueis set, which meaningfully reduces memory footprint. - Incomplete or malformed training rows are filtered in
load_pairsrather than crashing the whole build — a single bad JSON line in a 10,000-row file shouldn't halt training. - vLLM server cold-start latency — the first request after server startup pays a CUDA graph capture cost; issue a warmup request in your deployment health check before routing real traffic.
- FHIR token expiration (401) is caught explicitly in
push_to_ehrand raises a distinct error so the caller can trigger a token refresh flow instead of retrying with the same expired token. - FHIR validation errors (422) usually mean a missing required field for your specific EHR's
DocumentReferenceprofile (some vendors requiredocStatusor a specifictypecoding) — surface the server's OperationOutcome text directly rather than a generic failure, since it names the exact field. - Network timeouts to the EHR — do not silently drop the note on timeout; queue it (e.g., in SQS or a local outbox table) for retry so a transient network blip never loses a clinical note.
- Hallucinated content in fine-tuned output — fine-tuning reduces formatting drift but does not eliminate hallucination risk. Keep every
DocumentReferenceatstatus: preliminaryand require explicit clinician sign-off in the EHR before it becomes part of the legal record.
