Computer Vision & Multimodal AI
How to fine-tune Vision-Language Models (VLMs) for domain-specific document layout extraction?
Freeze the vision encoder, add a lightweight adapter to the language decoder, and train on OCR‑annotated layout tokens with a small learning rate.
A
Aravind Patel
👑 Tier 3 Elite
Aug 9, 2026 · 2 min read
Fine‑tune a Vision‑Language Model by freezing the vision encoder, adding a lightweight adapter to the language decoder, and training on annotated document images with layout tokens.
**Step‑by‑step workflow**
1. **Dataset preparation**
- Collect ~10k–50k scanned pages.
- Use OCR (e.g., Tesseract 5.3) to extract text and bounding boxes.
- Convert each page to a sequence of ` ` tokens; store in JSONL: `{\"image_path\":\"...\",\"layout\":\" ...\"}`.
2. **Model selection**
- `microsoft/Florence-2-base` or `openai/clip-vit-large-patch14` as vision encoder.
- `google/flan-t5-base` as language decoder.
3. **Adapter insertion**
```python
from peft import LoraConfig, get_peft_model
config = LoraConfig(r=8, lora_alpha=32, target_modules=[\"q_proj\",\"v_proj\"], bias=\"none\")
model = get_peft_model(base_model, config)
```
4. **Training configuration**
```json
{
"per_device_train_batch_size": 8,
"gradient_accumulation_steps": 4,
"learning_rate": 5e-5,
"num_train_epochs": 3,
"fp16": true,
"logging_steps": 50,
"evaluation_strategy": "steps",
"eval_steps": 200
}
```
5. **Loss** – use `CrossEntropyLoss` on tokenized layout; weight `` token with 0.1.
6. **Evaluation** – compute Exact Match (EM) of bbox coordinates (IoU ≥ 0.75) and BLEU for text.
7. **Deployment** – export with `torch.export` and wrap in a FastAPI endpoint.
**Adapter vs LoRA vs Full‑FT**
| Method | Params % | GPU h | Typical EM |
|--------|----------|------|------------|
| Adapter | ~0.5% | 2 h | 78% |
| LoRA | ~0.8% | 3 h | 80% |
| Full‑FT | 100% | 12 h| 83% |
**Gotcha:** OCR mis‑alignments in the training captions propagate to the model, causing systematic layout drift; always validate a clean subset before scaling.
Read the evidence
Sources used in this thread
Open the original material, compare the claims, and form your own view.