Distilling an LLM Diagram-Curation Pipeline into Local Classifiers

A frozen SigLIP2 encoder with a small logistic-regression head beats the vision-LLM that labeled its own training data. It's more accurate, runs about 200× faster, and costs nothing on a laptop.

0
gold accuracy (winner)
0
LLM teacher accuracy
0
per crop on a laptop GPU
0
labeled diagrams released
Read the paper (PDF) Code on GitHub Download dataset (37 MB)

By Adnan Abbasi, founder of Thothica · July 2026

We used a vision-LLM to label 2,000 diagrams, then asked whether a local model could take over the classification. It can, and it beats its teacher. A frozen SigLIP2 + logistic regression hits 86.0% against hand-checked truth [CI 81.5–90.5], against the LLM's 75.5% (McNemar p=0.0005), at about 52 ms per crop instead of 4 to 17 seconds. Zero-shot beats the LLM too. The surprise: the fancier models lost. A fine-tune (71.5%) and MLP heads (78%) memorized the teacher's 25% label noise; the simplest model couldn't. Check a distilled model against the LLM's own labels and you'll pick the wrong one.

The result: the student beats its teacher

Nine approaches, scored on 200 hand-checked crops. The simplest learned model wins.

Gold accuracy leaderboard
Figure 1. Gold accuracy with 95% confidence intervals. The dashed line is the LLM teacher.

The trap: silver validation picks the wrong model

Flip between what the LLM's labels say (silver) and what hand-checking says (gold). The order changes. The MLP looks best on silver, then falls apart on gold because it learned the teacher's mistakes.

Speed: ~200× faster than the LLM, on a laptop

Single-image latency, measured on an Apple-Silicon MacBook. The winner sits in the top-left corner: accurate and fast. The LLM gate is two orders of magnitude slower, and less accurate.

Accuracy vs speed
Figure 2. Accuracy vs. latency (log scale). Free local inference, fully offline.

How the data was made

A resumable pipeline turns arXiv PDFs into labeled diagrams for $20.36. It journals every decision, so the training set for the classifier falls out as a byproduct. The LLM that built it becomes the teacher we later beat.

Data pipeline flow
Figure 3. The extraction pipeline: 1,183 papers → 12,266 judged pages → 2,000 accepted diagrams.

Does removing the label noise help? No.

Cleaning out the noisiest training labels doesn't move the winner. It just removes data the model was already ignoring. It does rescue the model that overfit, but not past the simple baseline. To get past 86% you'd need more signal, not less noise.

Label cleaning ablation
Figure 4. Confident-learning label cleaning, evaluated on gold.

Reproduce it

It's all open source, and it runs on a laptop. No GPU cluster, no Colab.

git clone https://github.com/adoistic/stem-diagrams
cd stem-diagrams/arxiv_diagram_pipeline/ml
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

python assemble_dataset.py        # frozen paper-level splits
python extract_embeddings.py --backbone siglip2
python probe_experiments.py       # train the probes (seconds)
python evaluate.py --gold probe_siglip2_logreg   # headline vs gold

The paper has the full method: how the gold set was labeled, every model on the ladder, and the call on what to put in production.

×