LLM Post-Training · Evaluation
MailTune
LLM Post-Training & Evaluation System
Testing whether a small open model can learn a writing style without losing facts.
Illustrative style example
Before fine-tuning
Dear Sarah, I hope this message finds you well.
I wanted to reach out regarding the project
timeline we discussed last week, and to see...
QLoRA fine-tuned
Hi Sarah, quick update on the timeline,
here's where things stand...
73%
wins vs prompted
44% → 6%
fabrication rate
0.28%
parameters trained
Training
A QLoRA fine-tuning project built to test whether a small open LLM can learn concise, casual email style while preserving facts, using controlled baselines and an evaluation suite rather than a single before/after example.
Prompting alone can push a base model toward a style, but the project asks a sharper question: does low-rank fine-tuning move a small open model further than a well-engineered prompt, and does it do so without increasing fabrication?
Before / after style example
Illustrative style example, not real client content.
Before fine-tuning
Dear Sarah, I hope this message finds you well.
I wanted to reach out regarding the project timeline
we discussed during our last meeting...
QLoRA fine-tuned
Hi Sarah, quick update on the timeline,
here's where things stand.
Training pipeline
264 emails
injected facts
QLoRA fine-tune
0.28% params
3 baselines
raw, prompted, QLoRA
FastAPI + MCP
Claude Desktop
Training configuration
- · Qwen2.5-1.5B-Instruct
- · 4-bit QLoRA
- · 4,358,144 trainable parameters
- · 0.28% of total parameters
- · 264 synthetic training emails
Baselines
Raw-base and prompted-base baselines were established before attributing improvements to fine-tuning.
Pairwise evaluation
Result
100%
win rate vs raw base model
Result
73%
win rate vs prompted baseline
120 Claude-judged comparisons on verified-unseen topics.
Fabrication
Factual reliability was evaluated using prompts with embedded ground-truth facts. Fabrication measures whether the model invents unsupported facts; fact preservation measures whether it retains the facts it was actually given. A model can score well on one without scoring well on the other, so the two are reported separately rather than as one combined chart.
Fabrication rate
| Base | Prompted | QLoRA | |
|---|---|---|---|
| Fabrication rate | 44% | 17% | 6% |
Fact preservation
Eval fact
85%
target facts preserved
In the factual-reliability test, the QLoRA model preserved 85% of the target facts while reducing fabrication to 6%.
Serving
The fine-tuned model is served through FastAPI and exposed over MCP, with end-to-end Claude Desktop integration for drafting emails directly from a chat interface.
Inference latency
Result
7.91s → 2.91s
p50 inference latency
Measured p50 inference latency in the evaluated setup.
Stack
Limitations
- · The judge/human agreement sanity check has not yet been run.
- · The dataset is small by research standards.
- · No memorization or canary-string test has been run yet.