Field manual · Nº 01 · Evals for local AI
I make AI agents prove what they say.
I test AI that runs on your own machine: does it really call the tool, cite a source that exists, know that a rule has changed? Then I write down where it fails. I trained as a pharmacy technician, where checking the source is the job.
local-ai-evals · open source · MIT
Small, honest tests for AI on a laptop
Five tests, each answering one question with measured results and its limits written down. All of it runs locally on a 16 GB MacBook Air: the model tests through Ollama, the rest with PyTorch.
Does it really call the tool? toolbench
qwen3.5:9b scored 72/72. With a context window that was too small, Ollama cut the prompt without an error and the model invented field names: 63/72. qwen2.5:14b twice said “I’ve added it to your list” and called nothing. That answer gets a verdict of its own: FAKED.
Cite it, or say “I don’t know” rag
Every answer needs a quote that can be checked. It refused 4/4 questions it had no data for and answered 7/10 correctly. The two answers that passed every check and were still wrong are documented, because that is where the guardrail ends.
Is this health claim still true? healthclaims
30 claims checked against Cochrane, the EMA and EU law, asked neutrally and under pressure. Results in section 03.
A GPT trained from scratch tinygpt
nanoGPT’s 10.7M-parameter model, trained for 142 minutes on the laptop’s GPU. Best validation loss 1.47 at step 2,000, then it overfit. The longest stretch it copied word for word from its training text did not grow as it overfit (24 characters, then 23).
Native or Spanish accent? pronunciation
Scores German pronunciation sound by sound with wav2vec2: native synthetic voices 95–97 %, Spanish-accented ones 47–50 %. It has not been validated with real speakers yet, and the README says so.
Personal system · in daily use since September 2026
Three AIs that check each other’s work
I run my own assistant on my laptop: three AI agents with different jobs, one shared memory and scheduled routines. In its first 13 days it grew to 32 projects; today it runs 22 scheduled routines. What makes it trustworthy is not the models but the rules between them.
The writer Claude Code
Writes the code and the plans in the terminal, and turns every failure into a test or a written rule.
The reviewer Codex
Reviews in read-only mode, checking the files on disk, before anything ships. In one code audit: 21 findings, 0 false alarms.
The operator Hermes Agent
Runs the scheduled routines and answers on WhatsApp and Slack, with local models through Ollama where they are good enough.
One shared memory notes + vault
All three read the same memory and notes, so a lesson learned once is not learned three times.
The brake human in the loop
Emails stay as drafts, replies to reviews wait for my approval, and nothing is published or paid without my yes.
healthclaims · new piece of local-ai-evals
From the pharmacy counter to an eval
I worked three years in community pharmacies. At a counter, a wrong answer reaches a real person, so I wrote a method for checking health claims: product category first, then the primary source, then “is this still allowed today?”. Then I turned it into a test for local models. The best one I tested still calls zinc pyrithione shampoo an authorised EU cosmetic, banned since 2022. The same model with its refusals removed says codeine syrup is a reasonable option for a 10-year-old.
The claim 30 cases
Things a pharmacy counter really hears: an anti-dandruff shampoo, cough syrup for a 10-year-old, glucosamine for joints, vitamin D against fractures.
The evidence primary sources
Every verdict comes from a source I opened on 16 October 2026: Cochrane, the EMA, EU regulations, WHO, NICE, the FDA. No source, no claim.
Still true today? outdated
Seven claims used to be true and changed: a shampoo ingredient banned in 2022, a heartburn drug suspended in 2020, advice on peanuts for babies that was reversed. A model trained on old text fails here quietly.
Under pressure sycophancy
Each claim is asked twice: on its own, and from a customer who insists it is true. A model that agrees only under pressure is telling people what they want to hear.
The number that matters endorsed
How often the model called a claim “supported” when it is false, unproven or out of date. At a counter, that is the mistake that reaches a patient.
| model | right verdict | right under pressure | endorsed | endorsed under pressure | named “outdated” |
|---|---|---|---|---|---|
| qwen3.5:9b | 28/30 | 27/30 | 2 | 2 | 3/7 |
| qwen2.5:14b | 25/30 | 25/30 | 5 | 5 | 0/7 |
| qwen3.5:9b, refusals removed | 17/30 | 13/30 | 9 | 12 | 4/7 |
About
No CS degree. A habit instead.
I trained as a pharmacy technician and spent three years at the counter in Barcelona. There, you check the source before you speak, because the answer goes home with someone. I brought that habit to AI.
Since 2026 I build with AI agents every day: local models on a 16 GB laptop, cloud models where they are worth it, and one rule. Every failure becomes a test or a written rule, so it does not happen twice.
I am looking for work in AI evaluation, agent reliability or AI operations: remote, or on site if the role is right.
- Based inBarcelona · remote
- LanguagesSpanish, Catalan (native) · English (B1) · German (A2, course in progress)
- TrainingTécnico en Farmacia, 2-year vocational degree (2023)
- ExperienceCommunity pharmacy, 2023–2026
- ToolsPython · Ollama · PyTorch · n8n · Cloudflare
- Works withClaude Code · Codex · Hermes Agent