I wrote to a former PhD mentor this week to thank him for something he taught me ten years ago. I studied quality in my PhD: how to measure it, and how easy it is to measure the wrong thing. Like much of what we learn in school, I didn’t fully appreciate it at the time. I was treading water, just trying to keep my head above the surface while managing a teaching career, clinical practice, and family. But I use it now, almost every day, for work neither of us could have pictured back then.
Somewhere in the last few years I figured out that I’m not a researcher. I did some studies, I wrote some papers, I published some articles. All the while I kept wanting to write computer programs so I could build things. I’ve wanted to write code since my dad brought home a Commodore 64 a hundred years ago. But I never had the brains or the patience to learn it. What changed is AI. It lets me code in English: I describe what I want and the AI writes the Python (or SQL, or JavaScript, or R, or whatever). And I don’t even have to fully understand it. (In fact, here’s a little secret: when I say “I learned Python,” that’s not entirely accurate. I just learned that a “.py” at the end of a filename means it’s a Python file.) That’s how everything I’ve built in the past year got built.
Building this way creates a problem I didn’t expect. A new model ships every few weeks. Anthropic, OpenAI, DeepSeek, Google, plus the open models I can run on the Mac(s) in my office. Each one arrives with a screenshot of how it stacks up against the others and a wave of TikTok videos announcing it’s the best thing ever made. And every time, the question is the same: do I switch? Switching isn’t free. Every prompt, every procedure, every skill, and every agent that runs overnight was tuned on the model it runs on now. Change the model and some of that breaks in ways I won’t notice for weeks.
The benchmarks are useless for that decision. Not because the numbers are fake, but because the companies optimize for them. They “game” the system. Shivalika Singh and a dozen coauthors published a paper last year called The Leaderboard Illusion. They found that Meta tested 27 private versions of Llama 4 on Chatbot Arena in the run-up to its release, and only the winner got announced. They also estimated that Google and OpenAI together have received about 40 percent of all the data the arena collects, which lets them tune for exactly what the arena rewards. Then Mubashara Akhtar and colleagues looked at 60 language model benchmarks for a study at ICML this year and found nearly half of them saturated. The top models have run out of room to be told apart.
A benchmark is a lab value, not an outcome. An LDL of 55 tells you something. It doesn’t tell you whether the patient had a heart attack. A model that scores 92 on a coding test tells you something too. It doesn’t tell you whether it’ll write my accounting script correctly or keep the heart failure guideline straight when it drafts a practice question.
So I built a tool that evaluates models against my own outcomes. It took a few hours, written in English, while my wife and I drove to beach wedding in New Jersey. It puts any new model through the jobs I need done and reports which one gives the most for what it costs. Five parts.
My own cases. The tests are the work. A writing brief drawn from a recent article topic. A coding task built from a real incident I had to fix. A set of accounting statements with invented merchants. A batch of clinical questions sampled from my own guideline-grounded pool. Fresh every run, so nobody has trained on them. No student data, no customer data, nothing real that shouldn’t be there. It’s a simulation. Nursing professors love those.
A head-to-head, not a history. The candidate takes the same tests, in the same sitting, at the same effort levels, as the model I’m using now and as the candidate’s own predecessor. Nothing is compared to a result from last month. Models drift. Prompts drift. I drift.
The incumbent runs twice. If it disagrees with itself by more than half the switch threshold, the comparison is too noisy to call, and the tool says so instead of picking a winner. The control group checks itself.
A blind read. For writing, the scores don’t count until I’ve read the drafts without knowing which model wrote which and ranked them myself. If I don’t prefer the new model in two of three sets, no switch, whatever the judge said.
A bar set before I look. A switch has to deliver at least 20 percent more value than what I’m using, with quality no worse, and it has to hold at both effort levels. Anything between 10 and 20 percent goes on a watch list. The number matters less than the timing. You pick it before you see the result so the result can’t pick it for you.
Cost-effectiveness analysis is the tool you use to figure these things out. I’ve noticed that cost-effectiveness is a poorly understood term outside of economics. Most people confuse cost-effective with cheap. But cost-effectiveness isn’t about cost. It’s not about quality either. It’s about value, a term that burrowed into my head from health economics. This is the part my mentor would recognize. Value is quality ÷ cost. It’s what you get for what you pay, compared against what you’d otherwise use.
I keep my Apple computers in service for ten years or more because they’re that good (high quality, high cost). And I’m loath to spend $50 per million tokens on an AI model that won’t write better code than the nearly free open-weight one (high quality, low cost). Both are parts of the same cost-effectiveness analysis.
There’s more to it than fits here, so the next two pieces take it apart: first what value is and isn’t, with a drug that costs a penny and one that costs $600 a month Then where the switch threshold comes from and why you set it before you see the data.
Is this overkill for one guy and a Mac? Probably. But I’d rather overbuild the decision than rebuild everything downstream of a bad one.
My mentor taught me that quality is what you build into the structure and the process, and the outcome follows. I didn’t do much with that as a researcher. I do it every day now. The next model that wants to write a practice question for NPReady will take the same tests as the one doing it today, and it gets the job only if it clears the number I set before I looked.