Evaluating AI tools before you depend on them

Every AI tool I have ever regretted adopting passed its demo. That fact is worth sitting with, because it means the demo — the launch video, the landing-page GIF, the five-minute trial where everything sparkles — carries almost no information about whether a tool deserves a place in your workflow. The information you need lives elsewhere: in how the tool behaves on your files, in how it fails when it fails, in what happens to the price after you are invested, and in how much of your work you can carry out the door. None of that appears in a demo, and nearly all of it is checkable in about ninety minutes — before dependence sets in rather than after.

The stakes are higher with AI tools than with ordinary software for a blunt reason: AI tools are probabilistic, so a demo shows you a sample, not a specification. A spreadsheet either computes the sum or it does not, and one demonstration settles the matter. A background remover, a transcription model, a summarizer — these perform differently on every input, which means a vendor can truthfully show their best outputs forever without technically lying about anything. The gap between showcase performance and performance on your material is the single most expensive thing to discover after you have committed.

We sit on both sides of this transaction — we ship AI tools ourselves and we evaluate other people’s constantly — so what follows is the protocol I actually run, sharpened by the times I skipped it and paid. Five checks: run the candidate on a test set built from your own work, probe its failure modes deliberately, establish where the processing physically happens, price the tool at its worst month instead of its first, and rehearse the exit before you move in. Skip any one of them and you have not evaluated the tool. You have watched an advertisement with extra steps.

Demo assets are selected, and selection is the entire trick. The photos in a background-removal demo have crisp subjects, clean edges, and studio lighting. The audio in the transcription demo is one clear speaker in a quiet room. The document in the summarizer demo is short, well-structured, and mainstream. None of this is fraud — it is marketing doing what marketing does — but it produces a systematic bias: every capability you are shown sits at the friendly end of its difficulty curve, and your actual work does not live there.

Demo-driven decisions fail on a delay, which is what makes them expensive. The tool sails through its first week because your early usage unconsciously imitates the demo — you feed it easy cases while you learn the interface, and dependence quietly forms during exactly this honeymoon. Then a real deadline arrives carrying a real, messy input — the podcast with two crosstalking guests, the product shot with flyaway hair, the forty-page contract full of tables — and you discover the tool’s true envelope at the moment you can least afford the lesson.

The corrective is not cynicism; it is procedure. Treat every demo as an existence proof — under some conditions, this tool produced this output — and nothing more. The questions that should decide adoption are the ones a demo structurally cannot answer: what happens under your conditions, at your volumes, on your ugliest inputs, at next year’s price. We keep a sober snapshot of what on-device models genuinely deliver in What local AI can actually do in mid-2026, and even that should not substitute for the only test that settles anything, which is the one you run yourself.

Order matters here: assemble your evaluation set before you touch the product, because once an onboarding flow has walked you through its curated happy path, you will unconsciously test the things it taught you to test. Fifteen to twenty-five real items pulled from your own recent work is enough — not synthetic samples, not whatever happens to be on the desktop, but files from the last three months of actual output, because those encode the specific mess your work reliably contains.

A good set spans three tiers. The bulk is typical work: the inputs the tool will see every week, where quality needs to be dependably good rather than occasionally spectacular. A handful are known-hard: the cases that broke your previous tool, the edge conditions your niche generates — for us that means hair against busy backgrounds, transient-heavy audio, files at awkward dimensions. And two or three are trick entries with a known correct answer, including at least one where the correct behavior is to decline: a corrupt file, an input outside the tool’s claimed scope, a question whose answer is not in the document. You are grading judgment as much as output.

Then score like you mean it: the same set for every candidate, results recorded in a grid — even three columns of good, acceptable, and unusable beats an impression formed by poking around. Two things happen when I do this that never happen when I skip it. Tools that felt impressive slide to the middle of the table once their inconsistency becomes visible in rows, and unglamorous tools win on the only metric that counts, which is performance on the work I actually do. The set also compounds: it is reusable for every future candidate in the category, turning the next evaluation from an afternoon into an hour.

Every tool fails; the evaluation question is how. The failure you can live with is loud, early, and legible — the tool refuses the input, says why, and leaves your file untouched. The failure that costs you is quiet and plausible: the transcription that invents a sentence during crosstalk, the cutout that shaves a sliver off a product’s edge, the summary that states something the document never says. Quiet failures ship to clients. A serious evaluation therefore spends less time admiring good outputs and more time hunting for the confident wrong one.

Hunting means deliberately leaving the envelope. Feed the oversized file, the wrong format, the empty input, the audio in a language the model does not claim to speak. Overload it: the image at 8,000 pixels, the three-hour recording, the document that is mostly tables. What you are watching for is not whether it breaks — it will — but the texture of the breakage: does it degrade gracefully or fabricate silently, does it error with an explanation or a shrug, does a crash lose your work or preserve it? Ten minutes of hostile inputs reveals more about engineering quality than an hour of happy-path use, because failure handling is where a vendor either spent effort or did not.

With generative and analytical tools, add the falsifiable probe: ask questions you already know the answers to, on documents you know deeply, and count fabrications instead of admiring fluency. A tool that responds with “I cannot find that in this file” has earned trust; a tool that answers everything has earned suspicion. My standing rule after years of this: output confidence tells you nothing about output correctness, so a tool that expresses uncertainty honestly is displaying its most hireable trait. Vendors know users dislike refusals, which is exactly why a principled refusal is such a strong signal.

“Where does my file go” sounds like a privacy question and is really a dependence question, because the answer determines what you are adopting. A tool that runs on your device — as ours do — is close to owning software: the capability sits in your hardware, keeps working offline, and cannot be metered per use or degraded remotely without you noticing. A tool that runs on a server is a standing relationship with a company: capability, latency, price, and policy can all change underneath you, and the file changes custody every single time you use it.

Marketing blurs this line enthusiastically — “private,” “secure,” and even “local” appear on products that upload everything — so verify instead of reading. The developer-tools network panel settles it in two minutes: process a file and watch what leaves the machine. Airplane mode is a cruder instrument anyone can operate: if the tool still works with the connection gone, the model genuinely lives on your device. We published the full walkthrough in Is my data safe with browser AI tools? A plain-English audit, and the deeper map of which workloads can even run locally in What runs on your device vs in the cloud — some tasks genuinely need a server, and honest vendors say which ones.

Neither architecture is a scandal; undisclosed architecture is. A cloud tool that states plainly what it sends, why, and what it retains can absolutely earn a place in your stack — several tools we rely on are exactly that. What the location check protects you from is the mismatch: sensitive files flowing through a product you believed was local, or a workflow built on “free and unlimited” compute that was always someone else’s GPU bill waiting to be passed along. Which brings us to the bill.

The number on the pricing page is the least stable fact about an AI tool, because most AI pricing is still an experiment being run on you. Inference costs real money, venture-subsidized free tiers exist to buy usage data and market share, and the corrections arrive on a schedule you do not control: quotas shrink, “unlimited” acquires an asterisk, the feature you adopted migrates to a higher tier, per-seat quietly becomes per-use. None of this requires a villain. It only requires you to price the dependency at its plausible worst instead of its promotional best.

So evaluate the pricing model, not the price. Metered pricing means your bill is a function of your busiest month — model the spike, never the average. Per-seat is stable right up until you grow. “Free while in beta” is a countdown, not a tier. And a free tool with no visible business model is not free; the payment is happening somewhere you cannot see, a trade we broke down in The privacy cost of free AI tools. Two questions before adopting anything: what does month thirteen cost at twice my current usage, and what is my move on the day this triples in price? If the second answer is “I would be stuck,” the evaluation is already over.

Repricing risk also has tells you can read in advance. A vendor that publishes limits in plain numbers — file-size caps, quotas, model downloads stated in megabytes, the way we tier ours in Honest AI tiers: Lite, Standard, Pro — sized in gigabytes, not hype — is signaling that today’s deal was designed to survive contact with reality. A vendor whose pricing page is all “contact us” and whose free tier is suspiciously bottomless is signaling the opposite: the real price has not been decided yet, and your usage is helping them discover it. Neither tell is proof. Both deserve weight.

Export is the feature nobody evaluates and everybody eventually needs, because the average AI tool you adopt this year will not exist in its current form, at its current price, in three years. This category is young companies in a consolidating market: they get acquired, pivot, reprice, or fold, and every one of those events poses the same question — can you leave with your work? The time to learn the answer is before your work lives there, while the answer still costs nothing. I run the rehearsal even on tools I expect to keep forever — partly because expectations about young software are worth little, and partly because the export button is where a vendor’s respect for your work is either designed in or visibly absent.

Rehearsing is literal: during the trial, actually perform the export. Not read the documentation about it — do it. Pull a project out and inspect what arrives. Open formats you can use elsewhere, or a proprietary bundle only this tool can read? Do layers, edits, and transcripts survive, or does the export flatten months of structure into a JPEG and a text dump? Is bulk export possible, or does each item leave one at a time through the interface — tolerable at ten projects, a hostage situation at four hundred? A tool that exports open formats completely can be adopted lightly. A tool that traps structure has to clear a far higher bar before it holds anything that matters.

Lock-in also hides where the export button cannot reach, so inventory the switching cost honestly: the prompts and presets you tuned, the workflow muscle memory, the integrations wired into the rest of your stack, the collaborators trained on the interface. None of that exports. The practical hedge is keeping sources of truth outside the tool wherever possible — originals in your own storage, outputs mirrored out on a schedule — so the tool remains a processor of your work rather than the residence of it. Depending on a tool is fine; that is what tools are for. Depending with no exit is what this protocol exists to prevent.

Laid end to end, the protocol is smaller than it reads. Ten minutes assembling the test set from recent work. Thirty running it through the candidate and filling the scoring grid. Fifteen on hostile probes — oversized, malformed, out-of-scope inputs — watching the texture of the failures. Five with the network panel or airplane mode to establish where processing happens. Ten reading the pricing model against your worst plausible month. Fifteen performing an actual export and inspecting what comes out. Call it ninety minutes, most of it mechanical, none of it requiring expertise beyond knowing your own work.

Scale the ceremony to the stakes. A toy you might use twice deserves five minutes and no spreadsheet. The transcription tool your podcast depends on, the editor your client work flows through, the assistant wired into your daily writing — anything whose failure or repricing would genuinely hurt — deserves the full pass, and one reusable test set makes each subsequent candidate dramatically cheaper to judge. The asymmetry is the whole argument: the evaluation costs one afternoon, once; a bad dependency collects payment at the worst possible moments, repeatedly, with interest. Ninety minutes is what that interest rate looks like when you choose to pay it up front.

The protocol also has a side effect I did not anticipate when I started running it: it recalibrates what impresses you. After enough evaluations, launch-video sparkle stops registering, and a different profile stands out instead — the tool that publishes its limits, fails loudly, states its sizes in megabytes, runs where it claims to run, and lets you leave. Those traits cluster because they come from the same place: a vendor who expects to be evaluated. Build with that expectation, choose with it, and both ends of the tool economy get a little more honest.