How to evaluate an AI provider (before you sign)

For the past year, I have met companies that already talked to three or four AI providers and cannot tell them apart. The demos look alike, so do the promises. The problem is not finding someone who 'does AI'. It is finding someone who leaves you in a stronger position in two years, not more dependent. Here is the grid I use, as a buyer, to sort providers before signing.

Why choosing has become so hard

In 2026, 'doing AI' means almost nothing. The vendor added an 'assistant' button, the agency hired an AI profile, the freelancer took a course last month. Everyone can produce a demo that works on a clean case. What the demo never shows is what breaks in production, who pays the bill when volume grows, and who owns the code once the provider is gone. That is why my method starts with your data and your reversibility, not with the model: the order in which a provider raises these topics already tells you a lot about what is coming.

The four questions that sort providers fast

• Data: where does it go, who processes it, in which country is it stored? If they talk about the model before your data, the priorities are already backwards.
• Reversibility: if I part ways with you in a year, what do I keep and what do I lose? The answer must include the code, the prompts, the data and the connectors.
• Measurement: how will we know it works, with numbers, and what happens when the system gets it wrong? A serious provider talks about evaluations before talking about going live.
• Security: what rights does the AI get on your systems, and which actions require human approval? 'It's secure' is not an answer.

The red flags

• Numbers without proof: 'up to 10x productivity', 'AI cuts your costs in half', without ever saying on what, measured how, at equal quality or not.
• One model imposed: the provider has a single vendor in mind and warns you off every other one. Often because it is the only one they know how to wire up, or because they resell it.
• No evaluation: nobody can say how we will judge that the result is good, or what happens the day it is wrong.
• No clear answer on data location: you hear 'secure cloud' with no host, no country, no access details.
• The all-in monthly fee: build and run bundled into one price, so you cannot tell what you actually pay for or what you own.

A provider who talks about the model before your data has the priorities backwards. A model can be swapped. A data leak cannot.

The good signs

• They talk about data before the model: what stays with you, what leaves for a third party, and why.
• They plan the exit from day one: code delivered and documented, prompts and connectors yours, dependencies listed. You can take over or switch without starting from scratch.
• They quote honestly: a measurable target, not a slogan. '30 to 70% off an audited LLM bill, at equal quality' is a claim you can check; 'we will transform your productivity' is not.
• They build on open standards: wiring AI into your tools through a standard like MCP (adopted by Anthropic, OpenAI, Google and Microsoft, handed to the Linux Foundation in late 2025) rather than a proprietary hookup that locks you in.
• They show real work, not just slides: cases they have run, with what worked and what did not.

SaaS vendor, agency, freelance: who for what

• The SaaS vendor: you buy a finished product for a common need (support, note-taking, transcription). Fast and cheap upfront. In return, you adapt to the tool, your data lives with them, and you share their roadmap with every other customer.
• The agency: a team for a broad project, with several trades. Useful when the work is ambitious and long. In return, structure costs, AI profiles sometimes junior behind the salesperson, and team rotation that has you re-explaining the context.
• The freelance or independent: one senior point of contact who scopes, builds and hands over. Useful when you want expertise without an extra layer and want to stay in control of what is delivered. In return, one person: a scope to define and availability to check.

I am in the third box, so take this with that caveat. What I see work: an independent to scope, decide and lay the first foundation, then your teams take over. I go into that split in what an independent AI consultant actually does. What matters is not the label, it is that the person who decides is the one with their hands in the work.

How to read a proposal

• Build and run separated: how much the setup costs, once, and how much running it costs, every month. A single fee that blends the two usually hides a dependency.
• Ownership of code and prompts: who owns what gets produced? If the answer is not 'you', you are renting, not building.
• Dependencies: which third-party services it relies on, and what happens if one raises prices or shuts down? Simple test: ask what you keep the day you stop. A crisp answer means you are building an asset; a vague one means you are renting a service you do not control.

The POC is also a test of the provider

A POC (a proof of concept) validates a use case on your real data, in a few weeks, before you invest. But it tests the provider as much as the feasibility. Watch how they frame it: a numbered success criterion set upfront, or an open POC that never ends? A build billed separately, or a subscription in disguise they slot you straight into? I work with a short, scoped proof of concept billed on its own, yours whatever happens, ending in a clear go or no-go decision. How a provider reacts to that request already tells you a lot.

The copy-paste question grid

• Where does my data go, who processes it, in which country is it stored?
• If I stop in a year, what exactly do I keep: code, prompts, data, connectors?
• How do we measure that it works, and what happens when the system gets it wrong?
• What rights does the AI get on my tools, and which actions require human approval?
• What share of the price is build (once) and what share is run (every month)?
• Which third-party services do I become dependent on, and what happens if one raises its prices?
• Can you show me comparable real work, including what went wrong?

If a provider answers these seven questions cleanly, you have someone serious, whatever their status. If the answers stay vague, that is not a detail to fix after signing: it is the signal to stop. And if you want an outside opinion before you commit, let's talk: I will tell you what I would do in your place, even if it is not with me.