You don't need to understand how the model works. You need to know whether the thing they're selling will still be doing its job in eight months, and whether you'll be able to leave if it doesn't.
Those are both answerable without any technical knowledge. Here's how.
The demo is not evidence
Start here, because it's the trap.
Every AI demo works. It works because it was built to work, on data chosen because it works, run by a person who knows exactly which questions not to ask. That's not dishonest, it's just what a demo is. But it tells you almost nothing about your business.
So the first move is to take control of the demo. Bring your own material. Real ones, messy ones, from last week.
- Bring three actual voicemails, including the one where the guy mumbles.
- Bring five real inbound emails, including the confusing one.
- Bring the weird edge case that breaks your current process.
Say: run these. Right now, in this meeting.
Watch what happens. A good vendor says sure, and you learn a lot in ten minutes. A vendor who says "we'd need to configure that first" is telling you the demo only works on rails, which is exactly what you needed to know.
The nine questions
1. What happens when it's wrong?
Not if. When. Every AI system produces bad output sometimes. The right answer describes a mechanism: a confidence threshold, a human review step, a fallback, an alert. The wrong answer is "our accuracy is very high." That's a deflection, and it means they haven't thought about failure, which means you'll discover the failure mode yourself, in production, on a customer.
2. How will I know it stopped working?
Systems break silently. The integration expires, the credentials rotate, a vendor changes an API. If the answer isn't a specific alert going to a specific person, then the answer is "you'll find out in three months when you notice revenue is down."
3. Can I get my data out?
All of it, in a normal format, whenever I want, without asking permission. If there's any hesitation here, walk. Your customer list and your history are the only assets that compound. A vendor who makes them hard to extract is planning to charge you for that later.
4. What does it connect to, and how?
If the answer is "we have an API," that's not an answer, that's an invitation for you to hire someone. Ask what it integrates with out of the box, specifically, by name. Then ask what happens to that integration when the other vendor changes something.
5. Who else uses this in a business like mine?
Not a logo wall of enterprises. A ten-person service business in a similar trade. Ask to talk to one. If none exist, you're the beta, and beta customers should get beta pricing and say so out loud.
6. What's the real setup time, in my hours, not yours?
Vendors quote their implementation time. What you care about is how many hours you and your team have to spend, and when. This number is routinely understated by a factor of three. Ask specifically: how many hours of my staff's time, in week one, week two, and month two.
7. What happens to my pricing if my volume doubles?
Get the pricing curve, not the pricing. Some AI products are cheap at low volume and vicious at scale, which is a nasty surprise when you grow. Ask for the number at 2x and 5x your current volume.
8. If I cancel, what breaks?
The best question in the list, and the one nobody asks. If cancelling means your phone stops working or your customer records vanish, you're not a customer, you're a hostage. Know the exit before you take the entrance.
9. What is this actually doing under the hood?
You're not asking for architecture. You're testing whether they can explain their own product in plain English. If they can't say something like "it reads the transcript, pulls out these five fields, and creates a record," in one sentence, one of two things is true: they don't understand it, or there's less there than they're implying. Both are your problem.
Answers that should end the meeting
"It learns your business over time." Ask exactly what that means, mechanically. Sometimes it's real. Very often it means nothing at all, and the phrase is there because it sounds like a moat.
"You won't need staff for that anymore." Nobody who's actually shipped this stuff says that. The honest pitch is "your staff will spend less time on this and more on the thing you want them doing." A vendor promising headcount elimination is either lying or hasn't seen what happens in month three.
"We can't share a customer reference." For a business your size, this means there isn't one.
"The AI handles all of it end to end." The good ones tell you where the humans go. That's a sign of maturity, not weakness.
Refusing to run your data in the demo. Full stop. That's the whole test.
The pilot, done right
If they clear the questions, don't sign a year. Run a real pilot with three rules.
Rule one: a number, chosen before you start. Response time, leads captured, hours saved, reviews collected. Write down today's value. Sign nothing until you both agree what number you're moving.
Rule two: a defined end date. Thirty or sixty days. On that day, you look at the number and make a decision. Pilots without end dates become subscriptions by inertia, which is the business model.
Rule three: an exit that costs nothing. No data trapped, no process rebuilt around it, no phone number ported anywhere you can't port back.
If a vendor won't do a scoped pilot with a measurable outcome, that tells you what they think the outcome will be.
The uncomfortable one
Ask yourself, honestly, before you evaluate anybody: do I know what problem I'm solving, and do I know what it currently costs me?
If you can't answer that, no vendor evaluation will save you, because you'll be persuaded by whoever is most persuasive. That's how businesses end up with nine tools and no results.
Know your number first. Then the vendor conversation is easy, because you're not shopping. You're checking whether this specific thing moves that specific number, and most of them won't, and you'll be able to tell in one meeting.
If you want someone in the room who's built these and has nothing to sell you but an opinion, that's the service. Forward me the pitch deck and I'll tell you what's real. Get in touch.