Template · Vendor evaluation

Questions to ask any AI delivery vendor, including us

12 questions, the answer we would give, and what a bad answer sounds like. Forward it to whoever is running the shortlist, and ask us the same ones you ask everybody else.

How to use it

Every answer below is one we have given in a real conversation. Publishing them costs us the option of dressing them up later, which is the point. If a vendor cannot answer 9 of these 12 in plain language on a first call, that is the finding.

Who does the work

1. Who exactly will be on this, by name?

The people who scope the work are often not the people who do it.

Ours. The engineer who reads your systems has done the job. Our recruiters are engineers, so the technical screen is run by someone who could pass it, and you interview the people you get.

A bad answer. We will assign the right resource from the bench once the contract is signed.

2. Who screened them, and could that person do the job?

A screen run by a recruiter who has never shipped code tests for keywords.

Ours. Criminal background check, employment verification for every role on the resume, an interview with a Perform engineer, and reference checks, all completed before you see a name.

A bad answer. All our engineers are pre-vetted to the highest standard.

3. If the person leaves in month three, who owns the ramp?

This is the question that separates a staffing transaction from a delivery commitment.

Ours. We do. The same engineers stay on your systems, and the re-ramp is on us.

A bad answer. We would put forward a replacement candidate right away.

4. What will you tell me if I do not need you?

A vendor who has never talked themselves out of work has never given you an unfiltered read.

Ours. We say so. Our assessments sometimes conclude exactly that, and we turn down work we can see failing by month three.

A bad answer. We can definitely help with that.

How the work gets reviewed

5. Who signs off that the output is correct, and are they independent of the build?

A team grading its own AI work produces a sign-off that no downstream reviewer accepts.

Ours. A human is accountable for every output, and every output has a name attached to it. The independence is from the build. We do not stand outside your team.

A bad answer. Our QA process is fully automated end to end.

6. What happens when this reaches legal or compliance review?

Pilots rarely die on model quality. They die at the gate, where somebody asks for an audit trail and a name on the sign-off.

Ours. We build for the reviewer from the first week, because that gate is where the work either ships or stops.

A bad answer. That is usually handled on the client side.

7. Will this write to my system of record, and what happens the first time it does?

An agent that only reads is a demo. The risk starts at the first write.

Ours. The integration is the work. We scope what the agent is allowed to write to, and what happens when it gets it wrong, before anything goes live.

A bad answer. It integrates with everything through the API.

8. Who watches it after launch, and what does that cost per month?

Monitoring, drift and token cost are the parts nobody budgets for.

Ours. We price running it as its own phase, with the drift checks and the token cost named, because the number changes after launch.

A bad answer. Once it is live, it is live.

What you are buying

9. What does the first engagement produce, and can I stop there?

A first engagement you cannot exit is a retainer wearing a project name.

Ours. A read. A senior engineer looks at your systems for 2 to 4 weeks and it ends with a prioritized path and one first step. We sell the next step only from a result on the current one.

A bad answer. We recommend starting with a 12 month programme.

10. What are you not going to do?

A vendor with no exclusions has no specialism.

Ours. We do not take aerospace and defense, government contracting, or enterprise banking. We say out loud which parts of a scope we would decline.

A bad answer. There is nothing we cannot take on.

11. Whose standards does the work meet, yours or mine?

Work delivered to a vendor framework arrives as a deliverable and then waits for your team to absorb it.

Ours. Yours. The work goes through your review process and your definition of done.

A bad answer. We deliver against our own quality framework.

12. What proof can I check without asking you?

A reference call is curated. A published number with a source is not.

Ours. Two named customer stories with the numbers and the quotes, a client relationship that has run 18 years, and every figure on this site carrying the organization, the sample size and the date.

A bad answer. A percentage with nothing attached to it.

When this list will not help you

Anyone running a price-only comparison. These questions test whether a vendor can be held to something. If the decision comes down to the lowest hourly rate, the answers do not change it.

Start with a conversation

Ask us these 12 first.