What Should We Ask an AI Automation Company Before We Hire Them?

What should we ask an AI automation company before we hire them?

Ask an AI automation company these ten questions before you hire them, in roughly this order, and write down the answers so you can compare three firms afterwards rather than remembering which demo was shiniest.

Nine of those ten are about operations rather than AI. That is deliberate, and it is the pattern in the failure data: the modelling is rarely the thing that sinks a project.

Which process would you automate first, and what would you leave alone?

A firm that cannot name a process it would leave alone has not thought about your business. Ask both halves of this question in the same breath, because the second half is where you learn whether you are talking to an engineer or a sales deck.

A credible first answer sounds specific and slightly boring. Invoice coding from four regular suppliers. Quote follow up for jobs over a certain value. Shift confirmations the day before. It names what the process costs now, in hours, and explains why that one goes first: high volume, low variation, clear rules, a single system to write into.

A credible second answer names something and gives a reason. Work needing a judgement call nobody in the business can write down. Anything where the cost of a quiet error is larger than the cost of doing it by hand. A process about to change anyway because you are switching software next quarter.

The reason this question sorts firms so well is that the AI part is now the easy part. Reading a document, classifying an email, extracting fields and drafting a reply are solved to a standard that is good enough for ordinary business use. What is not solved is knowing which of your processes is worth the engineering, and that judgement is the whole difference between a system that pays and a pilot that quietly stops.

The scale of the pilot problem is in the Texas data. In the Federal Reserve Bank of Dallas special questions put to 313 Texas executives between May 12 and May 20, 2026, two thirds of responding firms said they use AI, and among those users 26% said the tools were still being tested or piloted while another 52% said a small share of employees used them regularly. A lot of companies have bought AI. Far fewer have changed a process with it. The firm you hire should be able to tell you which one you are about to become.

Who runs the system after launch, and where does it run?

Ask who runs the system after launch and where it runs before you ask about cost, because the answer changes what the cost means. Automations break for dull reasons: a supplier changes an invoice layout, a vendor ships a new API version, a login expires, a form grows a field. None of that is a modelling problem and all of it needs somebody whose job it is to notice.

Three answers are common and they are not equal. The firm operates the system on its own infrastructure, which puts accountability in one place. The firm builds it and your team runs it, which works if you have a team and fails quietly if you do not. The system lives on a no-code platform whose bill moves with your volume, which is workable and means you now own a subscription, a set of credentials and the failure modes that come with them. If the answer is a platform, Zapier vs Make vs n8n covers what you are taking on.

Push for specifics. Where does the thing actually execute. Who is accountable at six in the morning when the overnight job fails. What happens to the work in the meantime, because a queue that silently stops is worse than a system that never existed. How changes get made, and whether a new supplier format is a support ticket or a new project.

BLACKSIG runs what we build, on our infrastructure, and we are explicit that this is the business rather than a preference. A company with its own engineering team that wants the repository and the keys should hire a contractor on that basis, and we will say so on the call rather than after a proposal.

The reason to settle this early is that a build quote and a running arrangement are two different purchases, and firms that blur them are selling you the cheaper looking half. Who maintains an AI automation? covers the mechanics of the second half in detail.

What do we keep if we stop working with you?

Ask what you keep plainly, and expect a plain answer about three separate things: your data, your records and the running system.

Your data and the records the system has written should already be inside your own software, where they were going anyway. An automation that only writes into the vendor's own tool has created a dependency rather than removed work, and that is worth knowing before you sign rather than afterwards.

The running system is the part where answers differ. With us, ending an arrangement does not switch anything off: the system drops to maintenance, which covers keeping it alive rather than developing it further, and is set by how many systems are running and what they cost to operate. Other firms hand over code and infrastructure in full. Both are legitimate and they ask different things of you, because a system handed over is yours and is also yours to keep alive, and what breaks one is almost never the model.

Two questions get you the real answer. What do we keep, and who is accountable for the system continuing to work while nobody is paying for it. A firm that answers the first and dodges the second is describing a handover, not a continuation.

Worth asking about documentation too, though it is a smaller deal than the market pretends. A written description of what the system does, which systems it touches and what the escalation rules are is genuinely useful to a future engineer. A folder of generated technical documents nobody has read is not, and its existence tells you very little about whether the thing will still be running in March.

How will we know it is still working?

You will know it is still working from a report rather than an alert. Automations fail quietly, which is their worst property: the system keeps reporting success while processing nothing, and somebody notices in week five when a customer asks about an invoice that never went out.

Alerting catches hard failures, where something threw an error and stopped. It does not catch the ones that cost money: a supplier whose new invoice layout means every line now reads as zero, a classification rule that stopped matching in July, a queue silently filling behind a login that expired. All of those look like success from the inside.

What catches them is a report of what each system produced, what it escalated and what it skipped, read by somebody who knows roughly what the numbers should be. The skipped count is the one that matters and the one most firms do not show you, because it is the number that says the automation is handling less than it claims.

Ask what the report contains and who reads it. Then ask what the escalation path is when the system is unsure. A system that guesses on a low confidence match and writes it in anyway is building a data cleanup project for you. A system that parks the item and tells a person is doing the right thing, and the count of those parked items is your early warning that something upstream changed.

This is also the question that tells you whether a firm has operated anything. Firms that only build talk about monitoring in the abstract. Firms that run systems talk about specific failures they have watched happen, and what the report showed the week before.

What do you need from us before you can scope this?

A firm that needs nothing from you cannot scope anything. The right answer asks for two things: access to real historical records rather than samples, and time with the person who does the work today.

Real records matter because the variation in them is the project. Twenty invoices from the same supplier teach you nothing about the twelve formats you actually receive. The person who does the work matters because the rules were never written down, and what is documented is usually a sanitised version of what happens.

This is also where the data question gets settled, and the evidence says most companies are worse prepared than they assume. Cloudera and Harvard Business Review Analytic Services, in a report published on March 5, 2026 based on a survey of more than 230 people involved in AI data decisions and fielded in October 2025, found only 7% said their organisation's data was completely ready for AI, 27% said it was not very ready or not at all, and 73% said their organisation should prioritise AI data quality more than it does. A separate Cloudera survey of 1,574 IT leaders at companies with more than 1,000 staff, run by Researchscape in July 2025, found 9% saying all their data was available and usable for AI. Two different studies with two different samples, routinely quoted as one, and both pointing the same way.

What that means for your scoping conversation is narrow and useful: ask the firm what it will do when the data turns out to be messier than the demo assumed. Repricing the project halfway through is one answer. Building the cleanup into the first stage is another. Discovering it in month three is the one you are trying to avoid. Is our data good enough for AI? sets out how to check before anybody quotes you.

What has broken in production, and what happened next?

Everybody with real systems running has a story about what has broken in production, and it is almost never about the AI. Ask directly and listen for whether the story has a specific fix in it.

The pattern worth hearing is mundane and it goes something like: a supplier changed a form, the extraction started returning the wrong field, the report showed the skipped count climbing, somebody fixed the mapping, the backlog was reprocessed. That story demonstrates four things at once: the system was monitored, the failure was visible, somebody owned it, and nothing was lost permanently.

An unblemished record means a short history or a selective memory. A firm that tells you its systems do not break is telling you it has not run one for long.

The research on autonomous systems says the same thing from a different direction. TheAgentCompany benchmark, built at Carnegie Mellon with Duke and independent researchers and with Graham Neubig among its leads, set 175 tasks inside a simulated software company across software development, project management, data science, administration, HR and finance. The first version of the paper put the best baseline at about 24% of those tasks completed autonomously; the revision of September 2025 reports Gemini 2.5 Pro at 30.3%. The authors' own framing is that simpler tasks are often solved and longer horizon work is not. Any firm promising you an agent that runs a function end to end is promising something the benchmarks do not show yet.

Which is why the question is about failure handling rather than failure rates. The systems worth buying are the ones built on the assumption that something will break, with the escalation path and the report designed in from the start.

Which answers should end the conversation?

Four answers should end it, and none of them is about price.

The first is a proposal with no process named in it. A quote that describes capabilities, a technology stack and a number, without naming which of your processes it is for and what that process costs today, is a quote for something nobody has scoped. According to a Gartner forecast published on June 25, 2025, more than 40% of agentic AI projects will be cancelled by the end of 2027, and the firm has been blunt that a lot of what is being sold as agentic is not, a practice it calls agent washing.

The second is a refusal to say who does the work. The senior person who sells the project is often not the person who delivers it, and in engagements that go wrong that gap is usually where the trouble started. Ask to meet the builder before you sign.

The third is a timeline with no conditions on it. Anybody who has integrated into a real ledger knows the honest answer involves what they find when they get access. A firm promising a fixed date before seeing your systems is managing your feelings rather than your project.

The fourth is a firm that will not tell you what it is wrong for. Every firm in this market is wrong for somebody. A firm that cannot name who, cannot be trusted to tell you when you are the one.

Our own version: we are wrong for you if you want to own and run the code yourself, if the job is a data warehouse migration or a governance programme rather than an automation, or if what you need is software with screens and users. Those are real jobs and other firms do them better. Where we fit is a repetitive process eating real hours in a company whose tools are already in place, and the Universidad Maimonides case study is what that looks like finished: fifty to a hundred data files per batch becoming records built automatically, about fifteen hours a week back, running on our infrastructure ever since.

Frequently asked questions

Related resources

• Should you hire an automation consultant or build it in-house?
• Why do AI automation projects fail?
• What is an AI audit?
• How much does AI automation cost?
• Who maintains an AI automation?
• Is our data good enough for AI?