A full AI audit report, on a company that does not exist

The distributor's leadership put it this way: "Everyone talks about AI: where does it actually help here, and where do we start?" As it stands the question has no answer, because it is about a technology and not about a job. So I rewrote it as an auditable question, one that four days of work can answer: of the uses your teams raise on their own, which hold up against the data that exists today, at what cost in effort, and in what order should you take them?

Over those four days I ran 11 interviews, from the managing director to the warehouse operator, and 3 observations at the order desk. An interview tells you what people believe they do; an observation shows the moves they no longer think to mention: the re-keying, the back and forth between two screens, the checks that sit in no procedure. I also went through 2 ERP exports covering 12 months and sampled 400 documents at random.

The fourth day adds no new material; it sizes the effort and puts the five tracks in order. The readout happens in front of the people I interviewed, and that is the only filter that counts: if a sentence does not hold up in front of the person who does that work every day, it goes. The wider frame is in my method.

What this audit does not say

• This is not a security audit: I look at where the data sits, not at whether it is properly protected.
• This is not a review of the ERP code: I take the system as it runs, without judging how it is written.
• This is not a user test: three observations show moves, they do not measure adoption.
• A sample of 400 documents gives orders of magnitude, never rates: read the percentages in this report as markers for discussion.

What I describe here comes from 11 interviews, 3 observation sessions at the order desk, 2 ERP exports covering 12 months and 400 documents drawn at random. That is not enough to know everything. It is enough to see where the work is lost. I describe how things actually run, not how the procedures say they run.

The path an order takes

An order arrives by email. Someone opens the message, then the attachment: a PDF, a spreadsheet, sometimes a few lines in the body of the message. They read it, retype the part numbers one by one into the ERP, check the price, send a confirmation back to the customer. That path fills a large share of the day for the 14 people on the order desk.

Of about 4,800 orders a month, 61% arrive by email, 27% through the web portal and 12% by phone. That split decides what follows: the 27% from the portal already arrive structured and are never retyped, and the 12% by phone are not a job for a language model. The whole subject sits inside the 61%, and nowhere else.

The technical documentation

The distributor keeps 34,000 supplier PDF datasheets. From one supplier to the next, nothing matches: the structure, the vocabulary, the units. About 40% of these datasheets are scans, page images with no text inside.

The consequence is concrete. A keyword search cannot see those 40%. A salesperson looking for a diameter or a standard in a scanned datasheet finds nothing, while the information sits right there on screen. So they open them one by one. This is the point the interviews raise most often.

The state of the data

Two facts weigh more than the rest. The ERP history carries a 7 month hole in 2022, at the time of the migration: the movements from that period were never carried over. That hole rules out one precise thing: statistical sales forecasting. A seasonality calculation needs complete years to compare a month with itself, and 2022 is no longer a complete year.

The PIM, the repository that describes the products, is partial and filled in by hand. It rules out something else: serving as a source of truth. No assistant can state a price, a stock level or a specification on that basis, because nothing says whether the record was ever updated. I have written elsewhere about what makes a dataset usable: Is your data ready for AI?.

The 2025 POC

A customer chatbot was ordered from an agency in 2025, then stopped after 4 months. I hold nothing against that agency: the mistake is a common one, and it is structural. Three causes stack up, and none of them comes down to the quality of the work delivered.

• No source of truth was exposed for stock and prices: the chatbot answered with nothing to check against.
• No evaluation set existed, so nobody could say whether it answered correctly, or see any progress from one version to the next.
• No escalation path to a human was planned, so a stuck customer stayed stuck.

This POC did not fail because of the model. It failed because it was asked to answer on data it had never been given, with no way of knowing when it was wrong.

These three gaps can be closed, and they get closed outside the chatbot. That is why, further on, I propose starting with something else.

The order below is not a merit ranking, it is an execution order. C1 makes more noise inside the company: order entry occupies the order desk every day, and it is the first subject raised in the interviews. C2 still goes first, for a simple reason. It is the only one of the five cases whose quality can be checked before it reaches anyone's hands, and on ground where a mistake never reaches a customer. Starting there means learning to measure on a scope with no consequences, before applying the same discipline to work that touches real orders.

C2: searching the technical documentation

This pain is named in 9 of the 11 interviews, unprompted. A technician looks for a dimension, a tightening torque, an equivalence between two part numbers. They open three datasheets, they call a colleague, they end up calling the supplier. Nobody can say how long it takes. Everybody can say it happens several times a day.

The assistant does one thing and one thing only: it finds the useful passages in the 34,000 supplier datasheets, and it answers by citing the datasheet and the page. It writes no sales pitch, it guesses no price, it does not replace the supplier. When it finds nothing, it says so. That is the pattern described in the document assistant.

Three reasons make it the first project, and the verdict is yes, start here. The scope is bounded: one corpus, one question, one answer. The answer is checkable because it is sourced: whoever reads it opens the cited datasheet and settles it in ten seconds. And the evaluation set, meaning the list of questions whose right answer is known, can be built right now. Support has already received those questions. They sit in the tickets and the mailboxes, they only need gathering.

What would make it fail is known in advance. About 40% of the datasheets are scans, images with no usable text. If the image reading step is rushed, the assistant will give wrong answers on that share of the corpus, and never flag that it does not know. Second risk, a scope that widens along the way to take in prices, stock and commercial terms, three subjects whose source of truth sits elsewhere. One point still needs checking before launch: what the supplier contracts allow on internal indexing.

C1: entering orders received by email

The verdict is yes, but not as planned. The initial request was an AI that reads every order arriving by email. The ERP export says otherwise: 62% of order lines come from 23 regular customers. Those 23 customers can send a structured file, a format the ERP reads on its own. That leaves 38% of the lines, about 900 documents a month, in formats no integration will ever cover: scans, improvised spreadsheets, orders typed into the body of the message. That, and only that, is where AI has a role.

On the integrable share, a plain file integration beats AI on the three criteria that matter. It costs less to set up and to run. It is never wrong, because it infers nothing: either the file conforms, or it is rejected. And it consumes nothing in use, not one token. Putting a language model where a file format is enough means paying for a read on every document, for work that a mapping between fields settles once and for all.

The measurement rule is set before the first trial, not after. The model's success rate is not what gets measured: that number means nothing to the order desk, which counts orders, not fields. What gets measured is the share of orders integrated with no human rework, on a sample reviewed line by line. That single number decides whether to continue, adjust or stop. The document reading chain is covered in detail in this guide.

So the order is: file integration for the 23 customers first, AI extraction second, on the tail alone. Reversing the two means having a machine read documents that should never have been documents.

Part of the work in an AI project is not AI, and it is often the part that produces the gain.

Three of the five tracks do not start now. Two are deferred, one is ruled out in its current form. None of them is held back because the subject is uninteresting, but because a condition is missing, and that condition can be named.

C3: Tender response support

Verdict: later, and that is not a refusal. Answering a tender means producing a document where every technical claim commits the distributor. A language model is very good at writing a plausible paragraph. Without a reliable retrieval base behind it, it produces unsourced prose, exactly the opposite of what a tender demands. The risk is not lost time, it is stating a false product characteristic and being held to it.

What unblocks it is precise: C2 running in production, with sourced answers and an evaluation set kept up to date. The day a writer asks for a characteristic and gets back the supplier sheet that carries it, writing support becomes a thin layer on a sound base. Before that day, C3 is not a project, it is a bet.

C4: Customer chatbot on the website

Verdict: no, not now. A no about timing, not about principle. The 2025 POC stopped after four months for three reasons, and those three reasons have not moved. No source of truth is exposed for stock and prices. No evaluation set exists, so nobody can say whether the tool answered correctly. No procedure hands over to a human when the machine leaves its scope. Restarting without settling those three points means repeating the failure with a newer model.

The three conditions for reopening are checked, not declared.

• Stock and prices are machine-readable, in real time, through an interface supplied by the ERP vendor, and that interface returns the same value as a salesperson's screen.
• An evaluation set of at least two hundred real customer questions exists, with the right answer written beside each one, and the share of correct answers is measured before anything goes live.
• An escalation rule is written and tested: after two failures, or on any negotiated price question, the conversation goes to the order desk with its history.

Transparency comes on top: the European AI regulation requires the user to know they are talking to a machine. Of the five cases, this public chatbot is the only one concerned. One sentence on screen is enough, it is not a project. The framework is set out in the EU AI Act for businesses.

C5: Stockout forecasting

Verdict: this is not generative AI. Forecasting a stockout means reading outbound history, supplier lead times and seasonality, then applying statistics. A language model adds nothing here, and it introduces a source of error where the subject calls for steadiness. The real obstacle is elsewhere: the ERP history has a seven-month hole in 2022, at the migration. A forecast fitted on an incomplete year gets it wrong with confidence.

Instead: deal with that hole, set a simple calculation method on the references that matter, and measure the gap between forecast and actual for a quarter before automating anything. It sells less well than an AI project. It is more useful, and that data work then serves every other case, including the ones we start. The outline is in is your data ready for AI.

A case ruled out with a written condition is worth more than a case launched without knowing what would make it fail.

This report converts no gain into euros. I do not know your loaded hourly cost, on the order desk or in support. You know it, and so does your finance department. An invented amount in an audit report discredits everything around it, including the parts that hold. So I give hours per year, with the terms of the calculation set next to the result. The multiplication is yours: it takes ten seconds and it produces a number you can defend.

Two workstreams already carry a gain you can quantify: C2 on document search, C1 on the tail of orders arriving by email.

These hours do not turn into cut jobs, and I will not sell you the opposite. 630 hours a year spread over four branches and 14 people on the order desk comes to a few hours a week per site. What they become is more useful: orders keyed the same day instead of the next morning, a customer called back before noon, and the retyping nobody ever liked doing, gone. If your goal is to cut headcount, say so now, because that changes how the project is run, and I will not present it to the teams under another name.

An hour saved becomes a response time before it becomes a budget line.

Running cost is computed from volumes, not from a flat fee. C2 sends about 13.2 million input tokens and 0.88 million output tokens a month, the token being the unit of text the model bills. C1 sends about 4.86 million input and 0.54 million output. The formula is stable: cost = (input tokens in millions) x P_input + (output tokens in millions) x P_output. P_input and P_output are the published prices per million tokens of the model you pick, on the day you decide. I do not write them here: they move several times a year, and a rate copied into a report is wrong before it is read. Take the vendor's price of the day, put it in the formula, and add hosting and supervision, which are not tokens. The detail workstream by workstream, including the first indexing of the product records, is in appendix A4, and the levers that bring the bill down are covered here.

Implementation effort follows the same rule: the plan states it in days, never in euros. I publish no rate card, and the pricing page says why: a price posted blind would be wrong for your case. You get the number from the first exchange, before any commitment.

Four risks can stop one of these projects after it starts: regulation, supplier contracts, adoption, and dependency on one provider. The last three weigh more here than the first.

What regulation asks for here

The European regulation on AI, known as the "AI Act", sets a transparency obligation: a user must know they are talking to a machine. It applies to case C4, the assistant open to customers on your site. If you restart it one day, it has to say it is a machine, and the handover to a human has to stay visible.

The two projects I recommend starting fall outside that scope. Internal document search, C2, and order extraction, C1, are minimal risk: the regulation puts no specific obligation on them. The point to watch is elsewhere. Orders carry personal data, a buyer's name and contact details for example, and the GDPR applies to them as it already applies to your ERP. This is not a new subject, just the same one extended to one more processing operation. For the general framework, see the EU AI Act for businesses.

Supplier catalogs

The 34,000 datasheets come from your suppliers. Indexing them so your teams can search them internally is not the same as redistributing them to your customers. The first reading is defensible, the second is not without an agreement. I will not settle the question for you: it has to be read contract by contract, and your 11 main suppliers do not have the same usage terms. The answer belongs to your legal department, or to your outside counsel. Ask before the prototype, not after.

Adoption

This is the most underestimated risk on the list. An assistant nobody opens after three weeks has cost what it took to build and returned nothing. The POC stopped in 2025 belongs to that category, and getting from POC to production is largely decided there. What answers it is nothing spectacular, and it is decided before the first day of code.

• The assistant opens inside the tool already used every day, not in one more tab to remember.
• Three people from the order desk and two from support try the prototype and say what is wrong, before any wide rollout.
• Every answer shows its sources, because an answer you cannot check ends up unread.
• Weekly active users are tracked from day one, and so is answer quality.
• A stop condition is written in advance: if usage does not take off in two months, we stop and we say why.

Dependency

Open formats, data exportable at any time, and both the report and the code produced during the engagement belong to you.

On this engagement, the regulatory risk is the best documented and the least likely. The risk of abandonment is exactly the reverse.

This roadmap commits to two things: an order between the workstreams, and a starting condition for each. It does not commit to a result. Nobody can promise today that the document assistant will hold the target threshold, because the evaluation set that would let you check it does not exist yet. What I can set out is the order in which the uncertainties get cleared, from the cheapest to the most expensive.

The order follows a simple rule: no step is funded until the previous one has produced its proof. Effort is counted in implementation days. It excludes your teams' time, hosting, and model usage, which is worked out separately.

The workstreams in the first three months run in parallel, but they do not draw on the same people. Integrating the files of the 23 customers is an ERP matter, not an AI matter: it belongs to the vendor. On its own it covers 62% of order lines, with no model involved. If it stalls, the extraction prototype loses part of its purpose, since the AI would end up handling volume that should never reach it.

Stopping criteria

A workstream is better stopped early than late. Here are the points to check, and what should stop the work rather than extend it.

• At the end of the 4 scoping days, if the questions already sent to support are not enough to build an evaluation set the business accepts, the prototype is not commissioned.
• At the end of the C2 prototype, if the assistant stays below the threshold set at scoping, the move into production, 20 days and up, is not commissioned.
• At the closing review, if support staff stop opening the prototype unless reminded, the need was not where we thought.
• In production, if the share of answers reworked by hand rises two months running, we freeze new features and go back to the evaluation set before adding anything.
• Before the C1 prototype, if the ERP vendor has given neither quote nor date for the structured files, the AI part does not start.
• At each monthly review, if the cost recomputed at current rates goes over the envelope set at scoping, we switch models or we stop; we do not let it drift.

A workstream with no written stopping condition has no serious starting condition.

Splitting the work into scoping, then prototype, then production is not specific to this case: it is how I work, described in my method. Each step ends in a decision, and that decision can be to stop.