The demonstration looks promising. An engineer's rough job notes go in; a tidy customer report comes out. The office manager can see the appeal. The director asks when everyone can start using it.
Then the awkward questions arrive. Which job does the report belong to? Who checks it? What happens when the notes are incomplete? Does somebody still have to copy everything into Simpro?
That is where I'd start the AI implementation conversation: with the work surrounding the demonstration. A pilot should earn its place in the operation. Showing that a tool can produce something plausible is only the first step.
AI implementation starts with a finished job
My starting question would be: what does “done” mean for this piece of work?
For an illustrative field service business, it might mean a checked visit report attached to the correct job, available to the customer and ready for the office to use when preparing an invoice. A paragraph generated in a separate window does not meet that definition.
Write the finish line before choosing the tool. Include who needs the result, where it must land and what has to be checked before anyone relies on it.
Microsoft's implementation strategy guide recommends connecting use cases to business objectives and planning operational integration and ownership early. It is a supplier's guide, so don't read it as independent evidence of how often projects fail. The practical distinction is useful: proving a capability and running a process are different jobs.
If nobody can describe the finished job plainly, I'd pause the build. Our AI readiness check is a useful place to step back before adding more technology.
Give the workflow an owner, including its awkward bits
A director can sponsor the work without being the person who runs it. A software supplier can maintain an integration without knowing whether the office is getting usable reports.
I'd name an operational owner before widening the pilot. For this example, that could be the service manager. Their responsibility would be the reporting workflow. The internal workings of the AI model can stay with whoever supports it.
They should be able to answer:
- Who supplies the notes, photographs and job reference?
- Who checks the draft against the original record?
- Who deals with missing information?
- Who can stop the automated step and use the agreed manual route?
- Who decides whether a proposed change is ready for everyday use?
Put names beside those responsibilities. “The office” is a location. Owners are people.
Also agree what happens when the usual checker is away. If a workflow depends on someone quietly rescuing every awkward case, it isn't ready to hand over. Make those interventions visible during the pilot rather than treating them as free support.
Test the untidy work as well as the impressive examples
Picture the same illustrative business trying a report-drafting tool. The initial demonstration uses clear notes from a familiar engineer and a straightforward visit. Keep that example, but do not let it stand for the whole workload.
I'd build a test set from the kinds of records the team actually needs to handle, using material authorised for that environment. Include an incomplete note, an ambiguous asset reference, a return visit and a record where the engineer has explicitly said they could not finish the work.
For each case, have a knowledgeable person describe an acceptable result before reviewing the generated draft. That gives the checker something firmer than “looks all right”.
Microsoft's evaluation documentation distinguishes pre-production testing using evaluation datasets from monitoring after deployment. It includes checking whether outputs are grounded in their source material and whether tasks are completed. You do not need its platform to apply that distinction.
For our report example, my acceptance checks would be practical:
- No invented readings, actions or completed work.
- Unfinished work remains visibly unfinished.
- The job reference matches the source record.
- Missing information is flagged rather than guessed.
- The checker can find the original notes without a separate search.
I would keep technical judgements with the responsible person. Asking a tool to tidy language is a narrower proposal than asking it to decide what an engineer should have done.
Measure the whole task, including checking and correction
Before the pilot, follow a report through the existing process. Record the hands-on work, the waiting and the reasons somebody sends it back.
Then use the same definition of “finished” when reviewing the proposed process. Include preparation, checking, correction, copying between systems and dealing with rejected drafts. Do not count a fast draft as a saving while leaving its review effort out.
NIST's AI measurement playbook recommends defining acceptable performance limits, documenting performance before and after deployment, and regularly reviewing whether the measures remain appropriate. I would translate that into a short operational record that someone actually owns.
For this example, I'd track whether reports reach the correct job, what the checker has to change, whether the office still chases missing details and how much effort the finished report takes. Separate an awkward source record from an error introduced by the tool; they need different fixes.
Agree the decision rule before seeing the results. If checking and correction absorb the proposed benefit, simplify the scope or stop. An honest decision not to continue is more useful than declaring success because the software ran.
Design the handover before widening the pilot
I'd ask the person doing the job to walk through the proposed routine while the person who built it watches without taking over.
Where do they start? How do they recognise a draft? How do they reject it? Where does a failed attempt go? What tells them that a report is ready, rather than merely generated?
For our illustrative workflow, I would want:
- A visible distinction between draft and checked reports.
- A route back to the original job record.
- A place for incomplete or failed attempts that someone actively reviews.
- A manual fallback the office has actually practised.
- A clear instruction on who to contact and who can authorise a change.
Keep the scope narrow while that routine settles. Do not add customer emails, invoice preparation and scheduling simply because the report demonstration went well.
This is also the point to challenge the AI itself. If a structured form would capture the missing information more reliably, try that. If the real problem is transferring an approved report between systems, investigate an integration. Neither problem automatically needs generated text.
That is the approach behind Mise's Implement stage: put the agreed way of working into practice, using technology where it helps rather than making it the starting point.
Give the pilot an exit decision
This week, I'd take the current pilot and put its operating agreement on a single page: the finished job, the owner, the permitted inputs, the checking step, the fallback and the evidence needed to continue.
Book the review before expanding access. Use it to choose between a limited rollout, a specific revision or stopping. Avoid leaving the project in permanent trial mode, with nobody prepared to decide what happens next.
If it proceeds, assign time for reviewing failures and proposed changes. Keep the agreed test examples so the team can check whether a revision fixes a problem without spoiling something that already worked. Record why the decision was made as well as when the tool was switched on.
I would rather see a modest reporting improvement that the service manager can run than an ambitious assistant held together by its builder. The useful question is whether the next ordinary working day is better because the pilot exists.
If you have a promising pilot that has not become a dependable way of working, we can talk through what is missing.