An AI pilot avoids stalling when it targets one specific workflow with a named owner, measures a clear before and after, trains the people involved on their real work, and ends on a fixed date with an explicit decision to scale, adjust or stop.
Pilots tend to drift when they are open-ended, owned by nobody in particular, or judged on enthusiasm rather than evidence. The playbook below is designed to prevent each of those.
Who this is for
COOs, operations leads and team leads who have been asked to “try AI” in their area and want a structured way to do it in a few weeks.
The short version
- One workflow, one owner, one small group, one end date.
- Agree what success means before you start, in terms of the work.
- End with a decision. A pilot without a decision is just an experiment that fades away.
Why pilots stall
The reasons are usually ordinary:
- The scope is vague. “Explore AI for the deal team” has no finish line.
- Nobody owns it. It sits alongside everyone’s day job, so it loses.
- There’s no baseline. Without a before, nobody can say whether the after is better.
- People weren’t trained. Early poor results get blamed on the tool.
- Data questions block progress. Uncertainty about what’s allowed halts work midstream.
- There’s no decision point. The pilot never formally ends, so it never formally succeeds.
The one-page pilot charter
Before week one, write down answers to these questions on a single page:
- Workflow: which specific, recurring task are we changing?
- Owner: who is accountable for the pilot and has time set aside for it?
- Participants: which few people will use it, and are they willing?
- Tools: what will we use, and is it approved for the data involved?
- Baseline: how long does the task take now, and what does quality look like?
- Success criteria: what result would justify scaling? For example, time saved per instance, fewer revisions, or faster turnaround, with a threshold agreed in advance.
- Guardrails: what data is in and out of bounds, and who reviews output?
- End date and decision meeting: when will we decide, and who decides?
If you can’t fill in the owner, baseline or end date, pause and fix that first.
The six-week playbook
Week 1: Set up
- Finalize the charter and confirm tools and data rules.
- Measure the baseline on a few real instances of the task.
- Collect examples of good output to compare against.
Week 2: Train
- Run a hands-on session with participants using the real workflow.
- Build a first shared prompt, checklist or configured assistant for the task.
- Agree how people will record what they try.
Weeks 3 and 4: Run
- Participants use the new approach on live work, with review of every output.
- Hold a short weekly check-in: what worked, what broke, what changed.
- Improve the shared prompts and checklists as you learn.
- Note anything that would need engineering to work well, such as data that has to be copied in by hand each time.
Week 5: Measure
- Repeat the baseline measurement on comparable instances.
- Gather participants’ honest views, including frustrations.
- Note quality issues and how they were caught.
Week 6: Decide
- Hold the decision meeting and choose one of three outcomes:
- Scale: criteria met. Extend to more people, document the method and assign ongoing ownership.
- Adjust: promising but not there. Change one or two variables and run a short second round.
- Stop: not worth it now. Write down what you learned and move to the next workflow.
Stopping is a legitimate result. It saves time and builds credibility for the next pilot.
Choosing the right first pilot
Good first pilots are frequent, mostly text- and document-based, easy to check, low risk if a mistake slips through, and owned by someone who wants them to work. Avoid starting with high-judgment decisions, anything client-facing without review, or a workflow that would need significant engineering before anyone can try it.
Common mistakes
- Piloting a tool instead of a workflow. Ask “does this task get better?” rather than “do people like this tool?”
- Choosing enthusiasts only. Include at least one thoughtful skeptic, so the findings are credible.
- Moving the goalposts. Keep the success criteria you agreed at the start.
- Skipping review of output. Quality problems found late can undo trust.
- Scaling without an owner. A successful pilot still needs someone to carry it forward.
When you do not need us
If you have someone who can own the charter, run the training and keep the weekly rhythm, this playbook can be run entirely in-house. Outside help is most useful for the training step, for choosing the workflow, or when the pilot shows that something needs to be built.
How Ruby helps
- AI training: the hands-on week-two session, built on your real workflow.
- AI strategy: picking the pilot workflow, setting success criteria and making the scale-or-stop call with you.
- AI engineering: building the system when the pilot shows a workflow deserves one.
Kick off your pilot with one practical hour of AI training for the participants.