--- "title": "How to run an AI pilot that doesn't stall: a six-week playbook" "description": "A week-by-week playbook for running an AI pilot at a firm, with a one-page pilot charter, clear success criteria and a scale, adjust or stop decision at the end." "section": "Playbooks" "sidebarTitle": "Run an AI pilot" "published": "2026-10-16T18:49:46.043Z" "updated": "2026-10-16T18:49:46.043Z" --- An AI pilot avoids stalling when it targets one specific workflow with a named owner, measures a clear before and after, trains the people involved on their real work, and ends on a fixed date with an explicit decision to scale, adjust or stop. Pilots tend to drift when they are open-ended, owned by nobody in particular, or judged on enthusiasm rather than evidence. The playbook below is designed to prevent each of those. ## Who this is for COOs, operations leads and team leads who have been asked to "try AI" in their area and want a structured way to do it in a few weeks. ## The short version - One workflow, one owner, one small group, one end date. - Agree what success means before you start, in terms of the work. - End with a decision. A pilot without a decision is just an experiment that fades away. ## Why pilots stall The reasons are usually ordinary: - **The scope is vague.** "Explore AI for the deal team" has no finish line. - **Nobody owns it.** It sits alongside everyone's day job, so it loses. - **There's no baseline.** Without a before, nobody can say whether the after is better. - **People weren't trained.** Early poor results get blamed on the tool. - **Data questions block progress.** Uncertainty about what's allowed halts work midstream. - **There's no decision point.** The pilot never formally ends, so it never formally succeeds. ## The one-page pilot charter Before week one, write down answers to these questions on a single page: 1. **Workflow:** which specific, recurring task are we changing? 2. **Owner:** who is accountable for the pilot and has time set aside for it? 3. **Participants:** which few people will use it, and are they willing? 4. **Tools:** what will we use, and is it approved for the data involved? 5. **Baseline:** how long does the task take now, and what does quality look like? 6. **Success criteria:** what result would justify scaling? For example, time saved per instance, fewer revisions, or faster turnaround, with a threshold agreed in advance. 7. **Guardrails:** what data is in and out of bounds, and who reviews output? 8. **End date and decision meeting:** when will we decide, and who decides? If you can't fill in the owner, baseline or end date, pause and fix that first. ## The six-week playbook **Week 1: Set up** - Finalize the charter and confirm tools and data rules. - Measure the baseline on a few real instances of the task. - Collect examples of good output to compare against. **Week 2: Train** - Run a hands-on session with participants using the real workflow. - Build a first shared prompt, checklist or configured assistant for the task. - Agree how people will record what they try. **Weeks 3 and 4: Run** - Participants use the new approach on live work, with review of every output. - Hold a short weekly check-in: what worked, what broke, what changed. - Improve the shared prompts and checklists as you learn. - Note anything that would need engineering to work well, such as data that has to be copied in by hand each time. **Week 5: Measure** - Repeat the baseline measurement on comparable instances. - Gather participants' honest views, including frustrations. - Note quality issues and how they were caught. **Week 6: Decide** - Hold the decision meeting and choose one of three outcomes: - **Scale:** criteria met. Extend to more people, document the method and assign ongoing ownership. - **Adjust:** promising but not there. Change one or two variables and run a short second round. - **Stop:** not worth it now. Write down what you learned and move to the next workflow. Stopping is a legitimate result. It saves time and builds credibility for the next pilot. ## Choosing the right first pilot Good first pilots are frequent, mostly text- and document-based, easy to check, low risk if a mistake slips through, and owned by someone who wants them to work. Avoid starting with high-judgment decisions, anything client-facing without review, or a workflow that would need significant engineering before anyone can try it. ## Common mistakes - **Piloting a tool instead of a workflow.** Ask "does this task get better?" rather than "do people like this tool?" - **Choosing enthusiasts only.** Include at least one thoughtful skeptic, so the findings are credible. - **Moving the goalposts.** Keep the success criteria you agreed at the start. - **Skipping review of output.** Quality problems found late can undo trust. - **Scaling without an owner.** A successful pilot still needs someone to carry it forward. ## When you do not need us If you have someone who can own the charter, run the training and keep the weekly rhythm, this playbook can be run entirely in-house. Outside help is most useful for the training step, for choosing the workflow, or when the pilot shows that something needs to be built. ## How Ruby helps - **AI training:** the hands-on week-two session, built on your real workflow. - **AI strategy:** picking the pilot workflow, setting success criteria and making the scale-or-stop call with you. - **AI engineering:** building the system when the pilot shows a workflow deserves one. Kick off your pilot with [one practical hour of AI training](https://rubyadvisory.com/book-a-call) for the participants. ## Related pages - [Where should your firm start with AI?](/docs/where-to-start-with-ai) - [Why your team isn't using the AI tools you paid for](/docs/why-ai-adoption-stalls) - [How to keep client data safe when using AI](/docs/keeping-client-data-safe-with-ai) - [Custom AI agents for your business: what they are and who should build them](/docs/custom-ai-agents-for-your-business)