A pilot exists to answer the questions a demo structurally cannot: does this software survive contact with your workflow, your staffing, your patient mix, and your data? To be worth running, a pilot needs four things set before day one — a written question, explicit pass/fail criteria, a representative site, and a defined end date with a real decision attached. A pilot without those is not an experiment. It's an unpaid, unmonitored partial rollout, and it will quietly become the thing you're stuck with.
What a pilot is actually for
Demos are performed on clean data by people who built the software. What a pilot buys you is evidence about the things that only show up in production:
- Workflow fit. How many clicks does the real task take, performed by the real person, on the real schedule?
- Data reality. Your data is messier than the demo tenant. Duplicate patients, legacy codes, half-finished migrations, that one referring provider whose name is spelled four ways.
- Integration truth. "We integrate with your EHR" covers everything from a bidirectional interface to a CSV someone emails weekly.
- Support behavior. How fast does the vendor respond when it isn't a sales call? This is the single most underrated thing a pilot reveals.
- Adoption resistance. Which staff quietly stop using it in week three, and why?
Write the question first
Before anyone touches a sandbox, write one sentence: "We are running this pilot to find out whether ______." If you can't finish the sentence, you're not ready to pilot; you're still shopping.
Good questions are falsifiable and narrow:
- "...whether front-desk staff can complete check-in in less time than today without adding steps for the medical assistants."
- "...whether the interface reliably returns results into the chart without manual reconciliation."
- "...whether our no-show rate moves when reminders are automated."
Bad questions sound like: "...whether the staff likes it." Everyone likes new software in week one and resents it in week four. That's not a finding.
Define pass/fail before you start
This is the discipline that separates a pilot from a demo with a longer runway. Write down, in advance, what results would make you buy and what results would make you walk. Then hold yourself to them.
| Element | Example |
|---|---|
| Primary metric | Time to complete the core task, measured before and after |
| Guardrail metric | No increase in downstream rework for another role |
| Threshold | Stated as a number or a direction agreed by the team, written down before the pilot starts |
| Kill criteria | Conditions that end the pilot early — data integrity problems, unresolved support failures, a safety concern |
| Decision date | A calendar date, with the decision-makers already in the room |
Pick the right site (not the friendliest one)
The instinct is to pilot with your most enthusiastic team. Resist it — partly. There's a real tension here:
- An enthusiastic team gives the software its best chance and produces a fast, clean answer about the ceiling. It also produces results that will not replicate anywhere else in the organization.
- A representative team — average tenure, average tech comfort, ordinary patient volume — produces results you can actually generalize.
The usual right answer: pilot with a representative site, but make sure it has at least one credible internal champion who is respected by peers, not just eager. And avoid piloting anywhere that is simultaneously absorbing another major change.
Map the workflow, before and after
You cannot demonstrate that software improved a workflow if nobody wrote down what the workflow was. Before the pilot starts, document the current-state process for the task in question: who does what, in what order, with what handoffs and exceptions. Free public toolkits exist for exactly this — AHRQ's Workflow Assessment for Health IT Toolkit is built for ambulatory settings and includes examples, tools, and educational material for assessing workflow around a health IT change.
Then map the intended future state. The gap between the two is your training plan, your risk list, and your list of things to measure. Most pilot disappointments are workflow surprises that a two-hour mapping session would have caught.
How long, and how big
There's no universal number, but there are principles:
- Long enough to pass the novelty curve. Week one is enthusiasm; the honest signal shows up after the newness wears off and people start improvising workarounds.
- Long enough to hit a full cycle of whatever the software touches — a full billing cycle, a full scheduling cycle, a month-end close.
- Short enough to have a real end. Open-ended pilots become permanent by default, at which point you've made a purchasing decision by inertia.
- Small enough to reverse. If backing out of the pilot would be painful, it isn't a pilot anymore.
The contract side of a pilot
Pilots involve real patient data, which means the pilot itself needs the same protections the full deployment would:
- A signed business associate agreement before any PHI goes near the system. "It's just a pilot" is not an exception to anything.
- Clear terms on what happens to the data if you walk away — return or destruction, in what format, by when.
- No auto-conversion. Watch for pilot agreements that silently roll into a paid term unless you cancel by a date buried in the schedule.
- Pricing agreed in advance for the full deployment. Negotiate the production price before the pilot ends. After a successful pilot and a trained staff, your leverage is at its lowest point of the entire relationship.
Instrument it, or you'll be arguing about vibes
Decide up front how you'll collect evidence, and assign someone to do it:
- Baseline measurements taken before go-live — you can't get them retroactively.
- A simple issue log everyone can add to, with date, role, and what broke. This becomes your support-quality evidence.
- Short structured check-ins at fixed intervals, with the same questions each time, rather than one big retrospective at the end where recency wins.
- A record of every vendor support interaction — what you asked, how long it took, whether it got fixed.
How pilots fail
- No end date. The pilot becomes production without anyone deciding.
- Success criteria written afterward. Whatever happened gets narrated as a win.
- Piloting with the vendor's implementation team doing the work. You've tested the vendor's staff, not your own.
- Testing on clean sample data. You've tested the software, not your data.
- The champion is the only user. One person's fluency doesn't generalize.
- Sunk-cost drift. Six weeks in, the effort spent starts arguing for the purchase. Kill criteria, written down early, are the antidote.
The decision meeting
Book it before the pilot begins. Bring: the written question, the pre-agreed criteria, the baseline and post measurements, the issue log, and the support-response record. Compare results to the criteria you set when you had no emotional investment. Then make one of three decisions — adopt, kill, or run one more tightly scoped pilot to answer a specific remaining question. "Let's just keep using it for now" is not on the list.
Common questions
How long should a software pilot run?
Long enough to get past the novelty period and through at least one full cycle of whatever the software touches — a billing cycle, a scheduling cycle, a month-end close — and short enough to have a real, dated end. Open-ended pilots become purchases by inertia.
Should we pilot with our most enthusiastic team?
Usually not on its own. An enthusiastic team shows you the ceiling but produces results that won't generalize. Pilot with a representative site that has at least one credible internal champion, and avoid any site already absorbing another major change.
Do we need a BAA for a pilot?
If the pilot involves protected health information, yes — the same requirements apply as for a full deployment. "It's only a pilot" is not an exception. Also settle what happens to the data if you walk away.
When should we negotiate the production price?
Before the pilot ends. Once the pilot has succeeded and your staff is trained on the system, your leverage is at its lowest point of the entire relationship.