What a pilot is for
By this point you have a specification, a comparison and a cost model. All three rest on claims you have not tested: that the product performs as described in your environment, that your people will actually use it, and that the running costs are what the supplier said. A pilot converts those three assumptions into observations for a small fraction of the full purchase.
It is not a demonstration and not an extended sales meeting. A demo shows you the product working; a pilot finds out whether it works here.
Write the question first
The most common pilot failure is running one without a question. Units arrive, people try them, everyone forms an impression, and at the end the decision is made on the same soft evidence as before — only later and with more effort spent. Before anything is ordered, write a single sentence:
Pilot hypothesis
“If we deploy six of these on the night shift for four weeks, scan failures per thousand picks will drop below fifteen and no unit will need a battery swap mid-shift.”
Specific, measurable, time-boxed, and capable of coming out false. That last property is what makes it worth running.
Scope: small, real and time-boxed
- Sample. Enough units to see variation, few enough that failure is cheap. For most equipment purchases that is 10–20% of the eventual fleet, with a floor of two or three so a single faulty unit does not decide the outcome.
- Duration. Long enough to cross a full operating cycle — a full week of shifts, a month-end, a peak period. Two to six weeks suits most purchases. Longer pilots do not produce better data; they produce a de facto rollout nobody approved.
- Site. Choose a representative environment, not the easiest one. A pilot run in the clean, well-lit, well-staffed location tells you very little about the site where the problem actually is.
- Users. Include at least one sceptic. Enthusiast-only pilots pass every time and predict nothing.
Decide the measurements before you start
Pick three to five, mostly quantitative, and record how each will be captured and by whom.
| Measure | How captured | Pass mark |
|---|---|---|
| Task completion time | Timed sample, ten tasks per user, start and end of pilot | No slower than current method after week two |
| Failure or error rate | Daily tally sheet at the workstation | Below the agreed threshold in the final week |
| Unplanned downtime | Log every stoppage with cause and duration | No stoppage longer than thirty minutes |
| User rating | Same three questions, week one and final week | Improving, and no user rating it unusable |
| Support responsiveness | Log every ticket raised and time to resolution | Within the response time quoted |
Include a baseline wherever you can. “Faster” is not a finding unless you measured the current method the week before under the same conditions.
The commercial terms to agree in advance
Pilots go wrong commercially more often than they go wrong technically. Settle all of this in writing before the units ship:
- What it costs. Free, discounted, or full price with credit against a subsequent order. All are reasonable; being unclear is not.
- Who pays for what. Delivery both ways, installation, training, consumables, and removal at the end.
- What happens if it fails. Return at whose cost, and by when. Get the return shipping arrangement in writing — this is the term suppliers most often leave vague.
- No commitment implied. Explicitly state that the pilot creates no obligation to purchase, so that a successful trial does not become an assumed order.
- Data and configuration. If the product collects data or holds configuration, agree that it is yours and will be deleted or exported at the end.
- Price protection. Lock the quoted unit price for a period after the pilot ends, so a good result does not trigger a “revised” quote.
The trap
A pilot that quietly becomes production. Once the units are load-bearing, the pilot cannot fail — removing them now costs more than buying them. Set the end date, and keep the fallback method available for its whole duration.
Run it, then hold the review
Check in weekly rather than only at the end; a fixable configuration problem discovered in week one is worth more than a post-mortem in week five. Log issues as they happen, with dates — memory reorganises itself around whatever conclusion people already prefer.
At the end, hold a scheduled review with the users, whoever will sign the order, and ideally the supplier for part of it. Go through each measure against its pass mark and record the result plainly. Then make one of three decisions: proceed, proceed with specified changes, or stop. Write down which, and why.
When a pilot is not worth it
Pilots cost real time, and not every purchase justifies one. Skip it where the item is a commodity with a well-understood specification, where the purchase is small enough that being wrong is cheap, where the product is trivially returnable, or where a reference site running the same thing in the same conditions will answer your question in a phone call. Save the effort for purchases that are large, hard to reverse, or dependent on how your own people work.
Back to the start
If the pilot fails, the useful next step is usually not another supplier but a second look at the requirements spec — a failed pilot frequently means the requirement was wrong rather than the product. If it passes, feed the real numbers back into your cost of ownership model and into the supplier comparison before the order goes out.