The Pilot Study: Refining the Sample
Use a pilot to test categories, coverage and study feasibility without assuming automatic statistical guarantees.
Updated:
A pilot tests whether a study can answer the business question. Review categories, routes, safe access, recording and coverage before the main campaign. It does not guarantee removal of bias or reveal an exact “true” sample size.
What to test
Check whether observers apply the same rules, whether people and equipment can be viewed safely, actual round duration, hidden states and differential missing readings by shift. Define examples and a not-observable category without inventing activity.
There are no universal percentages of pilots with overlapping categories or reactivity. Do not remove a rare state merely because it is below 5%: it may matter for safety, quality or disruption.
Updating p₀ without claiming certainty
The simple reference is:
n = ceiling[z²p₀(1−p₀)/e²]
At 95% confidence, e=0.05 and p₀=0.5, n=385. A pilot suggesting p₀=0.30 gives 323; p₀=0.20 gives 246. The 385 design was not therefore wrong: 0.5 protects against initial uncertainty about p.
Before rounding, the 0.5 sample is 19.0% larger than the 0.3 sample and 56.25% larger than the 0.2 sample. The pilot also costs effort and its estimate may be unstable. Reducing the sample using an overly extreme estimate can worsen achieved precision.
There is no universal minimum of 20, 30 or 50 readings that stabilises every category. WorkSamp's product rule updating p after 20 visits and bounding it to 0.05–0.95 is a planning choice, not mathematical proof of stability or independence.
Teaching example
A pilot finds 20 waits in 100 records: p̂=0.20. The approximate normal 95% margin is 7.8 points; Wilson gives approximately 13.3–28.9%. For five-point planning, the central estimate suggests n=246 under the simple model. Consider retaining a prudent p₀ if waiting differs by shift or the pilot misses the future mix.
If cleaning and supervision overlap, define a primary state and separate secondary attributes. Do not add attributes as if they represented different time.
Include or exclude pilot records
Decide in advance whether pilot records enter the final analysis. Combining may be appropriate if population, selection, categories and protocol are compatible and the analysis respects the design. If definitions change, retain traceability and justify recoding or separation; do not delete data because results change.
Comparing pilot halves does not isolate the Hawthorne effect: mix, demand or learning may differ. Do not apply an automatic correction factor. Explain the purpose and document familiarisation without promising removal of reactivity.
Pilot deliverables
Provide versioned categories, incidents, actual round duration, planned coverage, sample parameters, missing-data treatment and analysis plan. Review safe access and data protection with competent staff. A pilot does not certify legality or ergonomic adequacy.
WorkSamp supports planning and recording. An activity proportion alone does not calculate OEE, which also requires ideal cycle time and total/good units.