Validating the 99% Confidence Level in Studies
What changes when planning a study at 99% confidence: critical value, precision, sample size and assumptions.
Updated:
A 99% confidence level is a choice about an inference procedure, not a certificate of accuracy. It may suit decisions where the cost of an erroneous conclusion warrants more observations, but it is not mandatory for investment, negotiation or auditing and does not automatically create legal evidential standing.
Confidence and precision are different
Under a procedure with nominal 99% coverage, intervals in repeated studies satisfying its assumptions would contain the parameter approximately 99% of the time. This does not mean observing 99% of workers, an error of 1%, or a 99% probability that a causal explanation is true.
At a fixed sample size, higher confidence widens the interval. Keeping the same approximate margin requires more readings. A biased sample remains biased even when its interval is narrow.
Required observations
For a proportion under approximately independent binomial sampling:
n = ceil(z²p₀(1−p₀)/e²)
p₀ is the planning proportion; e is the absolute margin. With no prior information, p₀=0.5 maximizes the variance of one proportion. This does not automatically accommodate clustered rounds, unequal weights or missing observations.
An illustrative plan with e=0.03 and p₀=0.5 gives:
| Two-sided confidence | z used | Unrounded calculation | Integer n |
|---|---|---|---|
| 95% | 1.96 | 1067.111 | 1068 |
| 99% | 2.576 | 1843.271 | 1844 |
The ratio before rounding is (2.576/1.96)²≈1.7273: about 73% more readings under these conditions. This is not an empirical cost benchmark or a 73% accuracy improvement. The z values are rounded; another declared approximation can change the final integer.
For n=1000 and p̂=0.40, the normal-approximation standard error is √(0.4×0.6/1000)=0.01549. Approximate margins are 3.04 percentage points at 95% and 3.99 at 99%. Near zero or one, consider a method such as Wilson; not every interval is symmetric.
Design before selecting a headline confidence level
- Define the population, shifts and period covered by the conclusion.
- Establish the decision threshold and useful precision; a small difference may have little economic importance.
- Agree observable categories and assess observer disagreement.
- Select times and resources with known probabilities. Several people in one round are not necessarily independent observations.
- Specify missing readings, incidents, weights and process changes.
Pilot information can update a plan. Repeatedly inspecting an ordinary interval and stopping when it looks favorable does not automatically preserve nominal coverage. Predefine the stopping rule or use an appropriate sequential method.
Analysis checks
A runs test may detect certain patterns, but does not prove independence or normality. No signal may simply mean low power. Examine study design, dates, subgroup composition and context changes. A changing period can still be described; projecting it as one stable process requires care.
Random scheduling reduces convenient-time selection. It does not automatically remove observation reactivity or anonymize data. A workstation, time and shift may indirectly identify someone. Where personal data are involved, assess purpose, legal basis, information and retention under GDPR and applicable employment rules.
Sampling, sensors and economic decisions
The methods can complement each other. Sampling supplies states and context; sensors supply signals whose interpretation also needs validation. Neither automatically establishes causes.
OEE requires A=run time/planned production time, P=ideal cycle×total count/run time and Q=good count/total count. Visit-state categories alone do not provide performance and quality.
WorkSamp supports visit planning and recording and provides Wilson intervals in advanced reports. Software does not make a study forensic evidence or eliminate recording errors. Cronometras can complement specific duration analysis; Induly working-time records answer a different question. Assess investments against costs, saleable capacity, demand and risks.