SCADA System Audit Using Observations
Compare SCADA states with observations while accounting for time alignment and each system’s definitions.
Updated:
Comparing automatic records with observations may reveal differences in definitions, timing, coverage or classification. This compares measurement systems: neither sensors nor people should be assumed infallible. Locate discrepancies and investigate causes before correcting indicators.
Define the comparison
Select equipment, period, states and mapping rules. “Motor drawing current,” “active cycle” and “processing a part” need not mean the same state. PLC logic may combine signals to distinguish many causes; not every system has only a running bit.
Observers also have limits: they may not see an internal jam or know why material is missing. Observation supplies context and hypotheses, not automatic root-cause proof.
| Protocol element | Question to resolve |
|---|---|
| Running state | Does it include running empty? |
| Time reference | Event, acquisition or reception timestamp? |
| Brief stops | What does the PLC, historian or dashboard filter? |
| Manual causes | Who assigns the code; can it remain unknown? |
| Scope | Same equipment and time windows? |
Distinguish acquisition, storage and aggregation
A brief stop may disappear when only periodically polling a state, but remain available in event logs or counters. The outcome depends on architecture, filters, latency and historian rules, not a universal one-to-five-second polling interval.
Check resolution, time zone, clock synchronization, delays and communication losses. A transition close to the reading time can produce apparent disagreement when timestamps are misaligned.
Current signals or cycle counters do not necessarily represent good units. Check definitions and production records. Do not attribute discrepancies to operator manipulation without evidence; incentives, ambiguous interfaces and technical failures are hypotheses to investigate.
Select and record comparable pairs
Select moments and resources with defined probabilities across included shifts, products and conditions. At each moment retain observed and corresponding digital states, synchronization information and uncertain-classification flags.
Random scheduling reduces convenient-time selection but does not ensure independence or eliminate reactivity. Pairs from one machine or round may be related. Inaccessible targets are missing readings to document, not silently replace with visible machines.
For a discrepancy proportion under approximately independent binomial sampling, planning may use n=ceil(z²p₀(1−p₀)/e²). Precision for a paired difference, state-specific sensitivity or rare category may need a different calculation. One sample size does not validate every metric.
Simulated example: frequencies do not establish record agreement
In this teaching set of 1000 time-aligned pairs:
| SCADA record | Observed running | Observed stopped | Total |
|---|---|---|---|
| Running | 700 | 100 | 800 |
| Stopped | 50 | 150 | 200 |
| Total | 750 | 250 | 1000 |
SCADA indicates running in80% of pairs versus75% by observation. The net difference is5percentage points, but total disagreement is(100+50)/1000=15%. Comparing two histograms alone conceals disagreements in both directions. Equal percentages could also hide compensating disagreements.
If pairs were independent, define D=digital state−observed state, coding states 1/0. Mean D is 0.05. A normal approximation uses:
SE(difference)≈√(((100+50)/1000−0.05²)/1000)=0.0121458.
The approximate 95% interval is 0.05±1.96×0.0121458, or 2.62–7.38percentage points. Do not analyze these as two independent samples; temporal dependence would require another analysis.
This suggests a systematic difference in the example's pairs, not which system is correct. Observation needs a sufficiently defined and checked reference before being treated as ground truth. Comparing85% digital with75% sampled and a±3point margin without checking denominators, periods and structure cannot invalidate an entire SCADA system.
Investigate and document correction
Examine discordant pairs by equipment, shift, state and proximity to transitions. Check event logs, work orders and authorized controlled tests. Repeat verifiable reference situations without compromising safety.
When a faulty rule is identified, correct logic or mapping and compare again. Retain versions and traces to explain changes. Do not apply one correction factor to an entire history without evidence that the same error and conditions applied then.
Review frequency depends on risks, changes and record quality. This method has no universal quarterly-audit requirement.
OEE, energy and evidential scope
OEE needs A=run time/planned production time, P=ideal cycle×total count/run time and Q=good count/total count. A poor running definition may affect A; production and ideal cycle also need checking. An observed state does not provide complete energy, performance or quality data.
State review does not replace a statutory energy audit or certify grant returns. A funding scheme's evidence and eligibility depend on its specific rules. Economic return needs verifiable costs, revenue or avoided costs, not a percentage difference alone.
Tools and safe review
WorkSamp supports sampling visits. Do not assume a SCADA connector, automatic paired histograms or certified calibration. Design and verify extraction and comparison. A pending local save is not a confirmed server record.
Cronometras can examine specific durations; Induly provides working-time records, not PLC states. The ASETEMYT directory lists specialist resources.
Coordinate access and tests with operations and automation owners. Prefer authorized read-only procedures; do not modify controls for study convenience. NIST operational-technology guidance provides security/reliability context, not endorsement of a sampling application.