The site wanted an alert when a line had been starved more than a set percentage of the last twenty minutes. The number lived in a cloud analytics stack nobody could get API access to. Every term in its formula was either a controller tag or a known constant.
"I need to know if a line has been over a set percentage starved over the last set time range. Say twenty percent over the last twenty minutes. And I guess we have to look back over a window, because it is a calculation, not a real-time value. Right?"
Right, and that question contains the whole design.
Two facts follow from it. Starvation has no instantaneous value: at any single instant a line is running with product, running empty, or stopped. Twenty percent starved is only meaningful as a ratio over an interval, so the window is mandatory and both the window and the threshold belong to the user.
And starvation is a calculated metric, not a tag. The site's performance backend does the math and publishes the number to a cloud analytics stack. Getting at that number through an API was blocked and stayed blocked.
The decision: compute it locally instead of waiting
The backend's formula was documented, and every term in it is either a live controller tag or a known constant:
Real time producing = production cycle time x items produced
Starved during production = producing event duration - real time producing
Starvation % = starved during production / producing event duration| Term | Source | Live? |
|---|---|---|
| Producing event duration | Time the running bit is true within the window | Tag |
| Items produced | Change in the line's counter across the window | Tag |
| Production cycle time | The line's design rate | Constant, adjustable per line |
Three tags per line segment, six segments: the running state, the item counter, and the downstream-disabled bit so blocked time is not counted as starved. Blocked is full behind, starved is empty in front, and attributing one as the other is the fastest way to lose an operator's trust in the number.
The one non-controller term is the design rate. It is exposed as an adjustable parameter rather than baked in, so it can be calibrated against the dashboard tile later instead of being asserted now.
The capability underneath is not about starvation
The monitoring layer that makes this possible knows nothing about OEE. It watches any numeric signal over a window and delivers to a webhook. The same tool alarms on tank level, motor temperature, or packet loss with no new code.
The pieces that matter:
- Windowed evaluation is pure and deterministic. Given a series of timestamped values, did the condition hold for at least X percent of the last N seconds? Samples are passed in explicitly and the clock is injectable, so the verdict is exact and testable rather than dependent on when it happened to run.
- Coverage is checked before the verdict. If the window is under-sampled, the result reports insufficient coverage and the breach is forced false. It will not fire on a gap.
- A dropped subscription is marked as a gap, not held. A disconnect can never be counted as stale covered time, so an outage suppresses a fire instead of manufacturing one.
- Alerts fire on the rising edge and are rate limited. A line that sits starved alarms once, not every minute, and re-arms when it recovers.
- Delivery is reported honestly. No webhook configured returns not delivered with a reason. There is no fake success.
- Secrets stay out of stored config. The webhook URL comes from the environment and is never persisted with the watch definition.
Composing the two
The starvation answer is a composition, not a feature. Poll the three tags per segment, derive a starved series across each sample interval, evaluate it over the window, and alert on the rising edge.
There are two defensible ways to build that series, and the choice belongs to the site rather than the engine:
- Rate based, which matches the dashboard: starved means running while the counter is not advancing at the design rate. This captures partial starvation, a line running under-fed, and it needs the design rate to be right.
- Presence based, which needs no rate assumption: starved means running while the counter is not advancing at all. It catches only hard starvation and makes a good cross-check on the first one.
The evaluation layer does not care which series it is fed.
Scaling, which is the part that usually breaks
Constant polling is where monitoring deployments fall over. The firing service holds one subscription per controller, over the union of every rule's and every watch's source nodes. Adding a windowed starvation watch adds its three tags to that one subscription. It does not open another connection.
Many users and many watches on one controller share one connection. Adding a watch widens the node set and nothing else. On a plant network, that property is not a nice-to-have.
The follow-up, four weeks later
The alerter shipped the same day and has run continuously since. The more interesting update is that the metric it watches moved.
Over the following month the site rolled out ratio gating across all six merge points, the change the merge slug-release investigation fed into. Inbound starvation fell from 36.05% to 26.3%, comparing the May baseline with gating disabled against July with gating enabled.
Worth being exact about what that does and does not say about this system. The monitoring pipeline did not cause that drop. The merge change did. What the pipeline provides is that the number is now watched continuously and locally, on a window and a threshold the site sets, rather than being read off a dashboard tile after the fact through an API nobody could get access to. The drop is the site's evidence. The alert is what makes the next one visible while it is happening.
What it did not close
The design rate is the sensitive knob. Set it too aggressively and starvation inflates. It stays per-line adjustable until it is calibrated against the dashboard for the same window, and that calibration is the real correctness check on the reconstructed math. Until cloud access lands, this number is internally consistent and not yet reconciled with the tile.
The denominator has to match the dashboard's definition too. Percentage of wall clock and percentage of observed producing time are both defensible and they do not agree, so the alert and the tile have to be set to the same one or the argument moves from the line to the meeting room.
Two smaller ones, noted for whoever tunes this next. Three of the six segments count one unit upstream of the discharge, which is fine for a per-segment trend and needs care in reconciliation. And the counter rolls over, so the delta has to wrap.