An operator reported a merge releasing in alternating 30-second slugs instead of its configured 4:1 ratio. A byte-level diff cleared the deployed routine in minutes, a live read confirmed the configuration matched intent, and the export everyone would have reached for turned out to carry stale values that pointed the wrong way.
An operator reported that a merge point was releasing in alternating slugs of roughly thirty seconds per lane instead of the 4:1 ratio it was configured for. Four cartons from the heavy lane, then one from the light lane, is the intent. Thirty seconds of one and thirty seconds of the other is not a ratio, it is two lanes taking turns.
The useful part of this session is not the answer. It is the order of elimination, and one trap that would have sent a manual investigation in the wrong direction on its first step.
Step one: is the deployed code the code we think it is
The ratio gating is a structured text routine, reviewed and version controlled outside the controller, called from the merge routine and driving the per-lane gate states. The first question is whether what is running matches what was reviewed.
The routine was extracted from a fresh controller export and diffed against the source of record. Identical logic. The only differences were export whitespace and one comment alignment.
The wiring checked out in the same pass. The call site is the jump to the ratio routine and nothing else, the original output instructions having been removed at deployment, and the downstream rung consumes the lane states as expected.
That took minutes and it removes the entire category. Nobody has to read the ratio logic looking for a bug, and nobody has to argue about whether the right version got downloaded. This is the least interesting kind of finding and often the most valuable, because the alternative is a day spent inside logic that turns out to be correct.
Step two: the trap
The same export carries decorated tag values, and reaching for them is the obvious next move. Here is what they said against what the controller said, read directly a few minutes earlier the same afternoon:
| Value | Live read, 2:57 PM | Same-day export, 3:14 PM |
|---|---|---|
| Ratio counter | 4 | 3 |
| Rotating position | 0 | 3 |
| Starvation override | Inactive | Active |
| Lane 1 release count | 465,097 | 27,681 |
| Lane 2 release count | 285,723 | 33,398 |
Release counters only go up. A count read at 2:57 PM cannot be seventeen times larger than the same counter in an export taken at 3:14 PM. The export embeds the project file's last stored offline values, not controller state at the moment of export.
Every conclusion an analyst would draw from that column is wrong, and each one is wrong in a direction that looks plausible. The starvation override reads active when it was not. The ratio reads 3:1 when it is set to 4:1, which is exactly the kind of near-miss that sends someone off to fix a configuration that was never wrong.
The live reads settled it in the other direction: the configuration matched the operator's expectation, 4:1, confirmed on the running controller.
Step three: what the live window actually showed
A two-minute capture during the reported behavior showed something the hypotheses had to account for. Both lane photo eyes read solidly blocked, the starvation override was inactive, the second gate was held closed and enforcing the ratio, and zero releases occurred on either lane for the entire window despite lane one being loaded and enabled.
That is not a ratio problem. A merge that releases nothing while gates are enforcing correctly is waiting on something downstream.
The capture also exposed its own limit. Eighteen tags per pass meant roughly five seconds per sample, which is far too slow to catch photo-eye chatter. A test that needs to see a bit flicker has to be built for that cadence, on a handful of tags, not on a comfortable list of everything relevant.
Where the session ended
Two mechanisms, both converging on the same failure shape: the starvation override latching most of the time, which opens both gates and hands arbitration back to the stock OEM bed instruction, which releases in trains. They differ only in why it latches.
- A photo eye reading falsely clear, or chattering. A falsely clear eye on the favored lane latches the override. Chatter creates release edges without physical cartons, which advances the rotating position and burns the heavy lane's turns, drifting the ratio even with no override at all.
- A starvation timeout set shorter than real inter-carton gaps. If normal gaps on the favored lane exceed the timeout, the override latches constantly. This is a documented failure mode of the design and it presents exactly as "we set 4:1 and observe 1:1."
A discriminating test was written to separate them, at a fast cadence on five tags, with a three-row verdict table. It was never needed.
What closed it
The merge ratio rollout across all six merge points ran live testing over the following weeks and found and corrected eight defects on the line. Two of them are the two mechanisms above: a reversed photo-eye hardware convention, and the starvation timeout raised from 150 ms to 1000 ms to stop normal carton gaps from tripping the override.
Both are runtime tag values. They were retuned in production with no download.
All six merges now run gating enabled at 4:1 with the ratio verified functional on each. The slug behavior is not reported as current on any of them.
Inbound starvation fell from 36.05% to 26.3%, a drop of 9.75 points, comparing the May baseline with gating disabled against July with gating enabled, from the site's own starvation rollup. That is the same metric the local starvation monitor watches continuously, built a few weeks earlier for a different reason.
What it did not close
The starvation figure is a before and after on a live site across a rollout that changed several things at once, not a controlled experiment, and the rollup that produces it lives with the site's change record rather than in anything we can publish. It is quoted with its source named and its limits stated.
The investigation itself did not produce the root cause. It eliminated the code, confirmed the configuration, documented the export trap, and reduced the fault space to a table an engineer could act on. The rollout found the defects. That division is worth being precise about, because the value of the session was the elimination, and elimination reads like a non-result until you count the days it does not cost.
The live starvation timeout was never sampled during the session. The export's value for it turned out to match what was deployed, but that was luck, and it was not treated as evidence at the time.