The screen read waiting on a host assignment. The host had dispatched four assignments during the stop and the controller rejected all four with the same status code. Two local causes, and the second one was created by recovering from the first.
The message on the screen said the cell was waiting on a host assignment. It had said that for five hours. The host was fine. It had dispatched four fresh assignments during the stop and the controller had rejected all four, silently.
The diagnosis took three sources at once: the export for the rungs and the constants, live reads for which condition was actually false, and the floor to know the pallets were physically empty. Neither root cause is visible in any two of them.
Timeline
| Elapsed | Event |
|---|---|
| T+0 | A daily flag unlatches by design at the start of the day. The check that is supposed to re-set it never runs. The cell begins rejecting every host move with status 106. |
| T+4h19 | Operator reports the cell waiting on the host. Investigation starts. |
| T+4h20 | Live reads: host heartbeat healthy, connection bit true. Not a comms fault. The handshake is frozen with a move returned to source. |
| T+4h37 | Operator cycles the controller mode. The host immediately dispatches a fresh move, and it is rejected with status 106. First root cause identified. The mode cycle also arms the second one, unnoticed. |
| T+4h49 | Robot abort, then a robot controller power cycle. The blocking output survives both. Meanwhile the host dispatches and the controller rejects two more moves. |
| T+5h15 | Operator runs the manual home sequence from the screen. The blocking output drops, the robot reports ready, the daily check finally runs and passes. Moves start succeeding. |
| T+5h15 to T+5h32 | Every move, including fresh totes, recirculates. Live read shows all six pallet positions reporting full while physically empty. Second root cause identified. |
| T+5h40 | Site controls engineer writes the pallet counts back to empty, online. |
| T+5h41 | Verified live: moves accepted, real destinations assigned, pallet counts climbing, no fault. Resolved. |
Root cause one: a quiet permissive behind a robot signal
The gate is a daily camera calibration check. One rung unlatches the passed flag every new day, and another re-latches it only when the robot completes its camera check program.
While that flag is down, the execution status routine stamps every in-progress move with the camera calibration status code. The move is marked invalid, completes as returned to source, the host backs off, and the diagnostics routine raises the message that renders on the screen as waiting on the host.
Which is how a five hour outage produces no alarm. Not one of the cell's five alarm words had a bit set. The waiting message is a plain output coil, not a fault.
So why did the check never run? Its robot program requires the robot to be ready, and ready requires no program active, and that requires all ten robot outputs to be off. One of them was stuck on.
That output survived an abort of all robot programs and a full controller power cycle, which is the detail that identifies it. A latched program output does not survive a power cycle. This one is asserted by robot-side logic or configuration, not by a paused program. It finally dropped when the operator ran the manual home sequence from the screen.
The type definitions in the export carried enough documentation to orient quickly on which robot signals mattered, which is a small thing that saved a lot of guessing.
Root cause two: the recovery armed it
On first scan, the controller deliberately assumes every pallet is full:
Main / Calls, rung 9
XIC(S:FS) XIC(PalletsEnabled) -> MOV(<full>, PalletPos1..6.ToteCount)That is conservative and correct. After a restart the controller cannot know what is stacked on each pallet, and it must never let the robot place onto an unknown build.
The mode cycle used to diagnose root cause one fired that rung. From then on all six positions reported present and full, zero capacity. So once the calibration gate finally cleared and moves started succeeding, the host did the only sensible thing with six full pallets and recirculated every tote, old and new.
The designed reset for this is the physical pallet exchange, which clears and re-registers a position. With the pallets physically empty and the loaded latches already consistent, verified live before anything was recommended, writing the counts back to empty was the equivalent. The site controls engineer made that write. The engine never wrote a tag.
Why the screen misled everyone for five hours
Three things compounded:
- The message blames the wrong side. Waiting on the host implies an upstream problem. The controller knew it was rejecting work. It had the status code in a tag, four times over, and never surfaced it.
- The calibration gate is silent. An overdue daily camera check produced no operator-visible alarm at all. It only manifested as host rejection plumbing.
- The recirculation looked like a host malfunction. It was the controller's own restart behavior working exactly as designed, on data nobody knew had been reset.
What we recommended to the site
- 01Split the message. Distinguish "the host has not assigned work" from "this cell is rejecting host work, reason: X". The controller has the status code. Show it.
- 02Alarm the calibration gate. The flag being down while the machine is in cycle deserves an explicit operator message, not silence.
- 03Put pallet counts on the restart checklist. After any first scan, counts read full by design. The screen should say so, and re-registration should be a step rather than folklore.
- 04Identify the robot output. Which output, what asserts it, and why only the home sequence cleared it. It cost roughly forty minutes of aborts and power cycles, and it is still the one open item.
- 05Watch rejections on the host side. Four consecutive rejections with the same status from one cell should page a human, not silently recirculate.
What it cost, and what it did not
Roughly fifteen live read passes across about two hundred tags. Zero writes by the engine. Every corrective action, the mode cycle, the robot power cycle, the manual home, and the pallet count write, was taken by a person.
The parts that stayed open: nobody has yet identified which robot output was asserted, what asserts it, or why only the home sequence cleared it. That question cost roughly forty minutes of aborts and power cycles during the stop and it is still unanswered. The robot side is not in the engine's corpus, so it could narrow the output down to a single assembly bit and no further.
The other honest note is on ordering. Root cause two was created by the operator's mode cycle at T+4h37, and it went unnoticed for thirty-eight minutes because nothing reads out pallet counts until moves start succeeding again. A live read of the pallet array at the time of the mode cycle would have caught it immediately. It was not on anyone's list to check.