IOMCU status read errors (RSErr) on Cube Orange+, no DShot. What’s actually causing them?
Been chasing this one for a while and I’m stuck, so hoping someone who knows AP_IOMCU or the Cube Orange+ hardware can point me somewhere useful.
Quick note so nobody wastes time on it: I’m not asking about Internal Error 0x1000 itself. I’ve read #28534 and #28725, I get that read_status_errors never resets and eventually trips the limit of 20. That part makes sense to me.
What I can’t work out is why the status reads are failing at all.
Setup
Cube Orange+, plain PWM outputs, BRD_IO_DSHOT = 0. Flights run several hours. I’m comparing ArduCopter 4.3.8 against ArduPlane 4.5.7.
What I’m seeing
Best data point I have is a flight where I had two Cube Orange+ units up at the same time on the same aircraft, one running Copter 4.3.8 and one running Plane 4.5.7.
The Copter one finished with RSErr at 0 and Nerr at 0. Not a single failed transaction the whole flight.
The Plane one started with Nerr at 1 (it does this every boot, always exactly 1) and then over the flight RSErr crept up to about 30, with Nerr climbing alongside it. Ended with 0x1000 iomcu_reset.
I’ve also got other Plane 4.5.7 logs where RSErr sits at 0 the entire flight and nothing happens at all. So it’s not like the firmware just always does this.
Why I don’t think it’s a bad link
Couple of things bug me about the marginal hardware explanation.
The rate is really low. Status gets polled every 50ms, so over a few hours that’s roughly 200k reads and I’m seeing about 30 failures. That’s 0.01%. If the link were genuinely bad I’d expect a lot worse than that, and if it were fine I’d expect zero. 0.01% is a strange number to land on.
They also trickle in rather than arriving in bursts. Once it starts it just keeps going at a slow steady rate for hours.
The Nerr = 1 at boot looks like an init artifact to me rather than anything physical. It’s exactly 1, every time, completely deterministic. My guess is it’s the first PAGE_CONFIG read during IOEVENT_INIT going out in the long packet format because is_chibios_backend hasn’t been set yet, failing once, then succeeding on the retry.
But the thing I keep coming back to is that Nerr climbs at the same rate as RSErr, not faster. total_errors counts everything: RC input, servo reads and writes, safety options, status. If the link itself were marginal I’d expect the other transactions to be failing too and Nerr would outpace RSErr. It doesn’t. Which says to me the status read is the only thing that’s failing.
What I suspect (tell me if I’m off base here)
In read_registers():
if (!uart.wait_timeout(count*2+4, 10)) {
protocol_fail_count++;
return false;
}
That’s a flat 10ms timeout no matter how many registers you asked for. And read_status() pulls sizeof(reg_status)/2 registers, which as far as I can tell is the biggest single read anywhere in the protocol.
If that’s right then PAGE_STATUS has the tightest timing margin of anything, which would explain why it’s the only transaction failing. And page_reg_status picked up extra fields between 4.3 and 4.5 with the IOMCU DShot work (rcout_mode, rcout_mask, others), so that margin would have got tighter. Which lines up with 4.3.8 sitting at zero and 4.5.7 showing a small but non-zero rate on the same hardware.
I might have this completely wrong though.
Questions
Has anyone else seen RSErr slowly accumulating on a Cube Orange+ with plain PWM and no DShot?
Is PAGE_STATUS actually the longest read in the protocol? And has that 10ms timeout been looked at since the status page grew?
Any idea what would make one flight completely clean and another one on identical firmware accumulate errors? Something load related on the IOMCU side maybe, SBUS out generation or RC decoding or the heater PWM?
Anything in AP_IOMCU, ChibiOS or the IO firmware between 4.3 and 4.5 that would plausibly change FMU to IOMCU transaction timing?
And if this really is a timing margin rather than a counting problem, would raising the status read timeout or trimming the status page be a better fix than the consecutive failure counter in #28725?