Plane 3.9.3 stable release

[tridge]

The ArduPilot development team are delighted to announce the 3.9.3 stable release of the ArduPilot plane code. This release includes a number of small but important fixes over 3.9.2:

• fixed error handling for corrupt RC input that could lead to a
  crash in unusual circumstances

• fixed a race condition in IOMCU event startup that could lead to
  the safety not being disabled on boot with BRD_SAFETYENABLE=0

• ensure surface speed scaling covers full range of configured
  airspeeds

• added builds for new boards F35Lightning, omnibusf4v6, mRoX21-777

• updated GPIO numbers on AUX pins on all boards to be consistent
  with docs

• updated KakuteF7 UARTs and buzzer

• added ESC sensor uart on OmnibusNanoV6

• fix for Benewake rangefinder at long ranges

• prevent attempts to erase dataflash logs while armed

Many thanks to the people who tested this release!

Happy flying!

[yak-54 (Colin)]

Ok Andrew got the logic analyzer today
just made the cable for the cube
installed Logic 1.2.18 test it its logging data what type log do you need Ascii & Hex Bin Dec Hex ?
how long to log for
updated cube to 3.9.3 ChibiOS

[jhw84 (Joost)]

Hi,

I have a problem with I2C in ChibiOS that might be related to this, if not I can open a new topic.

I am communicating over I2C with a custom device, that applies very long clock stretching (varying from 3-20ms) in a small fraction of its packets. In some cases, this has lead to a locked i2c bus for us.

I have investigated using a logic analyser, and found that the following happens in ChibiOS:
(1) Master sends a READ command to slave.
(2) Slave holds the clock for some time (likely to process some data before sending it).
(3) After some time, releases the SCL line and starts transmitting data correctly.
(4) In the middle of the transfer, the master determines that the transfer is taking too long (i2cMasterReceiveTimeout() called in AP_HAL_ChibiOS/I2CDevice.cpp returns MSG_TIMEOUT), aborts the transfer and stops cycling the clock.
(5) In the unfortunate case where the last transmitted bit happens to be a 0, the slave holds the SDA line low (active). The slave does not time out (!) so it keeps the line low, as it is still trying to finish transferring the data and is waiting for the next clock cycle. The low SDA is preventing the master from sending a new START condition,

This results in an unusable I2C bus.

In Nuttx, the problem did not present itself as clearly to us, since the master (AP) does not appear to time out there, allowing to slave to finish its transfer of the delayed data and leaving the bus in a clean state.

To emulate the NuttX behaviour, I tried to increase the timeout value of the i2c device in ardupilot, which seemed to have no effect. I even replaced the last argument of i2cMasterReceiveTimeout in I2CDevice.cpp to TIME_INFINITE (found in the ChibiOS documentation), but the transfer seems to time out anyway after about 3 or 4 ms, I don’t know why.

Until we make the slave behave a bit nicer, is there a way to call an i2c reset (by cycling the clock) from within ardupilot, or increase the timeout?

I am building arducopter from the master branch, and am using a pixhawk2.

Attached (below) is a file with the logic analyzer captures in both Nuttx, and ChibiOS.

In the case of Nuttx, the packet of interest is the 4th. For ChibiOS, the packet of interest is the 8th.

Thanks a lot for any advice! If any more information is required I’d be happy to provide it.

Logic Analyzer i2c.zip (24.7 KB)

[jhw84 (Joost)]

As an update to the post above: I have made a fix for my specific situation, it might help @yak-54 as well, if he is interested in trying I can provide him with the necessary code changes.
What I’ve done is adding a counter in the I2CDevice class to keep track of consecutive time-outs. If that counter exceeds some value, bus.clear_all() is called to recover from the hung bus by toggling the clock signal and forcing the slave through the transmission process.

I have a question about this, I am calling clear_all() from I2CDevice::_transfer; am I right in assuming that this is always executed off the main thread, and that therefore the blocking nature of clear_all() is of no concern to the actual flight control?

If desired, I can create a PR for this.

[tridge]

Thanks for your analysis of this! I think there is a good chance that this is the same issue that @yak-54 is seeing.
I would like to see your patch, although I think it may be better to sample SDA to check that it is being held low before we apply the hack to clear the device. We also wouldn’t want to use clear_all(), we’d want to use clear_bus(), as otherwise we would be interfering with other buses.

@jhw84

am I right in assuming that this is always executed off the main thread, and that therefore the blocking nature of clear_all() is of no concern to the actual flight control?

once we are fully booted, yes. We do some I2C ops in the main thread when probing for sensors, but not once we’re into the main flight loop. So it is safe, and each bus has its own thread, so it also doesn’t impact on other I2C buses.
I wonder how I can set things up to reproduce this?
Cheers, Tridge

[tridge]

I’ve reproduced the problem by artificially using small timeouts with a MS4525 airspeed sensor. I’m working on a fix using bus resets. It works, but I’m still refining it. I hope to do a PR soon.
Cheers, Tridge

[jhw84 (Joost)]

cool , you’re right that my patch was not ideal; I am still not completely familiar with all parts of the code.

[tridge]

I’ve put a fix for the issue here:

ChibiOS: detect stuck I2C bus and clear on timeouts

It works well for my test case, and I suspect it will fix the issue that @yak-54 has been getting. Note that it only applies the SCL toggle fix if the SDA line is low after a timeout. This means it should not do anything on a “normal” system.
Many thanks to @jhw84 for the suggested approach!
I’ll build a binary for @yak-54 to test as well
Cheers, Tridge

[brad112358]

Is there a git tag or branch for this release?

Thanks!

[tridge]

@brad112358

Is there a git tag or branch for this release?

yes, ArduPlane-3.9.3

[jhw84 (Joost)]

The fix works for me as well. One more question @tridge, while I was debugging this issue I tried to increase the value for the timeout-parameter in i2cMasterReceiveTimeout(), even up to TIME_INFINITE; but this seemed to have no effect at all on the actual timeout-duration. Any idea why? Is this a chibiOS issue? Hardware used is a pixhawk2.

[tridge]

@jhw84

Any idea why?

can you show me the patch where you increased the timeout?

[jhw84 (Joost)]

Nothing fancy, I just changed

        ret = i2cMasterReceiveTimeout(I2CD[bus.busnum].i2c, _address, recv, recv_len, chTimeMS2I(timeout_ms));

to

        ret = i2cMasterReceiveTimeout(I2CD[bus.busnum].i2c, _address, recv, recv_len, chTimeMS2I(10));

for a 10ms timeout, or

        ret = i2cMasterReceiveTimeout(I2CD[bus.busnum].i2c, _address, recv, recv_len, TIME_INFINITE);

for no timeout.

Both do not have the desired effect. Timeout occurs after exactly 3ms in all cases.

[tridge]

@jhw84

Both do not have the desired effect. Timeout occurs after exactly 3ms in all cases.

That’s odd. Did you have zero data to transmit? It uses i2cMasterTransmitTimeout() unless send_len is zero.
The only way I could see this happening is if the I2C peripheral raised an interrupt which triggered an error. You’d need to add some debug code in ChibiOS to find that.

[jhw84 (Joost)]

Yes, zero data; but I did change i2cMasterTransmitTimeout() in the same way to exclude this possibility. I was also thinking that chibiOS is probably not correctly setting the hardware registers for this time-out, or it is interpreting an interrupt wrongly. (Nuttx did not seem to have this issue).

I have a setup using a Arduino Due now which I can use to generate controlled clock stretches; but am a bit constrained on time and am not familiar with the internal structure of chibios. Do you have a suggestion where to look? I did look a bit online last week on the chibiOS forums, and even though I found out there are apparently many many problems with I2C drivers on some STM32s, I did not encounter anyone with this particular problem.

[brad112358]

Thanks! That was the tag I couldn’t find. Ether that tag wasn’t on github when I looked a couple of days ago or I was temporarily blind.

–Brad

[tridge]

@brad112358

Ether that tag wasn’t on github when I looked a couple of days ago or I was temporarily blind

it wasn’t there - I’d forgotten to create it when I did the release, sorry!

[tridge]

@jhw84

I have a setup using a Arduino Due now which I can use to generate controlled clock stretches; but am a bit constrained on time and am not familiar with the internal structure of chibios.

if you can point me at the arduino sketch you’re using I could try it here.

[jhw84 (Joost)]

Yes, I put it here https://github.com/jhw84/i2c-diagnostics

Fair warning: I did some more tests this morning and I don’t find back this 3ms behaviour I described earlier, I may have gotten some versions mixed up last week. It might be not an issue at all.

[thatsnailguy (Travis Bridges)]

Can you update the release-notes.txt when you get the chance.

Also, can you point me to a historical repository for past stable releases? I may just be dumb, at the firmware.ardupilot.org, I seem to be only able to find the latest dev build being saved historically.