Weird compasses hangs and readings on ChibiOs

I have two TS100 mini RadioLink GPS/Compass (QMC5883L) bought at different times. Both should be ok.
And they’re both works good on Nuttx 3.6.1rc2 on my pixracer.
But they both doesn’t work normally nor on first 3.6 versions nor on latest rc2.

So all what I found at the moment (this is only on ChibiOs):

  1. Compass #1 (external) hangs in 90% cases in 5-20 seconds after a battery connection. If the compass is selected as primary then “Bad compass health” error also appears.

  2. Compass #1 never hangs after soft reboot (CTRL+F => reboot pixhawk)

  3. Weird! But opposite to compass #1, compass #3 does hang after a soft reboot!

  4. Compasses #1 (external) and #2 (internal) reading are different after hard reboot and soft reboot. But compass #3 readings are always the same.
    I.e. on every battery connection mx=-207, mx2=330; but after every soft reboot: mx=319, mx2=140
    Opposite to that, mx3 reading always the same regardless of reboots.

  5. This all is repeatable and is not depends on power source. I did tests with battery and USB power.

So, how it looks like: I connect the power => FC boot up and external compass hangs after a few seconds. Reboot via mavlink command => FC boot up and external compass works good this time but internal compass #3 hangs.

Hope these facts could help someone to understand on what’s going on. Because I have no idea what to check further.

Adding tlogs and logs would be a good idea to help to trace the issue

Sorry, forgot to add logs.
Here is one https://www.dropbox.com/s/ha77rvprxn0do1l/00000334_36_rc10_compass_fail.BIN?dl=0

Hi,
so is there any way to debug that further? Maybe see the devices detected on boot, whatever could help to diagnose.
Any advices are welcome!

I found that actually things are different from what I’ve described above.
On power on my external compass is detected by firmware but it stop working after a seconds.
And on reboot the external compass is not detected at all. So internal compass becomes #1 in Mission Planner; that’s why I mistakenly thought reboot does fix the external compass, but it doesn’t.

So… still have no clue what’s wrong.

Is there a list with supported compasses?
There are HMC5983 and HMC5883l in a local store. So I want to buy one and check if these hangs will happen.
But it is not clear from source codes if they are supported.

Well, I bought QMC5883L board. Arducopter reports same dev_id for it as for TS100 - 855305. So it definitely the same compass chip.
And it works absolutely fine both with short and long wires.

So I don’t understand how it could be possible. Maybe some specific combination of unknown issues in TS100 schematic and ChibiOs I2C bus implementation.

Anyone else using Radiolink TS100 gps/compass module?

@tridge is this related?

please test with the latest builds from firmware.ardupilot.org and see if that makes any difference to the results
Cheers, Tridge

@tridge, yes there is a difference
Build 2018-11-15-11:11 - issue exists
Build 2018-11-16-02:11 - no issue
No issue on the latest build as well

@tridge may be you know why i2c bus never got stuck on NuttX but on ChibiOs we need that fix?

@rmackay9 is it worth to backport the fix to 3.6?

Hi,I have been using the version of Copter 3.5-rc7 (25-May-2017), here is the nuttx operating system, but I have encountered a problem that QMC5883L is not updated. At first I thought it was caused by the SDA pulling down and causing IIC stuck.If i2c bus never got stuck on NuttX ?I want to ask your opinion.@sergbokh
@tridge

As per my experience QMC5883 was stuck sometime on nuttx too.
But it were much rare cases - I only remember two cases happened mid-flight.
On ChibiOs instead QMC5883 stucks after a seconds on startup (before Tridge’s fix)

Thank you very much for your reply. I also encountered it during the flight, which is very dangerous. All I did was add 9 flips of the SCL and determine if SDA is 1 when I2C triggered the reset. The problem of this stuck has not been reproduced until now. But it has always been a hidden danger.Do you have a better solution in nuttx ?

I did a custom fix to detect stuck compass and to set it’s healthy = false. So arducopter could switch to another one compass. But I didn’t tested that b/c I just migrated to 3.6 Chibios.

It is a brilliant idea,but unluckily I only have a one useful compass :rofl:I will try to fix the another compass,I can"t promise is"s useful.thanks again

Hi guys, I am experiencing a similar problem on 3.6.7. Both Chibios and Nuttx.

I have disabled all compasses except the QMC using COMPASS_TYPEMASK, Arducopter is only seeing one compass with DEV_ID of 855305. The problem is that I am unable to calibrate it - no datapoints are coming from the compass. The internal compass of PixhawkLite is fine (when enabled).

I have tried calibrating the QMC compass with Qgroundcontrol and with both methods available in Mission Planner. Nothing is happening, the bars are not moving. I have also tried upgrading from Chibios 3.6.6 to 3.6.7 and flashing Nuttx 3.6.7. No joy.

Regards,
Sergey

@macrokernel
do you see mx, my, mz values in MS status tab?
Does they change some time after boot or they’re always zeroes?

Hi, after the boot they are -14, 10, 9 and are never changed even after power cycling.

I have tried flashing master (3.7.0-dev) - same behavior.

Perhaps something is wrong with my PixHawk Lite? Are 3.3 Volts normal on SCL and SDA pins?

Yes, I2C is 3.3v
On 3.5.x NuttX the TS100 worked well for me. At least it never stuck right on bootup. Also see @xiefengxiyu picture above - there is a data coming and minutes after it stops.
So maybe this is another issue. Do you have long wires? Telemetry/HD camera close to compass wires?