Failure of GPS Auto-Switch on Pixhawk Cube Orange with Here 3 GPS Units

I have a question regarding my setup with a Pixhawk Cube Orange and two Here 3 GPS units. as I think, the system should automatically switch to another GPS unit when one fails, as I have set the “GPS_AUTO_SWITCH” parameter to 1 (USE_BEST). However, during today’s flight, one of the GPS connections fell off the board, due to high vibrations, and the system did not switch to the backup GPS automatically. As a result, the EKF failsafe was activated and I had to return my copter using altitude hold mode.

I suspect that the issue may be related to my GPS setup, where I connected GPS1 to CAN1 and GPS2 to CAN2 without using an I2C Splitter. I would appreciate any insight or analysis on this situation and any suggestions for preventing similar issues in the future.

this is my log and parameter files :
https://drive.google.com/file/d/13FJjAD437mOUetvwZ6sgXTrDV1RwyG8v/view?usp=share_link
20230307.param (19.1 KB)

Thank you in advance.

Im using this extension board to connect the CAN GPS :

Unfortunately, one of the connection ports fell off due to intense vibration.

After conducting several experiments, I have discovered that the CAN GPS order is determined by the number of CAN ID, and that changing the CAN ID of the GPS can change the GPS order. Specifically, if the CAN ID is low, it will be selected as the primary GPS.

I have observed that when I remove the primary GPS, both GPS 1 and GPS 2 letter colors become red, even though GPS 2 has a 3D fix. I have attached an example of this in the form of a mission planner GPS letter color chart:

GPS 12 EXAMPLE 1

However, if I remove the secondary GPS, the GPS 1 letter color remains white and the GPS 2 letter color becomes red. I have attached another example of the mission planner GPS letter colors to illustrate this:

GPS 12 EXAMPLE 2

Based on these experiments, I suspect that removing the primary GPS may prevent the copter from switching to loiter mode.

I would appreciate any expert opinions on the matter, as I also believe that the EKF failsafe has a higher priority over GPS switching. Thank you in advance for your insights.

I would like to know if there is an option to switch between GPS units using an RC switch.

I have searched through the RCX_OPTION parameter, but have not found this function.

However, I have found the “scripting” and “user_function” options.
Can these be used to switch between GPS modules using RC input?
If not, I would like ArduPilot to consider adding this function in the future.

This is the log message for “MSG”.

When the physical connection is lost, the ArduPilot firmware automatically searches for the CAN baud rate instead of switching to a GPS with better communication. This can result in an unrecoverable EKF failsafe and the vehicle may not be able to return to loiter mode.

Can you try
BRD_BOOT_DELAY,5000
EK2_ENABLE,0
Also can you try a GPS in each CAN port instead of both on the same CAN bus with a splitter.

I have a quite large copter, so I have used an i2c/CAN splitter for extension, but it was not for CAN splitting. Instead, I am planning to create a very long wire to avoid using the CAN splitter extension that fell off during flight.

I appreciate your help Shawn, but I just wanted to clarify that CAN1 is already in use for GPS1 and CAN2 for GPS2, as I mentioned earlier.

Thank you for suggesting turning off EKF2. I’ve been considering turning it off for a few weeks now. However, since the copter was not built by me, I’m gathering information on the reasons and circumstances for turning on EKF2. after I gather all information, I will turn off it.

I’m planning to conduct an experiment by turning off GPS in the air using an Arduino Nano and then compare the results with EKF2 and EKF3. If I proceed with this experiment, I’ll share my experience on the Ardupilot forum.

and could I ask the reason of setting BRD_BOOT_DELAY,5000?

I heard that setting is for recognization of each CAN GPS.

I have created an Arduino device that can turn off the compass while copter is in the air. However, the auto-switching worked perfectly, with same environment as previous. I am unsure why the disconnection of the CAN caused the EKF failsafe to trigger. I intended to analyze the log data soon, more deeply.


I found the EKF variance message in my system, but I can’t determine which sensor caused it.

Despite trying different settings such as connecting CAN1 and CAN2 GPS separately, turning EKF2 on and off, adjusting BRD_DELAY_SETTING, and switching primary and secondary GPS, GPS switching worked properly without any EKF failsafe. I turned off GPS by arduino device.

so, I still don’t get any clue about why the EKF failsafe occurred when the GPS was disconnected. Can anyone provide insights?

It’s a very mysterious issue.

Despite testing many parameters and situations, I haven’t been able to replicate the EKF error that occurs when physically disconnecting the GPS while in the air.
changing GPS order, changing EKF2 parameters, BRD_DELAY parameters didn’t cause EKF error.

Currently, my Arduino device only disconnects the H line on the CAN GPS, in the air. I’ll now attempt to disconnect just the VCC on the CAN GPS and see if that makes a difference.

I believe the GPS failover should work as you expect in earlier posts.
It would be interesting to hear from others that might have tested this.

The EKF2 can definitely be disabled since it is completely superseded by EKF3.
EKF2 could be in place if the copter was upgraded through older versions of firmware and someone eventually enabled EKF3 but forgot (or was reluctant) to disable EKF2. It really doesnt add anything.

Have you tried
GPS_AUTO_SWITCH,4

I just turned off my EKF 3. The GPS_AUTO_SWITCH has been set to 1 since the copter was built.

I want to determine the cause of this accident and modify the GPS_AUTO_SWITCH configuration accordingly. the cause of this accident is mystery now. Thank you.

Hi
Truth_T

Currently,I am working on ardupilot dual GPS configuration ,so regarding the configuration ,I have try to built two Neo3 GPS connected at single CAN bus splitter ,but still it would be not working .

I want, supports from ardu community please help me to resolve that kind of issues.

Sorry as I have been on south pacific 2 months, I cant help you.

Please see mavlink CAN and ardupilot CAN GPS docs.