I have made a TREMENDOUS MISTAKE:
TECS_PTCH_FF_FF_K=-0.6
when it should have been TECS_PTCH_FF_K,0=-0.06
To devs attention, please review my issue, it could be a very bad tecs malfunction:
Is it possible that the TECS.iph bug is back?
I repeat, there was a wrong setting of TECS_PTCH_FF_K=-0.6 instead of TECS_PTCH_FF_K=-0.06.
@tridge we had a loss of a plane today due to an internal errors 3000. This was a new CubeOrange+, Mauch power supply, 2x iFlight BlHeli ESC’s, SDP3X airspeed sensor, Here3+, and LightWare SF30/C. After the 2nd takeoff and turn to the first waypoint, I looked at Mission Planner and saw INTERNAL ERROR, DISARMED, and (SAFETY) in the air while in auto mode. I had no ability to gain control with mode changes as safety disables our servos in the few seconds before impact. In gathering information, I also noticed POWR.VServo appeared to drop to significantly lower values a few seconds before this catastrophy.
Log here: Microsoft OneDrive - Access files anywhere. Create docs with free Office Online.
0x3000 indicates iomcu reset + iomcu fail, which seems pretty bad
Unsure if it’s related, and I haven’t dug into it, but I’m rather certain that I saw automatic disarms on the ground just a few seconds after arming in FBWA and maybe even Manual mode. I was assuming due to a safety switch (that parameters were set to disllow changes while armed). I’ll search for logs.
Possible link similar issue (Cube Orange) RC Inputs & Outputs freeze (crashlog) · Issue #22405 · ArduPilot/ardupilot · GitHub
@Naterater your IOMCU reset twice in quick succession, almost certainly due to power glitches as indicated by the logged voltage glitches
The rapid double reset is what prevented the code that automatically restores the safety state from working:
notice that time wrapped once to 805ms, then again down to 226ms. That is a rapid double reset. The logic for automatically restoring the safety state on IOMCU reset failed for the 2nd reset. It is not a sequence we’ve seen before
I am looking at the safety restore code to see if we can handle this sort of rapid double reboot of IOMCU
@tridge Help me analyze the reason for the automatic oscillation of the roll and pitch during the automatic landing phase of the Quadplane. Thank you so much!
@Naterater this PR allows us to survive a double IOMCU reset like you had:
it is still not entirely clear what caused your double reset. One possibility is large enough back-EMF on signal lines from actuators (servos, ESCs) causing the IOMCU to reset
If I was you I would not use that particular CubeOrangePlus again, just because it could be a hardware fault, even though there is no strong evidence it is faulty
2 Likes
I’ve just released plane 4.3.4beta1. Randy and I decided to do this release because a number of bugs had been found that we thought should be fixed quickly.
Changes from 4.3.3 are:
- don’t allow RC protocol change on IOMCU once detected
- fixed FBWA pitch limits when in VTOL qassist
- fixed handling of zero compass diagonals
- added an output buffer to MAVCAN
- set emergency status in OpenDroneID on crash or chute deploy
- avoid logging duplicate format messages
- fixed bug in alt error arming check with BARO_FIELD_ELEV set
- fixed handling of double IOMCU reset and safety disable
The two most serious issues are the RC protocol change and the IOMCU double reset.
The RC protocol change was observed on a CUAVv5, and involved re-detecting SBUS as DSM while flying, resulting in incorrect RC input. We had previously disallowed a change of detected protocol for FMU based detections by default, this changes the IOCMU based RC input to do the same.
The double IOMCU reset is the bug found by @Naterater where a power glitch caused the IOMCU to reset twice in quick succession, resulting in the safety state changing to safety enabled. We now ensure that even with multiple IOMCU resets we maintain the safety state.
Please report all testing of this beta!
1 Like
sorry for taking so long to get back to you on this.
The amount of horizontal position oscillation is under 5m in this log, but it is very distinctive:
here it is in map form, with each yellow grid square 1m:
A normal analysis of a log like this would conclude that the most likely cause is an imbalance between the Q_P_POSXY_P gain and the Q_P_VELXY gains. One of the rules with tuning the position controller is to keep the gains in proportion to each other.
The Q_P_VELXY_P, Q_P_VELXY_I and Q_P_VELXY_D gains have all been lowered from the defaults, but the Q_P_POSXY_P gain has not been lowered. This means the position controller is giving a disproportionate degree of control. So lowering Q_P_POSXY_P by 20% should help.
This bit that has me puzzled is this graph:
that is the east component of the XY velocity controller PID. See how jagged it is? For comparison, this is my Striver Mini landing yesterday:
nice and smooth.
I think the explanation is the CPU load. This log shows as being a “PixPilot-V3”, which isn’t a supported board in plane 4.3.3, so I assume you’ve added the hwdef.dat yourself (it would be great if you could open a PR for this to make it officially supported)
From looking at the log I see it is a STM32F427, and I see it has both EKF2 and EKF3 enabled, running all 3 IMUs with both EKFs. That means you are running 6 EKFs on a F427, which is just far too much CPU load.
What seems to be happening is some of the navigation tasks are not being updated on every loop, they are being skipped due to the extreme CPU load. So the position and velocity controllers are running with incorrect time updates.
If you look at PM.Load you’ll see it is often at 1000, which is the “cpu overload” value. Many loops are taking over 6ms, whereas they should be taking 3.3ms for 300Hz.
So I think the real fix for your aircraft is to set EK2_ENABLE=0. You are not using the output of EKF2 anywhere and I suspect it was left enabled by accident after an update from an earlier release.
Can you re-test with EK2_ENABLE=0 and the Q_P_POSXY_P gain lowered to 0.4 and send me a new log?
Thanks for your suggestion, I’ll get back to you after I test it.
Only open EKF3 test and land normally.
https://drive.google.com/file/d/1V5kZ7PqZdRUR-Q6CSdulvWkBhCr1m5jS/view?usp=sharing
1 Like
@tridge
I use ArduPlane4.3.3 firmware. After the M9N GPS is initialized, the configuration cannot be saved, and there is a problem.
I did the following tests:
- After I use the M8N GPS to power on and initialize, the configuration can be saved;
- I changed two M9N GPSs and the same situation occurred;
- I checked the reliability of the wire connection;
Below I provide error screenshots and flight control logs:
https://drive.google.com/file/d/1TyPoMqe9VbtLSC-SCcXMWhddogebS_Ge/view?usp=share_link
I saw a similar problem, not sure if this is the cause.
https://github.com/ArduPilot/ardupilot/issues/15135
What make is the GPS? Did you flash the latest firmware on it? That is the if it is CAN.
@pompecukor
It is the commonly used M9N module, which uses serial port link, not CAN. I’m using the default firmware version 4.04, which is the latest version as far as I know.
With 4.3.2, 4.3.3 and also with 4.3.4beta1 there is a problem with the F.Port connection of a receiver (FrSky RX6R) to a Matek F765 wing. Configuration as specified in the wiki.
The RC connection as well as telemetry works when the FC is powered. After about 10-15 seconds the FC then loses contact with the receiver.
If only the FC is restarted after that, no connection will be established. Only when the supply of the FC is interrupted and restored, the receiver is recognized again briefly.
I’ve just released 4.3.4beta2. This adds two key changes on top of beta1:
- fixed a home reset issue that can affect vehicles with bad GPS quality just before takeoff resulting in takeoff to the wrong height. Thanks to ARACE for reporting this issue
- fixed a GPS configuration arming error with non-M10 GPS modules, causing spurious unconfigured GPS errors
Thanks to everyone who has been testing!
2 Likes