When returing from the AUTO mission we are not using QRTL, only manual flying in FBWA and QSTABILIZE mode.
Battery failsafes have caused seriously dangerous situations in past. I will be disappointed if that is still the case. Have not tested yet.
I tested the 4.3.3 firmware on the Pixhawk V6 platform and everything works fine.
I tested the 4.3.3 firmware on the FMU V3 platform, and there was a problem in the automatic landing stage.
It is normal before it is automatically lowered to Q_RTL_ALT. It started to oscillate back and forth around 15 meters, and I don’t know why. I’ve flown a few times with the same problem.
I don’t think it’s a PID problem, I use q_loiter to test the attitude is very stable.
00000003.BIN.param (21.5 KB)
Below is my flight log:https://drive.google.com/file/d/1KB90fNi6mlNRACoefA1HIBeAKGm8SGoG/view?usp=share_link
Tested firmware version 4.2.3 on the same platform today and landed smoothly.
In the 4.3.3 version firmware, the problem of shaking during the automatic landing stage did not appear.
Today I was visited by the Guardian Angel of Radio Control.
https://drive.google.com/file/d/1xqcluucTIJvGcW6Q_NLZ1Qt3op1htz-H/view?usp=sharing
I had been flying for about 10 minutes without any problems. At some point I switched to cruise (all fine) and shortly after I decided to push the throttle to full throttle.
At 14:10 I started to lose altitude: red (barometer) and blue (TECS). The system (TECS) is aware of the desired altitude (green).
The TECS detects the negative vertical speed (red) resulting from the loss of altitude and therefore the demand (purple) is positive.
Despite having the elevator control down (yellow) I have hardly any authority… it continues to descend.
The roll control (red) does have authority (blue).
Despite fully down throttle (yellow) the engine is still at 100% (red)… and going down.
According to the barometer I escaped the disaster by 1.02m and according to TECS by 2.85m.
Here the crash was avoided (14:25) but this is still going on.
20’’ later I was still with the throttle control at zero and the engine at full throttle. To scare me, right?
I decided to switch it to RTL without any appreciable result and at the end I switched it to FBWA; everything was back to normal.
I would be very grateful if you could tell me what could be happening.
I have made a TREMENDOUS MISTAKE:
TECS_PTCH_FF_FF_K=-0.6
when it should have been TECS_PTCH_FF_K,0=-0.06
To devs attention, please review my issue, it could be a very bad tecs malfunction:
Is it possible that the TECS.iph bug is back?
I repeat, there was a wrong setting of TECS_PTCH_FF_K=-0.6 instead of TECS_PTCH_FF_K=-0.06.
@tridge we had a loss of a plane today due to an internal errors 3000. This was a new CubeOrange+, Mauch power supply, 2x iFlight BlHeli ESC’s, SDP3X airspeed sensor, Here3+, and LightWare SF30/C. After the 2nd takeoff and turn to the first waypoint, I looked at Mission Planner and saw INTERNAL ERROR, DISARMED, and (SAFETY) in the air while in auto mode. I had no ability to gain control with mode changes as safety disables our servos in the few seconds before impact. In gathering information, I also noticed POWR.VServo appeared to drop to significantly lower values a few seconds before this catastrophy.
Log here: Microsoft OneDrive - Access files anywhere. Create docs with free Office Online.
0x3000 indicates iomcu reset + iomcu fail, which seems pretty bad
Unsure if it’s related, and I haven’t dug into it, but I’m rather certain that I saw automatic disarms on the ground just a few seconds after arming in FBWA and maybe even Manual mode. I was assuming due to a safety switch (that parameters were set to disllow changes while armed). I’ll search for logs.
Possible link similar issue (Cube Orange) RC Inputs & Outputs freeze (crashlog) · Issue #22405 · ArduPilot/ardupilot · GitHub
@Naterater your IOMCU reset twice in quick succession, almost certainly due to power glitches as indicated by the logged voltage glitches
The rapid double reset is what prevented the code that automatically restores the safety state from working:
notice that time wrapped once to 805ms, then again down to 226ms. That is a rapid double reset. The logic for automatically restoring the safety state on IOMCU reset failed for the 2nd reset. It is not a sequence we’ve seen before
I am looking at the safety restore code to see if we can handle this sort of rapid double reboot of IOMCU
@tridge Help me analyze the reason for the automatic oscillation of the roll and pitch during the automatic landing phase of the Quadplane. Thank you so much!
@Naterater this PR allows us to survive a double IOMCU reset like you had:
it is still not entirely clear what caused your double reset. One possibility is large enough back-EMF on signal lines from actuators (servos, ESCs) causing the IOMCU to reset
If I was you I would not use that particular CubeOrangePlus again, just because it could be a hardware fault, even though there is no strong evidence it is faulty
2 Likes
I’ve just released plane 4.3.4beta1. Randy and I decided to do this release because a number of bugs had been found that we thought should be fixed quickly.
Changes from 4.3.3 are:
- don’t allow RC protocol change on IOMCU once detected
- fixed FBWA pitch limits when in VTOL qassist
- fixed handling of zero compass diagonals
- added an output buffer to MAVCAN
- set emergency status in OpenDroneID on crash or chute deploy
- avoid logging duplicate format messages
- fixed bug in alt error arming check with BARO_FIELD_ELEV set
- fixed handling of double IOMCU reset and safety disable
The two most serious issues are the RC protocol change and the IOMCU double reset.
The RC protocol change was observed on a CUAVv5, and involved re-detecting SBUS as DSM while flying, resulting in incorrect RC input. We had previously disallowed a change of detected protocol for FMU based detections by default, this changes the IOCMU based RC input to do the same.
The double IOMCU reset is the bug found by @Naterater where a power glitch caused the IOMCU to reset twice in quick succession, resulting in the safety state changing to safety enabled. We now ensure that even with multiple IOMCU resets we maintain the safety state.
Please report all testing of this beta!
1 Like
sorry for taking so long to get back to you on this.
The amount of horizontal position oscillation is under 5m in this log, but it is very distinctive:
here it is in map form, with each yellow grid square 1m:
A normal analysis of a log like this would conclude that the most likely cause is an imbalance between the Q_P_POSXY_P gain and the Q_P_VELXY gains. One of the rules with tuning the position controller is to keep the gains in proportion to each other.
The Q_P_VELXY_P, Q_P_VELXY_I and Q_P_VELXY_D gains have all been lowered from the defaults, but the Q_P_POSXY_P gain has not been lowered. This means the position controller is giving a disproportionate degree of control. So lowering Q_P_POSXY_P by 20% should help.
This bit that has me puzzled is this graph:
that is the east component of the XY velocity controller PID. See how jagged it is? For comparison, this is my Striver Mini landing yesterday:
nice and smooth.
I think the explanation is the CPU load. This log shows as being a “PixPilot-V3”, which isn’t a supported board in plane 4.3.3, so I assume you’ve added the hwdef.dat yourself (it would be great if you could open a PR for this to make it officially supported)
From looking at the log I see it is a STM32F427, and I see it has both EKF2 and EKF3 enabled, running all 3 IMUs with both EKFs. That means you are running 6 EKFs on a F427, which is just far too much CPU load.
What seems to be happening is some of the navigation tasks are not being updated on every loop, they are being skipped due to the extreme CPU load. So the position and velocity controllers are running with incorrect time updates.
If you look at PM.Load you’ll see it is often at 1000, which is the “cpu overload” value. Many loops are taking over 6ms, whereas they should be taking 3.3ms for 300Hz.
So I think the real fix for your aircraft is to set EK2_ENABLE=0. You are not using the output of EKF2 anywhere and I suspect it was left enabled by accident after an update from an earlier release.
Can you re-test with EK2_ENABLE=0 and the Q_P_POSXY_P gain lowered to 0.4 and send me a new log?
Thanks for your suggestion, I’ll get back to you after I test it.
Only open EKF3 test and land normally.
https://drive.google.com/file/d/1V5kZ7PqZdRUR-Q6CSdulvWkBhCr1m5jS/view?usp=sharing
1 Like
@tridge
I use ArduPlane4.3.3 firmware. After the M9N GPS is initialized, the configuration cannot be saved, and there is a problem.
I did the following tests:
- After I use the M8N GPS to power on and initialize, the configuration can be saved;
- I changed two M9N GPSs and the same situation occurred;
- I checked the reliability of the wire connection;
Below I provide error screenshots and flight control logs:
https://drive.google.com/file/d/1TyPoMqe9VbtLSC-SCcXMWhddogebS_Ge/view?usp=share_link
I saw a similar problem, not sure if this is the cause.
https://github.com/ArduPilot/ardupilot/issues/15135
What make is the GPS? Did you flash the latest firmware on it? That is the if it is CAN.