I have managed to reproduce the WDG on this one particular setup not sure why or how. the brains are a pixracer from mrobotics with crossfire running the latest version of RX/TX firmware from TBS 6.19. When I power this one up without my TX module on everything works fine as you can see from this tlog
2023-04-30 05-36-59.tlog (126.0 KB).
But when I power my TX module then plug the FC in via USB this is where things go crazy as seen from this tlog
2023-04-30 05-34-16.tlog (270.7 KB).
I have logging disarm turned on but it does not work while all this WDG is happening. Which oddly enough I have 2 other quads with exact same brains and they have not done this ever. Attched below is a parameter file for the problem child. If it is powered via a battery then hooked to USB I do not get any issues connects all good but do get failed to initialize mavftp.
210_pixracer_WDG.param (17.3 KB)
Ok, watchdogs are serious. It’s great that it can be reproduced and that you’ve provided logs though. I’ve added to our 4.4 issues list.
P.S. if you have an onboard .bin log that would be great too but that is probably not a blocker for us investigating this.
@406FPV many thanks for reporting this watchdog. Your crash_dump.bin allowed us to find the problem and we have done a PR to fix it here:
the problem was you ran out of memory and were unable to allocate both EKF3 cores. We were missing a check in logging for the allocation failure, so it tried to log EKF3 data from a NULL pointer.
@rmackay9 now we need to work out if 4.4 is using an unreasonable amount of memory
@406FPV please test this fixed firmware:
http://uav.tridgell.net/tmp/copter-pixracer-bdshot-4.4-beta1-fixed-watchdog.apj
you will need to reduce memory usage to make EKF3 work
Hello,
I’ve given the new version a try on two X8 copters with Cuav Nora (non-plus) FC. One is flying well, another one had bad problems with maintaining altitude in the Loiter mode. I’ve noticed that out of three barometers, the values reported by the one nr 0 are different from the other ones. Also, the fused (reported) altitude value is sometimes well below 0.
@tridge
I am not sure what’s going on with the EKF3 not enough memory. I have not to my knowledge changed anything among all three of these that I can think of that would cause this as the other 2 have not exhibited this same behavior. Can you point me in the right direction to some parameters that would cause not enough memory. I have loaded that firmware and no longer get the WDG. But still cannot find what would be causing the ekf3 not enough mem. I have attached some logs but logging still seems to be funky with dates of 1969 and 1979.
log 1
log 2
Edit: duh logs will have odd dates when there is no GPS fix hahaha not sure why this never dawned on me!
The allocation of cores is not deterministic but somewhat timing related, so it could just be that this board is unlucky because the EKF gets allocated later than other things that will disable themselves if they can’t get the memory they need. I actually made some changes to fix this ages ago, but actually the proper solution is to make sure that all boards have appropriate free memory for the features they enable. I suspect something is taking a little more memory in 4.4 and this is what has pushed things over the edge.
Thanks very much for testing 4.4 and providing feedback.
Any chance you could post a .bin log? It could be an sensor issue but it could also be a control or motor output issue so a log will help figure out where the problem is. Thanks again.
I’ve thought I’ve attached them, trying once more. I was on a bad quality gprs connection.
After changing the main baro to 1 all was fine again.
Hi people.
Is Custom build server ok or in service?
I make build but folder with SW give me Error 404. Same problem with Beta SW custom build server
Also I ask to unlock Wind Estimation for small boards like Matek-405
Thanks.
Thanks again for the report. We’ve recreated a problem and @shiv-tyagi has a fix coming. My guess is it will take another 24hrs for that to get out to production but let’s see.
EDIT: @shiv-tyagi has apparently fixed the beta custom build server now so maybe give it a try?
I was having some GPS issues yesterday with bad flights. I updated to master and it seems have gone. The GPS is very close to the FC so some issues would seem normal.
THKS!
Scrolling EKF3 code in Visuall Studio I found that wind estimation have weight in EKF calculation.
Also now I know how to configure GNSS module to send right data.
Any idea when KDECAN functionality will be added back in?
Fixed already, perhaps? KDECAN doesn't appear to be working · Issue #22952 · ArduPilot/ardupilot · GitHub
today I encountered another WDG on another pixracer frame. the logs from it flying pre-wdg are here Copter-4.4.0-beta1 available for beta testing - #16 by 406FPV they are labled 6in in all those logs. I set log disarmed to 1 but have been unable to capture anything and the crash_dump on mavftp errors out when using the latest beta MP. I can consistently get PreArm: Internal errors 0x8000 l:442 main_loop_stk and was able to get it to reproduce the wdg 3 times now but cannot get it to log anything. using sandisk 16 gb cards so not a quality issue with the sd. kind of at a loss here updated to 4.5 dev and errors persist
5/26/2023 19:01:46 : C246 IE32768 IEC15 TN:
5/26/2023 19:01:46 : WDG: T46 SL870 FL0 FT0 FA0 FTP0 FLR0 FICSR0 MM76 M
5/26/2023 19:01:38 : Frame: QUAD/X
5/26/2023 19:01:38 : IMU1: fast sampling enabled 8.0kHz/1.0kHz
5/26/2023 19:01:38 : IMU0: fast sampling enabled 8.0kHz/1.0kHz
5/26/2023 19:01:38 : RCOut: DS300:1-4 PWM:5-6
5/26/2023 19:01:38 : Pixracer-bdshot 002F002E 32375112 38343932
5/26/2023 19:01:38 : ChibiOS: 1ec9f168
5/26/2023 19:01:38 : ArduCopter V4.4.0-beta1 (867ff238)
5/26/2023 19:01:38 : Frame: QUAD/X
5/26/2023 19:01:38 : IMU1: fast sampling enabled 8.0kHz/1.0kHz
5/26/2023 19:01:38 : IMU0: fast sampling enabled 8.0kHz/1.0kHz
5/26/2023 19:01:38 : RCOut: DS300:1-4 PWM:5-6
5/26/2023 19:01:38 : Pixracer-bdshot 002F002E 32375112 38343932
5/26/2023 19:01:38 : ChibiOS: 1ec9f168
5/26/2023 19:01:38 : ArduCopter V4.4.0-beta1 (867ff238)
5/26/2023 19:01:38 : Frame: QUAD/X
5/26/2023 19:01:38 : IMU1: fast sampling enabled 8.0kHz/1.0kHz
5/26/2023 19:01:38 : IMU0: fast sampling enabled 8.0kHz/1.0kHz
5/26/2023 19:01:38 : RCOut: DS300:1-4 PWM:5-6
5/26/2023 19:01:38 : Pixracer-bdshot 002F002E 32375112 38343932
5/26/2023 19:01:38 : ChibiOS: 1ec9f168
5/26/2023 19:01:38 : ArduCopter V4.4.0-beta1 (867ff238)
5/26/2023 19:01:36 : C246 IE32768 IEC15 TN:
5/26/2023 19:01:36 : WDG: T46 SL870 FL0 FT0 FA0 FTP0 FLR0 FICSR0 MM76 M
2023-05-26 19-08-03.tlog (319.5 KB)
can anyone also explain this tone as this is how all this started today with a very weird boot up sequence where it was taking almost 2 minutes to boot had to go teach class and came back to investigate further to find that WDG. The only thing that was done to that quad after the logs posted above was I flew it a cpl more times to get some logs for magfit updated all that and flew again without any issues. drove about 600 miles back home then put it in a case and flew to Redmond didn’t get a chance to fly there and then drove to Portland to teach where I got a chance to fly and encountered all these issues. this actually was not the quad that was having the earlier wdg issues but was one that I had flown without issue multiple times.
20230526_065436.mp4 - Google Drive
I was told roughly a year ago that 8kHz of fast sampling for both IMUs is maybe too much for a F427-based Pixracer. Especially if you augment that by any kind of FFT-based filtering, which the bdshot version suggests.
I guess the issues in this case are timing-related and can really be transient, e.g. the CPU flies well in the first flight and then goes wild in the second flight, or it works normally with one firmware version and starts crashing with the next one. Something like that happened to me already in the past, precisely with Pixracers.
I have a sneaking suspicion that I may be pushing things but I do not have the fft based filtering just rpm and a simple m8n gps with a crossfire. I did not have the rc module fired up last time and I have seen some issues with crsf and rc a cpl weeks ago. The firmware is defaulted or at least to my knowledge is for that sampling at least I do not remember changing it.