Not sure where to post this so just let me know if there is a better thread elsewhere. Using Copter 3.6dev with UAVCAN compass and GPS module. I can see the data coming in on the status screen of Mission Planner so I know that the compass is working but the Mission Planner compass calibration does not even see the external UAVCAN compass or the internal either. This is Arducopter v3 on a Solo. So I had two Pixhawk cubes and found one worked and one did not. I diff’ed the parameter files for each and found that the one that did not work had EKF2 enabled and the one that calibrated fine had EKF2=0. One of the versions of 3.5 that I was testing must have changed the default to EKF2 enabled because I did not set it myself and when I loaded 3.6dev it did not reset the parameters.
I read that EKF2 is now default enabled since some version of 3.5.x but with it enabled I can not use Mission Planner to calibrate the mag. If I turn it off the mag cal works normally.
Mike,
It might just be a memory thing. So if EKF2 and EKF3 are both enabled at the same time, there might not be enough memory to do the compass calibration…? I think it’s best to make sure only EK2_ENABLED or EK3_ENABLED is 1.
Thank very much for your reply.
I have been hoping that it was something stupid like that. But researching old threads like Matts where he reports a similar problem with both enabled, I checked and I don’t have EKF3 enabled. I also checked to make sure that the P2.0 did not have the STM defect, not that it would stop it from using the whole 2mb of ram but it is a v3 STM so not an issue. In Matt’s thread there is a hint that the Solo used more memory that a V4 platform but that thread never got solved.
It is a catch 22 problem. If I turn off EKF2 and set AHRS_type=1 or 0 (it seems to ignore 1) I can cal the mag and the accel but it won’t arm because I am using 3.6dev and it wants EKF2 enabled. If I enable EKF2 it won’t cal so it won’t arm.
I am really a glutten for punishment so I’ll swap a green cube in and see if the problem persists and then I’ll erase all the parameter memory and make sure there is not a corrupted value some where.
I am using OlliW betacopter based on 3.6dev to get the UAVCAN additions but I also installed current master and it does the same thing.
Again I appreciate your time taking a stab at it and if you think of any thing else I’ll give it a try. I am so close to building a full UAVCAN machine.
I have tested three Pix2.0 cubes and a Green Cube with the same results. With 3.6dev and UAVCAN enabled with a UAVCAN GPS/MAG and EKF2 or EKF3 enabled the mag calibration in Mission Planner does not even start to work. If EKF2 is disabled the mag cal starts to work.
If I install 3.6dev in a PX4.V2 with the same setup, plugged into the same UAVCAN bus on the Solo. it works with EKF2 enabled. So it appears to be something with the PX4.V3 firmware or the P2.0 Solo FC or a bad Solo main brd but it is brand new.
I would also be good to post a log. A tlog from the MP when the calibrate failure happens and/or a dataflash log of the vehicle from a test when it’s failed the compass calibration so we can look at the parameters. For the dataflash log, probably best to set LOG_DISARMED to 1.
Thanks!
Will do.
First I’ll put on an I2C compass, with UAVCAN enabled and see if it appears related specifically to a UAVCAN compass, then turn off UAVCAN all together and see if with UAVCAN disabled if the I2C compass works.
OK this is what I found.
If I put a I2C Here compass on the Solo with 3.6dev, Pix2.0 and CANbus off. It works fine with EKF2 on.
However as soon as I enable UAVCAN D1 the I2C compass fails. CANbus off I2C compass ok with EKF2 on but with CANbus on the I2C compass acts just like the UAVCAN compass, turn EKF2 off and it starts to work.
OK spent 8 hrs today trying to get the logs.
Install 3.6dev.V3 downloaded from firmware.ardupilot.org dated 11-08-17 23882aaa
Loaded the solo param file.
UAVCAN d1 off d2 off ekf2 on
I2C Here compass will cal
UAVCAN d1 on, ekf2 turns off by itself on reboot. Can’t turn it back on. 4 12-31-1999 6-09-00 PM.bin (556 KB)
looks like most of the logs are too big to upload here. when i try to upload it just loops during the upload.
Tried to to the same test on a Green Cube P2.1 and it went nuts with 3.6dev. Took me two hours just to get it back to 3.5.3 and it is still not right.
I need a way to completely erase parameter memory, loading tracker does not do it. 2017-11-10 14-42-55.tlog (79.5 KB)
I cleaned out all the logs before each test but without gps lock the time-date stamps are not right and the .bin and the tlog files don’t have matching time-date stamps. So I am actually not sure these are from the same test.
I had a hard time getting the tests to be consistent and repeatable. Each test something would be different or I would have to reboot to get things to work right. Very very frustrating. Sometimes when I tried to get the dataflash logs Mission Planner would tell me there were no logs and then at another time they would be there. I changed out all the SD cards in the P2.0’s with newer newly formated cards.
Since the tlogs are small I’ll upload the lot of them in case one has the needed data. 2017-11-10 14-40-27.tlog (263.7 KB) 2017-11-10 14-38-07.tlog (215.0 KB) 2017-11-10 14-35-04.tlog (316.5 KB)
Looking in log 14-40-27 I see:
NavEKF2: not enough memory
tlog from the green cube where it runs out of ekf2 memory 2017-11-10 14-53-00.tlog (334.0 KB)
Hey Mike
I just show up here to tell that I think that your efforts are great.
I understand (from my own experience with the current UAVCAN implementation), that things can be frustrating since one can spend many hours on troubleshooting, but you are a pioneer here.
I actually find it surprising, unexpected, that v3 is so different from v2, v4.
cheers, Olli
The V3 has extra code for the Solo that might be causing memory problems.My strange trouble shooting day yesterday could be explained by conditions changing constantly due to the order I did things which caused the fc to run out of memory at different times. I don’t know enough about it. But I need to get more methodical. When I start from scratch and load firmware it is easy to miss one parameter change and the whole test is invalid. So I need to get my log book out and do it like I was working for a living again.
I really think CANbus is an important feature to allow Ardupilot builds to be easier. There are so many different connectors for every Pixhawk variation out there and different pinouts for every connector. A CANbus network would eliminate all that making easier for beginning DIY’ers to buy a GPS and not spend two weeks trying to splice up a cable for it to work. It would also improve the quality of a build by eliminating long home-run wiring for cleaner less noisy builds.
Back to work!
THe previous days troubelshoot ended up corrupting the IM6 brdon the Solo. Reinstalled MAtt’s Open Solo IMX6 image which restored the system to baseline.
Today’s session
11/11/17 Mission Planner 1.3.50 PIxhawk 2.0 Solo cube
Load Arducopter 3.6dev V3 build 23882aaa
No UAVCAN GPS/MAG
I2C HERE GPS/MAG
CANbus unpowered and dis-connected
UAVCAN disabled
EKF2=enabled
Logging replay enabled
Compass Cal =ok
Tlog 16-38-00 2017-11-11 16-38-00.tlog (254.7 KB)
cal compass=ok
reboot
Parameters hang at loading Stat_runtime
cancel parameter load
set Can_D1_UC_ESC_BM=0
set Can_D1_UC_SRV_BMC=0
Plug-in power up canbus
No UAVCAN GPS/MAG, Orel UAVCAN ESC’s installed on bus
tlog 16-42-16 2017-11-11 16-42-16.zip (454.4 KB)
Solo controlloer will not connect with IMX6 on Solo
Power down CANbus
reboot
Solo will not connect with controller
UNplug CAnbus
Reboot
Solo will noit connect with controller
Plug cube USB
connect via USB
Mission Planner hangs
KIll MIssion PLanner
restart MP
connect
connect failed
Unlpug cube
plug cube in restart fc
connect
never connects
Install new firmware, arducopter 3.5.3
complete install
connect
starts to connect, parameter download stops abruptly
Not conencted!
Reconnect
Mavlink connecting never starts downloading parameters
cancel
will not cancel, kill MP
Try to install 3.6dev, install says already on brd
Try to install 3.5.3, install says already on brd
can’t get the dataflash logs
Enough fun for one day!
11-12-17
Install Arduplanev3.8.2 in hopes of clearing out cube
Installing but only one tone DUH duh after complete.
Connected ok
disconnect
Install arducopter 3.5.3
plug in battery during install
successful tone sequence
connects and parameters load quickly
disconnect
unplug usb
attempt to connect to controller
power up Solo with battery
connected to Solo controller successfully.
connect to mp udp
parametrs load completely
disconnect
load firmware arducopter 3.6dev 23882aaa via wifi
successful tone seq.
connect MP via udp
successful load of parameters
load Solo 3.5.3 params
plug-in power on CANbus
GPS/MAG installed and OrelESC’s
reboot
connects to controller, to MP via udp
set
CAN_D1_UC_ESC_BM=0
CAN_D1_UC_SRV_BM=0
check compass set to and detected UAVCAN GPS/MAG=true
set
GPS_TYPE=9
log_disarm=1
Log_replay=1
write parameters
tlog:18-41-19 2017-11-11 18-41-19.tlog (315.1 KB)
reboot
connects to controller
connect to MP udp
loads all parameters quickly
rename old MP log directory.
compass cal-succesfull UAVCAN GPS/MAG
reboot
check to see if EKF2=1
EKF2=0
SET
EKF2=1
write
reboot
connected
get dataflash logs - reports no logs to download.
check ekf2=1 true
test compass cal - fails
set
ekf2=0
write
reboot
connected
check compass detected=true
check EKF2=0 true
check dataflash logs-23 logs available -download logs
logs are numbered 1-23 in the dataflash download list but the time stamps are
not consistant. log 1 is 12/31/1999 6:37:34pm
log 23 is 12/31/1999 6:22:22pm with higher and lower time-stamps in between 23 12-31-1999 6-22-22 PM.zip (573.8 KB)
I assume log 23 is the last one?
clear dataflash logs
OK UAVCAN MAg detected as #2, internal off
EKF2=0
Start UAVCAN GUI bus capture UAVCANbus_ekf=0.zip (159.0 KB)
Finally a repeatable test with hopefully matching logs!
Unfortunately this forum can not handle most of the logs either too big or fails to upload or loops uploading forever. I do have all the logs.
In summary:
With Arducopter 3.6dev compiled for the V3 platform. Pixhawk 2.0 in Solo, with UAVCAN enabled and EKF2 enabled then
Compass Calibrate in Mission Planner does not function. Not a matter of not getting a good calibration, it does not work at all. CANbus compass does work, just can’t be calibrated.
Compass calibration starts working when EKF2 is disabled.
Dataflash logs are not found by Mission Planner, when EKF2 disabled logs are available to download.
Can’t disable ARMING checks. or does not ignore ARMING checks when they are disabled.
With I2C compass installed just turning on UAVCAN will cause calibration failures.
I checked QGC vs Mission Planner and it also fails to cal the compass with EKF2& CANbus enabled but yields the error message mavlink_cmd(42424) fails.
I think Mike will tell you that it’s working fine for a v2 build (he got a batch for being the first UC4H user ;)). I myself can report that I did not observe that for neither v2 nor v4.
I had issues though with getting 3 compasses calibrated, one internal, one external at I2C, and one external on uavcan. Can’t judge if that is related to the above or not.
I’d like to seize the opportunity to add that there are also issues with the startup of gpses (document in one of my other threads). I think this has found its way into the github issue list, and it appears it is known what the problem is. Would be good to have it solved though. Also, in a dual GPS setup not all combinations of GPS settings, which one would think should work, do work; only few did (document ion one of my other threads).
Yes James this does not manifest itself with a FMUV2 build. Some of the errors appear to indicate a EKF2 core is running out of memory. I don’t pretend to know Arducopter well enough but I believe the FMUV3 firmware has some stuff in there for Solo that might take up more memory than a FMUV2. I also had a lot of very weird things happen when I tried to use a Pix2.1 green cube. It did not seem to like 3.6dev at all, at least with CANbus enabled. Thanks for your interest. I still would like to get my all CANbus Solo flying!
I was able to get it working. I was testing the setup of a new hex CANbus aircraft and I found it did not work even with a FMUV2 flight controller that I got to work before. So in troubleshooting that problem I erased all parameter memory and reloaded the firmware. Then I set the CANbus settings but I did not load the Solo 3.5.3 parameters. I found it now worked. So I did the same with the frankensolo, erased the parameter memory, reloaded 3.6dev FMUV3, set the required parameters one by one for CANbus and the compass now calibrates with EKF2 enabled. I am going through the Solo_3.5.3_parameter file entry by entry now to try and figure out what parameter is causing the problem. I just need to button it up and maiden my all CANbus Solo.
