Copter-4.0.0 released!

Hi,
Regarding the PreArm: Internal errors (0x400) - had a similar setup - 2 X UAVCAN GPS (mRo Location One).
attaching a log & a tlog - used GPS_AUTO_SWITCH = 2 (Blend)
Seems that the error comes up when the 2nd GPS gets a 3D FIX

Adding some more info:
When testing with one Here2 module and one mRo Location one the PreArm doesn’t come up.
When changing GPS_AUTO_SWITCH = 1 (UseBest) the PreArm doesn’t come up as well.

1 Like

@RomanD,

Looks like the file requires permissions to access, can you grant open access if possible?

@rmackay9
Done, changed the permissions as well.
Sorry for that.

1 Like

@RomanD,

OK, thanks for that. So from a quick look at the logs it seems like both GPSs are UAVCAN GPSs and as you say there is this “Internal Error 0x400” which is for “constrained NaN”. I’ve added this to our Copter-4.0 issues list so we don’t forget about it. I’ll see if I can recreate it with a couple of Here CAN GPSs.

2 Likes

In our setup it could not be recreated with Here CAN GPS. Even with one Here and one LocationOne. Only happened when both GPSs were LocationOne.

Is the “internal error” sticky or is it cleared periodically ?

the only line of code I could find in GPS blending code that uses constrain operation is this:
float min_alpha = constrain_float(_omega_lpf * 0.001f * (float)(state[i].last_gps_time_ms - _last_time_updated[i]), 0.0f, 1.0f);
No clear how it could ever get a NaN parameter …

@VadimZ, @RomanD (I guess you guys work together or just coincidentally have the same issue?)

I had a chat with @pkocmoud from mRo and @tridge and we wonder if perhaps the two CAN GPSs are both using the same NodeID. Apparently they will come from the factory with the same ID so one of them needs to be changed manually.

Changing the NodeID is done using Mission Planner and SLCAN. We’ve got a discussion here and a wiki page on SLCAN here.

By the way, the firmware on these GPSs is apparently AP_Periphs and the upcoming version will support “dynamic node allocation” which will mean manually setting the NodeID won’t be necessary anymore.

One way or the other we shouldn’t have an “Internal Error 0x400” as the way users are warned of this issue so we need some better error handling and wiki documentation.

P.S. the internal errors are recorded in variables that are cleared when the vehicle is rebooted.

EDIT: we think we’ve reproduced an issue with the dynamic node allocation when more than one LocationOne GPS is used.

@RomanD, @VadimZ,

We’re not able to reproduce the “internal error 0x400” issue sadly. It’s possible (although perhaps it’s wishful thinking) that we’ve somehow resolved the issue between Copter-4.0.0 (which is what @RomanD’s log shows) and Copter-4.0.2 (the version we tested with).

Could you upgrade to Cotper-4.0.2 (stable) and see if the problem is resolved or not? It probably isn’t but testing will reduce the number of possible causes of the issue.

i meet message “internal error 0x400”

could you check the log?

@rmackay9,
We’ll try to check the issue with V4.0.2 and verify if the error still comes up.
Regarding the NodeID - both units are already set to allocate the NodeId dynamically and we’ve double checked that they had different NodeId’s.

1 Like

Hi @rmackay9,
We’ve encoutered the issue also with V4.0.2 - see screenshot attached:

.

Log file

@rmackay9
regarding you recommendation to use dynamic node allocation: won’t it play havoc with compass calibration ? It seems calibration is indexed by device ID which for CAN device includes node ID. So without consistency guarantees for CAN ID allocation between boots, compass calibration would not be found.

Edit: I see now there’s a mechanism that attempts to re-issue ID based on CPU unique ID. However we’re investigating an occasional “Compass not calibrated” when multiple CAN compass devices are present. So that’s the leading theory so far.

Edit 2: Indeed there is a problem with assigning IDs to CAN compasses. However it is not where I originally thought - the problem also happens with static IDs. On some boots the IDs are switched between the two compass devices: Compass 1 gets the node ID of Compass 2 and vice versa, messing up the calibration.

Edit 3: There is indeed a race condition in CAN compasses initialization, and their ordering is timing-dependent.

Edit 4: https://github.com/ArduPilot/ardupilot/pull/12629 is referring to this problem exactly: “Compass_UAVCAN driver is modified to reorder the detected devices, so that the compass with larger node id is sent for registration first”. Without this fix, multiple CAN compasses can not be used. Hopefully this PR makes it into 4.0 soon.

@RomanD,

Thanks for confirming the “internal error 0x400” issue still occurs on. We’re going to come up with a more detailed message that tells us where the “constrain NaN” is coming from so we can figure out how to fix it. I suspect this will take us a few days to come up with but it’s on our to-do list so it won’t be forgotten.

1 Like

Would it make more sense to statically assign a node ID? That way you know which is which?

@smartdave - I know which one is which, the issue seems to be internal on how the AP is handling the allocation (I’ve reported 2 different issues that might be related to the same core issue, not sure though…)

I have multiple CAN gps’s, so if you want me to test anything for you I can.

Just to see if I see it as well

I don’t know if this is related, but we’re getting the internal error 0x400, but with only 1 mRo location one. This is Copter 4.0.3 on CubeBlack, a Serial/i2c Here 1 gps, andf the location one connected via can. It seems we got the errors when we had blending turned on, but if auto switch was set to useSecond we did not see the issue.

https://drive.google.com/file/d/1c8iLd1f2kR7c5Kovqds4BoIZfX16Jh0j/view?usp=sharing

@mike_d,

Yes, this has been reported before actually and is listed on the issues list (issue, discussion).

The issues is that the older version of “AP_Periph” that is shipping with the LocationOne GPSs does not do “dynamic CAN node allocation”. So I guess the solution is to either update the firmware on the GPS or to manually set the node id. I’m actually not personally sure how either of these can be done but I have raised an issue for the wiki here: https://github.com/ArduPilot/ardupilot_wiki/issues/2557

Internal error 0x400 means that something produced a NaN which is generally not good.

Maybe @tridge and/or @pkocmoud have some more input on this.

I’ve created a new issue here as well: https://github.com/ArduPilot/ardupilot/issues/14879

i am getting this similar kind of error => PreArm: Internal errors 0x400 l:269 ,cnstring_nan

I’ve found a way to reproduce it, i am using 7 inches quadcopter running arducopter 4.1.0-dev on pixhwak 2.4.8

On my drone this issue can be reproduce during compass/motor calibration, when doing compass/motor calibration, after calibration gets successful, sometimes there seems no issues, but when repeating this calibration multiple times, sometimes just after successful compass/motor calibration i can see this error popping up on compass/motor console and on Mission Planner HUD message console.

I have also seen that this issue seems to be related to barometer on my quadcopter. When this error message pops up and afterwards if i uncheck barometer arming check from prearm checks then this error message stops. But i didn’t get what causing this error ? i have 2 similar kinds of drone with exactly same setup and this error is reproducing in both the drones.

I have another concern that, i have few other small drones too, but when i do compass/motor calibration on them, i have never seen successful compass/motor calibration unless i finish it manually, but one particular drone after starting the compass/motor calibration showing successful calibration by itself, i don’t think it supposed to finish calibration itself.

This forum thread is not for 4.1.0-dev version. And for that particular version we need to also get the git hash, so that we understand exactly which codebase you are using. So please open another thread.

Could we get some onboard logs and tlogs, please?