Crash - spinning quad on Pixracer R15, Arducopter 3.6.6

ArduCopter Copter 3.6
1

Hello, I just changed the autopilot on a quad from Pixhawk Cube to MRO Pixracer R15 flashed with Arducopter 3.6.6. I am using the ublox SAM M8Q GPS / compass. The orientation of both the autopilot and GPS/compass is normal, both pointing to the front of the vehicle.

I have disabled both internal compasses and set the external as primary. I have performed Live calibration (multiple times) outdoors.

When taking off, the vehicle immediately start spinning to the right. On Mission Planner, I get messages like “EKF2 IMU0 in-flight yaw alignment complete” and my EKF Status often goes berserk for the compass (for example, around 81% of the telemetry log). Afterwards, I find that my compass alignment has been changed to Yaw 180 without me having touched it.

Compass_screen

However, even after the autopilot forces this change, the problem remains. I have run out of ideas, can anyone help?

Telemetry and .bin logs can be found here.

2

Well, now I have a crash after trying the internal compasses.

I first tried disabling compass 1, making compass 2 primary, recalibrating, rebooting. The copter rose spinning, just like before. This time, I got “EKF2 IMU0 ground mag anomaly, yaw re-aligned” on Mission Planner but the problem did not go away.

I then tried disabling compass 2, made 3 primary, recalibrated, rebooted. I took off and now it span really fast, nothing like the gentle rotation of compass 1 and 2, this looked ugly and I immediately knew I would not be able to put it down in a pleasant way. I landed and it immediately broke off a leg and started flopping around smashing the props. I tried to disarm by down-left throttle but it would not disarm (or perhaps I did not hold it long enough). I went for the safety switch and that gave a prop the opportunity to slice right through a fingernail and the yummy flesh underneath. The strange thing is that, although MP reported a “Crash: Disarming” message, the motors were spinning quite happily till I pressed the switch.

After the crash, I also noticed a “GPS Glitch” and “GPS Glitch cleared” message. I do not know if they had anything to do with the crash, I did not hear anything about a GPS glitch before throttling up for take off and I was right next to the laptop but I might have missed it.

The two new logs (.tlog and .bin) have been uploaded to the same folder here, prefixed as CRASH. The .tlog contains the whole sequence, including compass calibrations and autopilot reboots. The bin is only the final (crash) flight. If anyone is interested in the non-crashy .bin files I can upload them.

Looking at the telemetry log, I see that the final flight arming occurs at around 82.8 % of the log. Current draw rises rapidly to around 12 Amps by 84.3% of the log, followed by EKF status for the compass going immediately red, followed by mad spinning of the heading. At around 85% of the log, current draw has jumped to approximately 25 Amps, possibly indicating motors having hit the ground. After this point, I see the
“GPS Glitch”, “Crash: Disarming” and “GPS Glitch cleared” one after the other, while the current drops to zero. It is possible that my grabbing the vehicle while reaching for the switch actually triggered the crash detection just as I pressed the button.

3

I don’t have anything to look at logs where I am now… But…
Does it fly OK in stabilize mode? I’m pretty sure the compass is not used for anything other than a long term correction of yaw… Not enough to cause the copter to spin. So if it’s spinning in stab, maybe something else is wrong.
And, have you done a compass motor calibration? not just the compass calibration, but the one where you strap down your copter w/ props on it, and spin up to full speed. (Can be scary!)
Last thing I can think of is it’s possible it’s still using the disabled compasses…

4

There is such a big discrepancy between motor outputs it would appear to be a purely mechanical issue.

42%20am
Screen Shot 2019-02-18 at 6.54.42 am.png2303Ă—962 288 KB

Your clockwise motors are flat out while your anticlockwise have all but shut down.
Wrong props?
Motor direction?
Motor order?
Frame Class and Type seem to be correct.
Calibrated ESC’s?

1 Like
5

I did not try stabilize mode, my take offs were Alt Hold or Loiter.
I have not done a compass motor calibration.

The vehicle was flying fine with these motors, ESC, props with a Pixhawk 2 Cube before I swapped to Pixracer R15. There has been no change to anything except for the autopilot, GPS/compass, power module and telemetry radio (the props were not removed either, they are exactly as before).

@mboland, if you are looking at the final .bin, I am not surprised one pair of motors is flat out, the performance looked more like demonic possession than a flight. However, this was not the case when I had tried compasses 1 and 2. With those compasses, it was spinning but in a very controllable manner. Sure, I didn’t really have directional control to speak of but I could put it up and down without fear. In the final flight, the moment it was off the ground I knew I had lost it. This, to me, sounds like something weird going on with the compass.
Specifically, to your questions:

Props: good (not changed from previous build with Pixhawk 2).

Direction: good (“docile” spinning flights were not too bad, spinning aside).

Motor order: good to the best of my knowledge. All my cables are labelled and they seem to be in the appropriate sockets.

ESCs: you might be on to something here. They were not calibrated (these are the ESCs in question). They are OneShot125 and in my previous build (Pixhawk 2, cube) had MOT_PWM_TYPE set to 2. In this build, I forgot it and left it at 0.

Summing up, ESC calibration might have been a contributing factor but I don’t think it is the main cause. The motors were all starting to spin in apparently reasonable sync when arming, throttling up slowly while looking for the liftoff point did not seem to produce any noticeably weird effects as might have been the case if some motors were seriously performing differently. In both the “docile” and the “wild” spins, it felt as if the trouble started the moment the legs left the ground. This is also my guess given that the big difference in performance happened when I switched to compass 3, the compass bar in the EKF status going red, the “EKF2 IMU0 in-flight yaw alignment complete” and “EKF2 IMU0 ground mag anomaly, yaw re-aligned” messages and forced change to my external compass orientation after I had performed live calibration without any apparent issues. It seems that something compass related was making the EKF have a fit but I have no idea what it could have been.

6

I have the same GPS unit and a pixracer R14. The arrow points in the wrong direction, but the auto-orientation works for me. With that unit and a pixracer R14 I cannot get all the compasses to calibrate - I gave up on the second compass. I also had to re-mount the GPS unit away from interference.

7

@andyp1per this is very interesting. Which arrow is wrong? The one on the autopilot or the one on the compass?

8

The one on the compass/GPS

9

Aha, this explains the “EKF2 IMU0 ground mag anomaly, yaw re-aligned" message and the forced changing of the external compass to Yaw 180. What it does not explain is why I also got this message when having apparently disabled the external compass, unless the internal compasses are wrongly aligned on the Pixracer R15 (extremely unlikely). It might hint that, as @wicked1 mentioned, the autopilot is still using the (seemingly) disabled compasses. It will take me a while to rebuild (I need to get new props at the very least and time, which I do not have that much of right now) but when I do, I will try again with external compass flipped around, proper setting for SingleShot125 for the ESCs and possibly forcing COMPASS_TYPEMASK to get rid of the internal compasses.

10

3.6 will auto-orient the external compass if you don’t set it explicitly so it shouldn’t matter. It only occurs at calibration time

1 Like
11

OK, I found time to work on this and am documenting what I find.

There is some weirdness going on in the (current state of the) documentation for COMPASS_TYPEMASK.

The docs imply there are 16 bits in the mask (0-15) but skip one (number 10).

image
image.png169Ă—551 7.28 KB

Still, not such a terrible thing. 16 bits sounds about right (I wonder how this will be modified when there are more magnetometers to support) and perhaps there is a good reason for skipping the10th digit (maybe it’s considered unlucky in binary or something).

Now, for more fun. My mags are:

On the GPS, a LIS3MDL, as mentioned in the MRO site.

On the board, one Invensense MPU-9250 3-axis accelerometer/gyroscope/magnetometer and one ST LIS3MDL magnetometer as mentioned in the MRO site.

This, immediately makes it hard to disable specific devices at the driver bitmask level, since I have two LIS3MDL, one internal and one external. To make matters more fun, the Invensense MPU-9250 is not mentioned in the docs, though, of course it is detected.

So, I started by disabling everything and then enabling one at a time. Sure enough, I got a hit when I enabled AK8963, which I could have figured out if I had the brains to google info for the MPU-9250.

Now, how do I know what I detect? To the best of my knowledge, the only bootup messages MP shows are under the “Messages” tab and they do not say much, other than report the GPS model. What I do, is fire up the tuning screen and plot my, my2 and my3 and see what gets displayed and moves when I move the vehicle (if there is another way I would love to hear it please). The problem is, I have no idea which mag was the compass 3 which caused the trouble to begin with. I presume it was the AK8963, since it is the odd one out but I don’t know how to check it. If I enable all mags, I get them all detected but I have no idea how the board decides which one is external (since the external one is the same as the same as one of the internals) and no idea how it decides which one of the internals is 2 and which is 3.

So, to recap, at the moment, I have fixed the vehicle, disabled the AK8963, changed MOT_PWM_TYPE to 2 and did a compass calibration (indoors). The weather is atrocious but am expecting to go fly in the next few days. If anyone has any ideas, now would be a terrific time to tell me :wink:

I will report back after I fly.

12

The Device ID’s identify which compass’s are assigned. This python script will tell you what they are and what bus they are on.

Compass decode.zip (1.2 KB)

1 Like
13

Hi @armadillo,

Did you get to the bottom of the crash? I agree with Mike Boland’s diagnosis that this is most likely a mechanical issue.

We can see from the logs that this is an uncommanded rotation. The autopilot software is actually trying to spin the vehicle left (counter-clockwise) but it is spinning to the right (clockwise).

yaw-rate
yaw-rate.png1204Ă—429 16 KB

I suspect that all the propellers directions are reversed so maybe compare the vehicle’s propeller direction and motor numbers very carefully vs the QuadX diagram on this wiki page.

Another possibility is that an ESC calibration is required. Testing the motors with the motor output screen on the MP would be good as well i think.

14

@rmackay9 I have not yet gone to test fly a second time, the vehicle is ready, the weather isn’t.

It is almost impossible to have been a mechanical issue, the vehicle was flying perfectly for months without problems with a ProfiCNC Cube. When I changed the electronics I didn’t touch anything beyond the autopilot, the props are exactly as they had been with the Cube, screwed and thread locked.

The ESC calibration seems to be the only obvious hiccup, though I personally doubt it is the cause. It doesn’t explain the phenomenal difference in behaviour when switching compasses.

Beyond fixing the MOT_PWM_TYPE, I have also upgraded to the latest Arducopter. I believe I will be able to test over the weekend and report back.

15

Update. I have successfully taken off in Alt Hold, indoors, with no apparent problems.

To recap, the changes I have made were:

  • flashing latest Arducopter (v3.6.7)
  • fixing MOT_PWM_TYPE to 2
  • recalibrating my ESCs (this should not have been necessary since the quad had been flying fine before but can’t hurt either).
  • Disabled the AK8963 internal compass by setting COMPASS_TYPEMASK to 7135
  • Mounted the ublox SAM M8Q GPS / compass front-to-back (thanks to the tip by @andyp1per )
  • Disabled compass 2 in MP and left compass 1 alone as primary.
  • Did a compass calibration on the one, external compass.

Before taking off, I armed, throttled up a bit while holding the quad above my head and tried giving stick inputs to make sure it was responding to pitch, roll and yaw as expected (it did). I then tried a very careful indoors take off and it performed absolutely beautifully. I also did not get the "yaw re-aligned” messages so it seems Andy’s tip was spot on.

Although I have to admit it surprises me a lot, it seems that most likely, the problems were related to wrong ESC settings. This has now been solved.

A new and somewhat worrying development however is that I now cannot get a GPS fix. At the time of writing this, I have left the quad for over an hour outside on a balcony, rebooted multiple times but am not getting a fix (satcount=0, gpsstatus=1). I do not think the GPS has been damaged in the crash, since I had been getting a fix while powering up indoors when fixing the crash damage to the quad’s body. The GPS is actually being found when powering up, I am getting:

GPS 1: detected as u-blox at 115200 baud

and

u-blox 1 HW: 00080000 SW: ROM CORE 3.01 (107888)

messages in MP when powering up but then nothing happens. I am not sure if I should be getting worried and how long waiting for a fix is too long. If anybody has any ideas, please do not be shy to share. I will keep trying and post if there any developments.

16

Hello again, I would love to be able to consider this thread closed but events do not allow me to. Today, I did an outdoors test with mixed results (following my previous post, let me note that the GPS eventually got a fix after hours outside). Telemetry and bin logs are here. Video of the test is here.

First of all, these are the changes I made:

  • I have enabled all three compasses
  • made the first compass primary
  • Set EK3_ENABLE to 1
  • changed AHRS_EKF_TYPE to 3
  • Did an outdoor compass calibration (completed without any apparent problems) followed by reboot.

Now, I knew from the indoor test that the spinning that led to the initial crash was hopefully fixed and had been most likely attributed to the ESCs. What I wanted to check now was performance in Stabilize, Alt Hold and Loiter, in that order.

So, let me go over the video:

From the start to 1:43 I am doing the Stabilize test (notice mode changes to Stab 0:16). This is the first power up with EKF3, outdoors on a windy day so am taking my sweet time getting it off the ground. Once airborne, Stabilize seems to be doing fine.

1:54 to 2:50 is another short Stabilize test, just to make sure.

2:56, notice that I change mode to Alt Hold for the next test which runs until 4:30 without any problem.

4:35, I change mode to Loiter for the next tests. Shortly after take off at 5:06, I get “EKF3 IMU0 ground mag anomaly, yaw re-aligned” (this event is at 74.45% of telemetry log “2019-03-25 13-48-05.tlog”). After that, I immediately land, disarm and do a software reboot.

After the reboot, I put it in Alt Hold (6:27) and do a short flight till 7:43. Nothing interesting happens in Alt Hold.

At 7:50 I switch to Loiter and try it out again. I try a few slightly aggressive manoeuvres to see if anything interesting will happen. Pitch and roll don’t seem to be having much of an effect, but the moment I try yaw at 9:12, I get “Error pos vert variance” and EKF changes to 0 and then 1. The Position (Vert) bar in the EKF status window is all the way up and red, this happens at around 96.80% of the “2019-03-25 13-48-05.tlog” telemetry log. Again, I cautiously land.

So, this is where I am at the moment. My question basically is, is there anything wrong with this vehicle or am I being paranoid and are these errors to be expected?

17

It is time to close this thread with acknowledging that I most likely got a hardware fault. After the last post, I swapped out the Pixracer and installed an old Pixhawk. The vehicle again performed flawlessly. So, we have flawless behaviour with a Cube and a Pixhawk and weirdness with a Pixracer, it pretty much all points to a bad board, with the exception of the spin, which was likely attributable to bad ESC configuration.
I have sent back the board to MRO who have been super helpful throughout.

18

Hi, what software did you use to create this graph? And was it a tlog or data log file? Thank you