Dronecan dataflash log esc[n].err

Hello,

I would like to ask help on finding the cause of esc[n].err dataflash log entries (and possibly fix it).

I am attempting to configure a redundant fc-esc connection ( with parallel bdshot300 and dronecan ), for a hopefully reliable long range quadrotor system. This is working on a test setup, but if I look at the dataflash logging, esc[0].err reads a number around 50… 53-ish with some peaks (see plot).

Because the whole point of a redundant setup is higher reliability, I would like to figure out what the err number means, and if its a cause of concern. The doc page for the esc ( Hargrave Technologies DroneCAN ESCs — Copter documentation ) says ‘0 indicates no errors’ (but does not state what a number different to 0 means). If I look on the logs on the esc, no errors are reported.

Also because the setup now has both bdshot and dronecan connected, does the esc.err log entry also register bdshot errors? are they mixed somehow or are they captured somewhere else?

Test setup:

Tbs lucid h7 fc, Hargrave microdrive lpi esc, Apd pdb 500. wiring from fc->pdb->esc in the same way as in the drone ( and as recommended by hargrave/apd ), just in the test setup the esc telemetry wire is not connected to make it easier to see what is happening. ap rev 4.7.0, microdrive rev 3.2.0.

Dronecan setup by following dronecan setup page and dronecan esc setup page for the specific esc. Telemetry message frequency, 10, 50, 100 & 200Hz ( I would like to use telemetry rpm based notch filters, so high tlm frequency is tested… however that does not seem to make much difference for the error messages I’m requesting help on ).

Param. file attached, this is a setup for testing, i.e. no flight setup yet…

parallel_dshot300_dronecan_setup.param (23.0 KB)

small zipped log of a testbench-run ( motor test )

30 1-1-1970 01-00-00.bin.zip (885.3 KB)

Any help very much appreciated!

Boris

I may not be much use to you yet, but I’m setting up something similar with the Nanodrive 4PLi.

Errors are here, but the graph you show looks more like number of errors or something like that instead of an actual error code.

Also try setting CAN_LOGLEVEL to 3 or 4 and running the motor tests again.
Also try connecting only DSHOT or CAN and see which interface is returning the errors.
Is the CAN Termination setting enabled in this ESC?

Try assigning priority to the CAN and DSHOT signals too, CAN would be primary and DHSOT (serial) would be secondary.
Once everything is working correctly you can also set ‘Require All Signals to Arm’

EDIT:
Actually any errors should be accessible via the Hargrave Configurator and will be persistent too, so you should see any historic errors until you reset them.

Hi Shawn,

Thanks for the advice, helped a lot! short summary of tests:

  • Looked at the esc logs, and the esc errors, no errors or anything strange on the esc side.
  • Set can_loglevel to 4, redid test, esc[0].err looks nearly identical to the plot I included in my last post, can parameters look very good as far as I can tell, ( not very much experience with can yet ).
  • Connected dshot only (no tlm wire, no can wire), redid test, esc[0].err still looks almost identical!.. so this may be a dshot - only issue, but to make sure…
  • Connected can only, redid test, esc[0].err shows one spike ( when motor starts ), rest of plot is 0… so this now very much points to dshot only…

So I’ll be troubleshooting what causes bdshot300 errors, but advice helped alot narrowing down the issue…

Reply to the other points you mention:

  • can termination is set in the esc, and is included in the fc according to tbs.
  • priority is set in the esc, canbus first, dshot second.
  • will test the ‘require all signals to arm’ setting, was disabled for these first tests.

Thanks for help!

Boris

Hi @Boris,

Thanks for the report. I don’t think we officially support using both CAN and DShot as a way to provide signal redundancy to an ESC but I’ve added this to the 4.7 issues list so it will at least be discussed at the next dev call (tomorrow)

If it were me I would instead use parallel CAN buses rather than attempting two different protocols (e.g. DShot + DroneCAN). If I were going to mix protocols, I would use DroneCAN and good old fashioned PWM.

As a side note, I’ve never personally seen data that shows that parallel communication between two pieces of hardware improves reliability. Surely ESC failures are far more common that FC<->ESC signal failures and so for a real jump in safety I would use an octa-quad rather than trying to improve a quadcopters safety. That’s just my opinion though of course.

Hi @rmackay9 ,

Thanks for reply, and for taking the time to add this to the issues list!

Pls note, I’m just testing this because the esc’s used offer this option, and it already seems to work. I can interrupt the can or bdshot signal line in any order and the motor keeps running nicely. The only issue that I can see so far, are the strange .err readings that look like bdshot related instead of the dual signal path related.

The esc’s do not offer parallel can inputs, so attempting parallel can busses would be difficult with this hardware setup. But that would be more logical I agree. The reason I tested bdshot instead of pwm is that I would like to keep rpm filtering going also on the ‘backup’ signal line if at all possible.

The main reason we are pursuing this is to reduce the chance of losing a system without a way to really find the cause / getting hold of the .bin file… ( we had an incident like that a couple of weeks ago, with most likely cause that we can find ‘that something went wrong in the signal path from fc to esc’, that system was running bdshot600, no can yet, just .tlog link to help request for this incident ).

Improving reliability by choosing an octa-quad ( or hex ) configuration was certainly a consideration, but in our experience, is hard to combine with (very) long endurance. So we went for a ‘simple’ quadrotor, a classic engineering tradeoff I guess.

Thanks for the help!

Hi @Boris,

Ok great, a very logical explanation. Let’s see if other devs have ideas!

Hi @Boris,

We discussed at the dev call and I’ve had a bit of a peek at the code:

The ESC.Err field is incremented if ArduPilot receives a DShot telemetry packet from the ESC that it cannot decode properly. In particular if the RCOutput::bdshot_decode_dshot_telemetry function returns false the error is incremented.

It’s not clear why the packet is garbled but it could be:

  • The Hargrave ESCs are sending invalid packets
  • The ArduPilot side has some issue with it’s parser
  • Noise on the wire between the autopilot and ESC (I think this is unlikely but it might help to do a bench test with a single ESC to see if errors still appear in the logs)

As for the next steps, I’ve reported this to Hargrave (they’re an AP Partner) but as a paying customer you might also want to report this to them directly.

It would be nice if we could determine if all Hargrave DShot ESCs show this problem or is it just this one set of ESCs.

Capturing the message from the ESCs would help us determine which side has it wrong (the ESCs or AP).

Maybe this has been asked, where are the capacitors installed?

Hi @Boris , we’re having a look into this now, we’ll report back when we know more. Thanks for pinging us @rmackay9 .

Hello,

Thanks for taking the time to look at this, I’m not very experienced at coding, but I can answer some of the questions…

I looked at the signal with a scope ( while running a single esc - ap motor test at 6S voltage i.e. not a high load situation ), in that condition it does not look very noisy to me.

I am building the next system right now, so will be able to test with 4 esc’s in a couple of days if all goes well, will post if these errors happen to all 4 esc’s. The esc’s I tested indicate no errors in their logs.

Perhaps also relevant, the previous system flew several hours with identical hardware, dshot-only wiring between fc and esc’s and no errors in the esc[n].err logging except at boot time sometimes. It had different firmware on both the fc and esc ( AP 4.6.3 and MicroDrive fw 3.1. ). Current setup runs AP 4.7.0 and MicroDrive fw 3.2.0. So it is possible that something changed on both sides.

I am not sure if I can help here, I have limited experience in capturing this, may ask around for help though…

Will post results of 4-esc setup once all is up & running.

esc’s come with capacitors on the esc pcb, according to manual these should be sufficient with fairy short wiring, and wiring is <1m for the motor wires, <0.25m for the signal wires.

If you get the chance, try DHSOT only (instead of BDSHOT) and also try turning off the extended telemetry info. Any difference could help to identify where the problem lies.

Hello @xfacta

I’m not sure if I can test this in the way you suggest, but here goes:

  • tested like before: bdshot on, only (b)dshot wire connected, no can wire, no tlm wire. ran mp motor test for 15s @10% throttle → gives same results as before, but perhaps relevant I see no bdshot info other then rpm, i.e. no voltage, current etc… not in the logs, not in the mission planner status pane. I think those are part of the edt info, that is maybe not coming through. the esc[0].err rate remains around 50…53-ish.
  • tested with edt off: set servo_dshot_esc from 3->1 in ap & reboot. I dont think edt can be switched off on the esc side ( it looks like that is possible for the can bus but not on bdshot I think) so esc unchanged and set to “serial signal type: bidirectional dshot, baud rate: dshot300”. Again running motor test gives again almost identical results, rpm feedback, esc[0].err around 50, no volt, amps etc.
  • tested with dshot only, set servo_blh_bdmask = 0, reboot. set serial signal type in esc to ‘auto’ and baud rate to ‘auto’. running motor test gives no esc data in status pane of mp, and no log entry obviously, but motor spins happily.

Hope this is helpful! & thanks for the troubleshooting!

Hi @Boris,

Txs for the testing.

Re this request from me, “Capturing the message from the ESCs would help us determine which side has it wrong (the ESCs or AP)” this is almost a note to myself because we would need to provide you with a special version of AP that logs the message received from the ESC to the AP logs.

I can certainly do this although I’d like to hear back from @saami_hargrave on whether they’re seeing errors as well on their setup.

Hi everyone, we’ve been testing this quite a bit over the past day. Our current thoughts are there’s an issue in AP where if less than one ESC sends data to the FC, the logging for the ESC error breaks and reports incorrect values of 50, instead of the actual error rate. The issue only occurs when less than 4 ESCs are connected.

Our baseline was two working quads running Arducopter 4.7, one running a nanoDRIVE 4LPi, one running 4 x microDRIVE LPs on GK3.2 firmware, that have consistently displayed the correct ESC error rate (params file attached).

We then disabled the serial signal in the ESC firmware to make AP register only one ESC, which caused the issue on the previously working quad.

We don’t believe CAN has any relation to this issue from our testing.

Also uploading logs and screenshots from the tests.

Let us know if you’ve got any questions.

Working Acro.param (21.7 KB)

TBS_LUCID_H7 - Working Nanodrive config on LPi.bin (496 KB)

TBS_LUCID_H7 - Non-Working Nanodrive on 4lpi (3 ESC Disabled signal).bin (880 KB)