In-flight Watchdog reboot on 4.0.4 and mro zero f7

Daniel_Met · October 2, 2020, 11:13am

Hi there,
yesterday we experienced an in-flight watchdog reset of the Flight Controller which caused a crash.
The Logfile after the reboot contained this Watchdog information

Does anyone have ideas what might have caused this ?
We are flying 4.0.4 and a mission controlled by a companion.
The crash happened after about 15 minutes of flying
Thanks,
Daniel

amilcarlucas · October 2, 2020, 12:48pm

@tridge do we have a bug here?

tridge · October 2, 2020, 9:59pm

looks like we got a fault in the UART thread, the address is in this function:

ChibiOS::UARTDriver::write_pending_bytes_DMA(unsigned long)

the address matches this line of code:

github.com

ArduPilot/ardupilot/blob/Copter-4.0.4/libraries/AP_HAL_ChibiOS/UARTDriver.cpp#L797


        uint32_t max_tx_bytes = 1 + (_baudrate * 250UL / 1000000UL);
        if (tx_len > max_tx_bytes) {
            tx_len = max_tx_bytes;
        }
    }


    dmaStreamDisable(txdma);
    tx_bounce_buf_ready = false;
    osalDbgAssert(txdma != nullptr, "UART TX DMA allocation failed");
    stm32_cacheBufferFlush(tx_bounce_buf, tx_len);
    dmaStreamSetMemory0(txdma, tx_bounce_buf);
    dmaStreamSetTransactionSize(txdma, tx_len);
    uint32_t dmamode = STM32_DMA_CR_DMEIE | STM32_DMA_CR_TEIE;
    dmamode |= STM32_DMA_CR_CHSEL(sdef.dma_tx_channel_id);
    dmamode |= STM32_DMA_CR_PL(0);
    dmaStreamSetMode(txdma, dmamode | STM32_DMA_CR_DIR_M2P |
                     STM32_DMA_CR_MINC | STM32_DMA_CR_TCIE);
    dmaStreamEnable(txdma);
    uint32_t timeout_us = ((1000000UL * (tx_len+2) * 10) / _baudrate) + 500;
    chVTSet(&tx_timeout, chTimeUS2I(timeout_us), handle_tx_timeout, this);
}

now we just have to work out how we can generate a hard fault at that address.

tridge · October 2, 2020, 10:05pm

hi @Daniel_Met, can you please tell me the exact UART setup you have. What devices are attached to what ports?
My guess from the log is you have a telem ratio on Telem1, you also have ESC telemetry enabled on SERIAL5. Was that active in the flight? I don’t see any ESC data logged.

Daniel_Met · October 2, 2020, 10:50pm

Hi, thanks for looking into it.
You are correct, we have
Radio on 1, Companion on 2 and
Telemetry on serial 5
I added the logs here https://github.com/ArduPilot/ardupilot/issues/15488
The esc logs are present

tridge · October 2, 2020, 11:01pm

do you have flow control pins to the companion?

Daniel_Met · October 2, 2020, 11:03pm

No we don’t use hw flow control

tridge · October 2, 2020, 11:26pm

@peterbarker and I are looking at this, but could you also see if you can reproduce on the ground? leave it running with companion active overnight. I’ve also setup the a mRoControlZeroF7 with the same params on my desk.

tridge · October 2, 2020, 11:27pm

any guesses why the board voltage is varying like this?

period of over 20s

Daniel_Met · October 2, 2020, 11:32pm

I will try to recreate it. We had several hours of flight time with two systems before this happened so it might take a while

Daniel_Met · October 2, 2020, 11:33pm

I actually missed that when reviewing the log. Ill look into this

tridge · October 2, 2020, 11:33pm

thanks, also check if the same VCC pattern is in earlier flight logs

Daniel_Met · October 2, 2020, 11:44pm

The pattern is in earlier flight logs too.
It very likely comes from a onboard strobe light on the drone but i have to verify that

Daniel_Met · October 7, 2020, 8:01am

Hi we had a second Watchdog reset
i added the Logfiles to the Git Issue

tridge · October 7, 2020, 7:49pm

@Daniel_Met I managed to reproduce the watchdog and I’ve proposed a fix here:

we plan on including this fix in the upcoming 4.0 point releases

tridge · October 7, 2020, 9:30pm

@Daniel_Met I have built 4.0.5 with the DMA fix here:
http://uav.tridgell.net/DMA-fix/
this should fix the bug. If you have a chance to fly this and report back that would be appreciated!

Daniel_Met · October 7, 2020, 9:50pm

Thanks, i saw it earlier today we already installed it and will test

Daniel_Met · October 7, 2020, 11:07pm

Out of curiosity, was this issue present in earlier versions and on what hardware could this happen?
Im just curious why we didn’t have this issue before

tridge · October 7, 2020, 11:10pm

the issue is most likely to happen when a UART TX DMA channel is shared with another high rate sensor. For the mRoControlZeroF7 the UART you are using at 921600 is shared with the SPI barometer.
Also note that I only managed to produce the bug once in my testing. I plan on trying some other approaches to making the bug happen more often later today.

Daniel_Met · October 13, 2020, 11:24am

We did several testflights and so far the issue has not happened.
I think its prettly likely that the bug is fixed
We will continue to monitor it and if something happens again Ill report it.