new FFT Filter setup and review web tool

[iampete (Peter Hall)]

We have recently added a new filter setup tool that should speed up the processes of filter setup. It can be found at: https://firmware.ardupilot.org/Tools/FilterReview/

The key feature is the ability to estimate the post filter results from a pre-filter log, this eliminates the need to do several post filter flights.

You just need a log with either “Raw IMU” selected in LOG_BITMASK or batch logging enabled.

Load the log and select a stable section in the flight data plot to for a clean look at the vibration profile. After selecting a section hit the calculate button. (Note that the it can take quite a while to processes logs, your browser might warn that the pages is not-responding.)

The FFT plot then shows the noise, as you may be familiar with from our other FFT tools. You can plot each IMU and each axis. There is pre-filter, post-filter and estimated post-filter. If pre-filter data is available a post filter estimate will be calculated, this estimate can be updated by changing the filter parameters and re-calculating the results (don’t forget to hit the calculate button).

The spectogram shows how the noise changes over time and plots the notch tracking sources, the tracking should follow the peaks.

The bode plot shows the mean transfer function along with the range, this can be used to see the how much phase lag the filters generate.

I have done a demo with some example logs that covers how the tool works in more detail:

ArduPilot Filter Review Tool Demo

The code can be found on Github here:

GitHub - ArduPilot/WebTools

Feedback and Pull requests welcome!

[timtuxworth]

Would this work (and make sense) with a Fixed Wing - or does it only apply to multi-rotor/vtol vehicles?

[Allister (Allister Schreiber)]

This looks amazing! Thanks for all the hard work you are putting into it.

Can you clarify for me: If I use it on a flight without a notch filter enabled (initial test flight for example), would the filter configuration window suggest values or will it allow me to try out my own values to see in the estimated post-filter graphs?

[iampete (Peter Hall)]

Yeah, works fine for plane, just be aware that the throttle tracking is for VTOL throttle not forward throttle.

[iampete (Peter Hall)]

It will not suggest values, it just pulls the current config from the log. But it will allow you to try out new configurations without having to repeat the flight and log post filter.

[timtuxworth]

Do you think batch logging or raw IMU logging would be best? What are the considerations?

[timtuxworth]

Can we get a student to build an AI that would be able to do that?

[Yuri_Rage]

Probably wouldn’t even need an AI. Just iterate base frequencies until area under the curve is minimized, then reduce bandwidth until an increase in integral area is observed. I’d think that would get pretty close.

Similar procedural analysis of available telemetry and frequency peak count could arrive at recommended options/modes.

[iampete (Peter Hall)]

Raw logging removes the quantization noise of batch logging and gives continuous data so your spectogram is much nicer. However this results in much larger logs, F4’s might struggle to keep up with the logging rate.

Batch allows logging of post filter or logging pre and post at once and gives much more manageable log sizes.

Personally I have been using raw logging.

[xfacta (Shawn)]

@iampete Fantastic work!
I will definitely be using this extensively.
I’ve been testing it out on a couple of logs with a spread of frequencies that dont quite make sense and this has really helped to simplify and better target the HNOTCH settings.

I would point out this is not for the feint hearted or first-time user - you have to know your harmonic notch filter settings and what to look for - but it sure cuts down on the number of test flights.

@iampete Updates
Browser: Firefox 114.0.1 (64-bit), Win11 - Update2, flight data section works OK in Chrome.
I havent been able to get this (in general) to work with older logs, even around copter 4.2 (and maybe even earlier 4.3) but I’ll see if I can find more logs to test - selecting the log to load doesn’t produce any outcome or CPU load - I’m watching the CPU graph to tell when processing is underway or finished.
I havent seen the attitude/throttle graph populated yet, although that is not very important.

[timtuxworth]

I don’t know if this helps at all @iampete - but I flew this today on my “no GPS” test vehicle. I turned on raw IMU logging and ran it through the tool. It looks pretty clean. If you have any thoughts about tuning, I’d appreciate it.
This is the log: Dropbox - 00000120.BIN - Simplify your life
And here is what the tool shows:

[xfacta (Shawn)]

I’ve had a verified win with the Filter Review Tool ! (well it’s someone else’s copter, not mine…)

As mentioned in the linked discussion, I dont think I would have been able to predict the required HNOTCH settings using the MissionPlanner FFT graphs alone. I probably would never have tried the frequency that was actually required, and would have resigned myself to thinking the “old way” was as good as it was going to get, with some noise remaining, not so effectively filtered.

Harmonic Notch Tuning. I must be doing something wrong

@iampete (see graphs in previous posts) Well these HNOTCH settings derived from some time spent in the new Filter Review tool totally knocked out any noise - more so than we could have done using the normal methods and theory with only the MissionPlanner FFT graphs. To achieve the same effectiveness would have taken more test flights specifically to experiment with HNOTCH values, not knowing what the results would have been. The revelation is the frequency is set lower than we would have previ…

I’m still waiting to hear back from another person where I used the Filter Review Tool to improve their rather ineffective HNOTCH settings.

I changed to using Chrome with the filter tool, Firefox just wasn’t working properly. There is still a few minor display or usability issues - but nothing that affects the results. For example clicking on a pre-filter heading (like Pete does in the video) doesn’t select all X Y Z check boxes. I still have to click each individual X Y Z for the pre/post/estimate graphs I want.

[iampete (Peter Hall)]

Great, glad to hear is useful!

@xfacta

For example clicking on a pre-filter heading (like Pete does in the video) doesn’t select all X Y Z check boxes.

Sorry, I miss spoke in the vid, you have to double click the headings.

To be honest I have only ever tried it with chrome. Optimizations for different browsers is well outside my expertise.

[dkemxr (Dave)]

…

@iampete

To be honest I have only ever tried it with chrome. Optimizations for different browsers is well outside my expertise.

Brave works fine.

[xfacta (Shawn)]

2nd win!
I was struggling to get much improvement with the old way of eye-balling the MP graphs and trying to guess at the HNOTCH settings. There was always a mess of frequencies and noise even in the post-filter.
Now it’s clean, and I’m sure this lack of noise almost makes PIDs less prone to oscillations and odd behaviour.

New tuning needed when change from Lipo to Li-ion

Yes! - to me that looks great now. Attitude control is tight but could maybe do with an autotune or a couple of tiny tweaks, but nobody is going to notice. Thanks for persisting and running more test flights. Let us know your thoughts on the tuning as it is now, and feel free to ask if there’s more you want to do with it. The yaw is a bit fuzzy and could do with a yaw Autotune. The noise and harmonics shows a BIG change: Pre-filter [image] Post-filter There’s nothing here of significance …

Pre

Post

[Michael_SC (Michael)]

That’s a major difference! Was it a case of trial and error in the filter tool until the post-filter estimate looked good?

[xfacta (Shawn)]

@Michael_SC

Was it a case of trial and error in the filter tool until the post-filter estimate looked good?

Somewhat, yes - there’s not much trial and error when you can so easily see the effect of changes

Note that in the Filter Review Tool the tendency is to try and set INS_HNTCH_HMNCS so that every significant peak is covered by a notch, and you end up with a value like 47 , yet in reality a value of 7 usually does the job and all those extra harmonics disappear anyway.

[MindProbe]

need help deciphering filter settings for this log

2023-06-23 12-39-56.bin

62inch propeller and u15xxl motor from tmotor

[xfacta (Shawn)]

I couldnt load the log in the filter review tool, it’s too big - at least for the FFT tools
Have you got a smaller log?

The only useful thing I could do was extract parameters, and normal log review.
You’ve got default Rate PIDs, you could probably try lower judging by the attitude control

ATC_RAT_PIT_D,0.0025
ATC_RAT_PIT_I,0.100
ATC_RAT_PIT_P,0.100
ATC_RAT_RLL_D,0.0025
ATC_RAT_RLL_I,0.100
ATC_RAT_RLL_P,0.100

In you log bitmask, add PID to logging.

FFT has likely picked up on a harmonic instead of the base frequency:

FFT_BW_HOVER,36.95223
FFT_FREQ_HOVER,131.4387

Probably about double what the real base frequency would be. So you would need to change to
FFT_MINHZ,15
guessing a bit, but it was 25

I think these are reversed:

INS_ACCEL_FILTER,20  //  should be 10
INS_GYRO_FILTER,10   //  should be 20

If that was a short flight then you would have to disable the IMU RAW logging and just use:

INS_LOG_BAT_MASK,1   //  or 3 or 7
INS_LOG_BAT_OPT,4

The other things to note are:

• no battery failsafe actions
• no geofence
• no Motor Emergency Stop RC channel assignment

If you’ve seen what damage a 30inch CF prop can do, you wouldnt stand within a mile of this thing without every possible safety feature in place (and then some).

[iampete (Peter Hall)]

@xfacta

I couldnt load the log in the filter review tool, it’s too big - at least for the FFT tools

It loads for me, but as you say its very big. Peak memory usage of about 3.7 GB. I’m not really sure how much we can do about this, we could give the option to not load all the IMUs that are in the log.

I have seen logs that give a out of memory error, I’m not sure if that is a hard coded limit that is the same for every browser/device or if it depends on how much memory you actually have free.

Mostly were using memory to save time, for example on first load we run the FFT over the whole log then if you change your analysis time it just re-averages the original FFT over the new time period. That means we have to keep all the FFT data in memory. We could throw away that data and recalculate each time, this would save a bunch of memory but mean that re-plotting takes just as long as the first load.