Found an issue when upgrading Pixhawk from APM:plane 3.5.3 to 3.7.1. The upper input limit of the RPM is 1.2KHz in newer versions. Using orange brushless RPM sensor from hobbyking with a 4 pole motor and 5.2 gear box. The sensor can output up to 1.6KHz for an RPM of 9300. Changing the RPM scale did not improve the performance
Input frequency less than 1.2K seems to work fine
I confirmed my max RPM setting was 10M. Using AUX 5 for RPM input with relay 1 pin set to -1. Also BRD_PWM_CNT set to 0.
Confirmed this using a function generator and o-scope. Reverting back to older firmware versions makes the issue go away with reliable RPM values exceeding 3KHz. I did a test this morning on a new Pixhawk 1 and newer APM:copter revisions gave the same results.