Hello.
I already posted my problem in 3.6 thread. My assault reaper 500 heliquad wobbles on roll axis in loiter mode, here is my log file, please help me, I am new to reading logs data 
2018-08-09 17-31-41.bin (828.2 KB)
ar500last.param (14.2 KB)
@Alex_Pilot I finished testing this for helicopter and I found the problem to be in the PSC (Position Controller) settings. They were too āhotā for helicopters for the velocity request. I have a PR I am going to send up shortly to fix that for TradHeli.
But I donāt know if this is your problem with the Reaper500 or not. Perhaps @tridge can help out here as he has some experience with these. For heli, the velocity xy P gain was 2 and the I-gain was 1, D gain .5. I had to turn the D-gain completely off, and cut both the other values in half. This sets the PSC back to what it was in Copter3.5. And it took care of the problem for heli. I first found the problem with my turbine helicopter (quite big machine running 1,730 rpm headspeed). Was able to duplicate it with my gassers with the governor turned up. Like my Raptor 716 gasser was fine with the defaults at 1,700 rpm. Switch to Idle2 @ 2,300 rpm in Loiter and it became COMPLETELY uncontrollable. Switching to Acro or Stabilize straightens it right out.
What confuses me about your PSC settings is that @Leonardthall who is the designer of New Loiter recommends setting the velocity I to half of velocity P. This is your settings in the param file you posted
PSC_ACC_XY_FILT,2
PSC_ACCZ_D,0
PSC_ACCZ_FF,0
PSC_ACCZ_FILT,20
PSC_ACCZ_I,1
PSC_ACCZ_IMAX,800
PSC_ACCZ_P,0.5
PSC_ANGLE_MAX,0
PSC_POSXY_P,0.7
PSC_POSZ_P,1
PSC_VELXY_D,0.5
PSC_VELXY_D_FILT,5
PSC_VELXY_FILT,5
PSC_VELXY_I,0.7
PSC_VELXY_IMAX,1000
PSC_VELXY_P,0.3
PSC_VELZ_P,5
Your machine is obviously already using custom settings by somebody who tuned one of these, as I note both P and I are considerably lower than the defaults set in the code. I suspect the velocity D-gain is the problem. But Iād better let hopefully @tridge or @Leonardthall comment on this because I donāt know for sure.
@ChrisOlson Did you look at Alexās log file. Check out the IMU gyrX and accZ. The accZ is less concerning to me as it goes away once he takes off. However the gyrX has significantly high noise which carries through to the roll attitude signal in ATT. I think this needs to be addresed before moving on to the loiter issues. Thoughts?
Yes, I did see that and the CP quads are quite prone to severe vibration. But normally that affects handling in Stabilize and Acro too. Thatās why Iām not totally sure. It is very hard to get rid of the vibration in the CP quads because of the drivetrain. I remember @tridge saying he added lead weights to the controller to damp it.
Well it is strange that there is significant noise in the roll axis but not in the pitch axis. It would make more sense if I saw noise in both, especially since it is an X configuration. If it was a + configuration then I could buy that vibrations were caused by a motor or rotors vibrating. I question the roll sensor because the lateral and longitudinal accels and pitch gyro all look like they have low noise. But then why am I also seeing this in the other X gyro. very strange.
Yeah, I think it is experiencing quite a bit of vibration that the IMUās arenāt handling well. But I think that should only cause some problems with position holding and poor overall attitude control, and not necessarily the quite severe oscillation it appears to go into in Loiter.
If it were me I would try setting the PSC_VELXY_D to zero and see if it stops the oscillation. That velocity request D value was hard-coded to zero in Copter 3.5. Leonard moved it to the params and it was assigned a default value of 0.5 in Copter 3.6.
So I would set it to zero and see if it stops oscillating. Then set PSC_VELXY_P to 1 and PSC_VELXY_I to 0.5 and see how it holds position. Those were the settings in Copter 3.5 that seemed to work well for a long time.
I think those velocity request gains were turned up for New Loiter for some reason.
I looked thru this again and I think I have to agree that vibration needs to be fixed in this aircraft first. The clipping on the IMUās is ballistic and it is not going to fly good until you get this clipping issue fixed.
Thank you for your participation in this. I will try to fix vibration as much as possible, maybe even add some lead weights on Pixracer on top.