Recommended minimum prop tip clearance on quad

Turnigy S500 Quad, Emax 810 Kv motors, Pixhawk.

With 12x5 Aeronaut props tip clearance about 35 mm the quad is perfect . But with 13 inch props and 9mm clearance the motors pulse about 2 or 3 times a second, and the quad twitches up and down several inches. This is with 2 different makes of prop. They have all been balanced, and the 13 inch props work fine on bigger quads.

Could it be interaction with tip vortices with 13 inch props.